Subject: RE: [nfsv4] Create Session for pNFS DS

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 6 Feb 2007 12:07:25 -0500

> The NFSv4.1 spec doesn't attempt to define a "single server".
> Instead it talks about server owners (with major and minor fields),
> server scopes, SSVs, and target GSS principals, and thus allows for
> degrees of server singularities. 

Well it does have a chapter on single-server namespace, and one
on multi-server namespace, but I think it is just of question
of getting those (relatively) old chapters to "get with the 
program".  One of them was reworked and the other esentially
written for 4.1, but early in the cycle.

I'll create an issue for this.  It won't be addressed in draft-09.

> Perhaps the spec should expand on this.

I think this needs some introductory text because it is easy
to assume the older model.  I'll create an issue and assign it
to you.  If you think something needs to be in the introductory
chapter as opposed to the protocol structure chapter, let me know.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Tuesday, February 06, 2007 11:48 AM
To: [email-masked]
Subject: Re: [nfsv4] Create Session for pNFS DS

--- "Talpey, Thomas" <[email-masked]> wrote:

> At 12:27 AM 2/6/2007, Mike Eisler wrote:
> >  A pnfs server cluster can return the same clientid and
> >  sessionid to each EXCHANGE_ID to an MDS and DS if it
> >  wants (in a loosely coupled pnfs server cluster this
> >  might be the norm).
> 
> I don't think the sessionid should come into play here. It's only
> unique in the scope of a single server, so its absolute value in
> the network is somewhat meaningless.

I don't know what you think a "single server" is.

The NFSv4.1 spec doesn't attempt to define a "single server".
Instead it talks about server owners (with major and minor fields),
server scopes, SSVs, and target GSS principals, and thus allows for
degrees of server singularities. Perhaps the spec should expand
on this.

> >  Similarly, if each MDS and DS return the same clientid
> >  and sessionid, to prevent the client from erroneously
> >  attempting session trunking,
> 
> You mean that the client might attempt a second CREATE_SESSION
> to a given DS, and get back the same session it already has from

No that is not what I meant (besides which, multiple, unique,
not replayed, CREATE_SESSIONs with the same clientid to the same server
network address [or a different network address but the same
server_owner and server_scope] always produce a new sessionid). 

>From the specification's section entitled "Trunking":

   When two connections have a matching so_major_id and so_minor_id, 
   the client may bind both connections to a common session; this 
   is session trunking. When two connections have a matching
   so_major_id, but different so_minor_id, the client will need 
   to create a new session for the clientid in order to use 
   the connection; this is clientid trunking. 

Although draft-08 neglects to mention that the server_scope needs
to match in both session and clientid trunking; corrected in draft-09.

> the first? That violates the session scope - not unique on a single
> server. The DS can reject the second session, of course. That would
> certainly prevent it.
> 
> Tom.
> 
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 6 Feb 2007 12:10:58 -0500

I think you have the right solution on EXCHANGE_ID.  Who
was the idiot that first suggested that we not do an
EXCHANGE_ID on the data server?  Actually, I think it was
me.  Sorry about that.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Tuesday, February 06, 2007 12:28 AM
To: [email-masked]
Subject: Re: [nfsv4] Create Session for pNFS DS

Today at Connectathon a large group of pNFSers discussed
this.

There are several issues with draft-08 and the impending
draft-09 (as it exists in the cvs server):

- What sequenceid does CREATE_SESSION take when invoked
  against a DS?

-- Also in pending draft-09, the CREATE_SESSION
   sequenceid is global across the pNFS cluster, so this
   means the CREATE_SESSIONs to all DSes for the same
   clientid have to be serialized. Not good.

- The clientid is currently global across all nodes in
  the pNFS cluster.  However, consider a vast cluster
  with lots of MDSes and DSes, where a given DS could
  potentially be holding parts of files from many
  MDSes. When given a clientid in a CREATE_SESSION call,
  there is no efficient way for the DS to figure out which
  node has the MDS for that clientid.

- There is no may to bind a session to a DS or MDS for the
  case when both listen on the same network address. It
  would be nice to allow a DS to have an optimal
  implementation that just does the DS operations. This
  is actually fixed in the impending draft-09.

During today's meeting we came to the realization that
if we changed to the "MUST NOT" that bans EXCHANGE_ID to
a DS to "MUST", that each DS and MDS that the client
presents a given co_owner to can return a unique
clientid to the client. Doing this solves the "what
CREATE_SESSION sequenceid should the client use to the DS"
problem. It allows CREATE_SESSION to each DS to happen in
parallel. It solves the problem of routing to the correct
global clientid because there is no global clientid.
One thing that has to change is that EXCHANGE_ID and not
CREATE_SESSION takes a flag that indicates whether the
client is trying to create a clientid (versus creatting
a session) to an MDS or DS.

This still leaves an issue.

When a COMPOUND is sent to a DS, with a READ or WRITE
operation, the layout stateid does not map to the MDS,
because there is no longer a global clientid, and the
sessionid in the SEQUENCE doesn't map to the global clientd
(it maps to the local DS clientid). We do need a mapping
of the stateid to location of the MDS. What we settled on
before the end of the meeting was that the DS file handle
(passed via the PUTFH that precedes READ and WRITE) can
contain the MDS location.  We observed that DS file handles
are allowed to be different than MDS file handles. We
were mistakenly worried about the DS file handles going
stale if the MDS location changed. However, we corrected
our mistake after we observed that the DS file handles are
part of the layoyut, and they are allowed to be different
than the corresponding MDS file handle. Because the DS file
handles are part of the layout, the DS file handles can
be recalled (invalidated) as part of recalling the layout.
One objection to this is that caching is a problem. Another
mistake: caching isn't a problem because while a client has
a layout, the MDS file handle needs to map to each DS file
handle, and each DS file handle needs to map to its MDS
file handle. When the layout is recalled, the cache must
destory the DS file handle to MDS file handle mappings,
but the data can still live, it is always indexable by
the persistent MDS file handle.

One other observation that I thought of after the meeting:

  A pnfs server cluster can return the same clientid and
  sessionid to each EXCHANGE_ID to an MDS and DS if it
  wants (in a loosely coupled pnfs server cluster this
  might be the norm).  If each DS and MDS do (or can)
  return the same clientid from EXCHANGE_ID, to prevent
  the client from erroneously attempting clientid trunking
  (and getting an error when attempting to READ or WRITE
  a data range the DS does not "own"), the combined
  eir_server_owner.so_major and eir_server_scope results
  from EXCHANGE_ID for the MDS and each DS may need to
  be different.  The client can compare the RPCSEC_GSS
  target principal of the MDS and each DS to prove the
  claim of clientid trunking.

  Similarly, if each MDS and DS return the same clientid
  and sessionid, to prevent the client from erroneously
  attempting session trunking, the combined so_major,
  so_minor, and eir_server_scope results from the MDS and
  each DS may need to be different. The client can use
  the SSV to prove the claim of session trunking.