Subject: RE: [nfsv4] Create Session for pNFS DS

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 5 Feb 2007 17:53:59 -0500

Correct me if I'm wrong but it sure sounds to me like
your answer to Spencer's question about why metadata can 
handle any number of fs's in a session (directing to
the appropriate implementation) while you require separate
sessions for eahc fs the data server is simply that that 
is the way you have chosen to implement things.

Is there some other reason that makes it particularly 
desirable to have the protocol have sessions tied to
particular fs's?  I'm looking for something more general
than simply allowing you to maintain an implementation
choice you have made.  Offhand, it seems to me that sessions
are very near to the communication layer and I don't see
any obvious reason to tie them to file systems.  What am I
mssing?

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]] 
Sent: Monday, February 05, 2007 4:52 PM
To: Spencer Shepler
Cc: J. Bruce Fields; [email-masked]; [email-masked]
Subject: Re: [nfsv4] Create Session for pNFS DS

Spencer Shepler <[email-masked]> wrote on 02/05/2007 12:22:01
PM:

> 
> On Feb 5, 2007, at 2:13 PM, J. Bruce Fields wrote:
> 
> > On Mon, Feb 05, 2007 at 01:14:33PM -0600, Spencer Shepler wrote:
> >> Can you explain further as to why there must be multiple sessions
> >> at the DS when that is unnecessary at the MDS even though both
> >> may be servicing different filesystems?
> >
> > It's possible that a single host could have multiple roles--as a
> > metadata server, or as a data server to any of several possible 
> > metadata
> > servers.  The host needs to know which role it's playing before it
can
> > respond to a given request.  So the metadata server needs to insert 
> > this
> > role information into something that the client will pass to the
data
> > server.
> >
> > Prototypes have been using the filehandle for this.  Maybe the 
> > clientid
> > could be used instead?  But the clientid is smaller and already 
> > subject
> > to a number of other requirements (to identify stale clientid's, 
> > etc.),
> > so that carving up the clientid space between metadata servers may
> > require more complicated coordination.
> 
> Ah, okay, this is what I was looking for...  It isn't tied to the
> underlying filesystem but the potential dual-role that a server may
> be playing.

This for both the dual-role and the need to associate create session
with 
an underlying file system so we can know which one to use to validate
the 
session on the DS. In the Linux server implementation all the 
communication among the machines are through the backend file system.
Marc.
 
> This issue has arisen recently and a couple of us, including Mike,
have
> been thinking about how to differentiate for this reason but for
> others as well (to correctly set up the initial sequencing of the
> session with the data-server as well).
> 
> So, the requirement is the ability of differentiating the multiple
> roles a server may be playing and ensuring that we have a method
> of allowing the server to communicate that view to itself through
> the normal client interaction of the pNFS behavior?
> 
> Spencer
> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 05 Feb 2007 17:16:05 -0600

> any obvious reason to tie them to file systems.  What am I

I agree in that the intent of sessions is to deal with
communication issues specific to NFS' use of an underlying
transport (multiple TCP connections over the lifetime of the
client/server interaction) and hence the need for the EOS.

I can understand the requirement to deal with a system acting
as MDS and DS(es) and the need to effectively differentiate
and possibly linking the differentiation to the deviceid.

Creating a relationship with individual filesystems seems
like a crossing of an architectural boundary that will eventually
lead to undesirable changes in the sessions model we have today.

Spencer

---

From: Marc Eshel <[email-masked]>
Date: Mon, 5 Feb 2007 20:22:35 -0800

We had a session discussion this afternoon at the connectathon and Mike 
have a clean solution for the problem I raised as well as some other 
session issues that he will share soon.
Thanks, Marc.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 5 Feb 2007 21:27:49 -0800 (PST)

Today at Connectathon a large group of pNFSers discussed
this.

There are several issues with draft-08 and the impending
draft-09 (as it exists in the cvs server):

- What sequenceid does CREATE_SESSION take when invoked
  against a DS?

-- Also in pending draft-09, the CREATE_SESSION
   sequenceid is global across the pNFS cluster, so this
   means the CREATE_SESSIONs to all DSes for the same
   clientid have to be serialized. Not good.

- The clientid is currently global across all nodes in
  the pNFS cluster.  However, consider a vast cluster
  with lots of MDSes and DSes, where a given DS could
  potentially be holding parts of files from many
  MDSes. When given a clientid in a CREATE_SESSION call,
  there is no efficient way for the DS to figure out which
  node has the MDS for that clientid.

- There is no may to bind a session to a DS or MDS for the
  case when both listen on the same network address. It
  would be nice to allow a DS to have an optimal
  implementation that just does the DS operations. This
  is actually fixed in the impending draft-09.

During today's meeting we came to the realization that
if we changed to the "MUST NOT" that bans EXCHANGE_ID to
a DS to "MUST", that each DS and MDS that the client
presents a given co_owner to can return a unique
clientid to the client. Doing this solves the "what
CREATE_SESSION sequenceid should the client use to the DS"
problem. It allows CREATE_SESSION to each DS to happen in
parallel. It solves the problem of routing to the correct
global clientid because there is no global clientid.
One thing that has to change is that EXCHANGE_ID and not
CREATE_SESSION takes a flag that indicates whether the
client is trying to create a clientid (versus creatting
a session) to an MDS or DS.

This still leaves an issue.

When a COMPOUND is sent to a DS, with a READ or WRITE
operation, the layout stateid does not map to the MDS,
because there is no longer a global clientid, and the
sessionid in the SEQUENCE doesn't map to the global clientd
(it maps to the local DS clientid). We do need a mapping
of the stateid to location of the MDS. What we settled on
before the end of the meeting was that the DS file handle
(passed via the PUTFH that precedes READ and WRITE) can
contain the MDS location.  We observed that DS file handles
are allowed to be different than MDS file handles. We
were mistakenly worried about the DS file handles going
stale if the MDS location changed. However, we corrected
our mistake after we observed that the DS file handles are
part of the layoyut, and they are allowed to be different
than the corresponding MDS file handle. Because the DS file
handles are part of the layout, the DS file handles can
be recalled (invalidated) as part of recalling the layout.
One objection to this is that caching is a problem. Another
mistake: caching isn't a problem because while a client has
a layout, the MDS file handle needs to map to each DS file
handle, and each DS file handle needs to map to its MDS
file handle. When the layout is recalled, the cache must
destory the DS file handle to MDS file handle mappings,
but the data can still live, it is always indexable by
the persistent MDS file handle.

One other observation that I thought of after the meeting:

  A pnfs server cluster can return the same clientid and
  sessionid to each EXCHANGE_ID to an MDS and DS if it
  wants (in a loosely coupled pnfs server cluster this
  might be the norm).  If each DS and MDS do (or can)
  return the same clientid from EXCHANGE_ID, to prevent
  the client from erroneously attempting clientid trunking
  (and getting an error when attempting to READ or WRITE
  a data range the DS does not "own"), the combined
  eir_server_owner.so_major and eir_server_scope results
  from EXCHANGE_ID for the MDS and each DS may need to
  be different.  The client can compare the RPCSEC_GSS
  target principal of the MDS and each DS to prove the
  claim of clientid trunking.

  Similarly, if each MDS and DS return the same clientid
  and sessionid, to prevent the client from erroneously
  attempting session trunking, the combined so_major,
  so_minor, and eir_server_scope results from the MDS and
  each DS may need to be different. The client can use
  the SSV to prove the claim of session trunking.

---

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 06 Feb 2007 11:28:32 -0500

At 12:27 AM 2/6/2007, Mike Eisler wrote:
>  might be the norm).

I don't think the sessionid should come into play here. It's only
unique in the scope of a single server, so its absolute value in
the network is somewhat meaningless.

>  attempting session trunking,

You mean that the client might attempt a second CREATE_SESSION
to a given DS, and get back the same session it already has from
the first? That violates the session scope - not unique on a single
server. The DS can reject the second session, of course. That would
certainly prevent it.

Tom.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 6 Feb 2007 08:48:08 -0800 (PST)

> the network is somewhat meaningless.

I don't know what you think a "single server" is.

The NFSv4.1 spec doesn't attempt to define a "single server".
Instead it talks about server owners (with major and minor fields),
server scopes, SSVs, and target GSS principals, and thus allows for
degrees of server singularities. Perhaps the spec should expand
on this.

> to a given DS, and get back the same session it already has from

No that is not what I meant (besides which, multiple, unique,
not replayed, CREATE_SESSIONs with the same clientid to the same server
network address [or a different network address but the same
server_owner and server_scope] always produce a new sessionid). 

>From the specification's section entitled "Trunking":

   When two connections have a matching so_major_id and so_minor_id, 
   the client may bind both connections to a common session; this 
   is session trunking. When two connections have a matching
   so_major_id, but different so_minor_id, the client will need 
   to create a new session for the clientid in order to use 
   the connection; this is clientid trunking. 

Although draft-08 neglects to mention that the server_scope needs
to match in both session and clientid trunking; corrected in draft-09.