Subject: [nfsv4] layout stateids and pNFS callbacks

From: [email-masked]
Date: Sun, 8 Jul 2007 21:16:57 -0400

> > > LAYOUTGET will take a normal share reservation stateid
> > > (as returned by OPEN) or a delegation stateid as
> > > an additional argument. If successful, it will return
> > > a layout stateid. For a given file/layout type pair,
> > > the same layout stateid is always returned by LAYOUTGET,
> > > except when there is a server restart.
> > 
> > Hmm, I thought we're going in a direction where the
> > seqid part of the stateid is incremented by the server
> > to enforce strict ordering of LAYOUTGETs and LAYOUTRETURNs.
> 
> I don't recall such a discussion, but that seems like a reasonable
> thing to do; consider my proposal as so amended.

As part of doing this, can we use stateids instead of sessions
to sort out pNFS race conditions?  This approach requires the
server to pass the stateid in callbacks so the client can order
the callback wrt the sequence being tracked by the seqid part.

FWIW, our existing system (MPFS) does this, so we have lots of
experience with the details that have to be gotten right.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 09 Jul 2007 12:10:10 -0400

> the callback wrt the sequence being tracked by the seqid part.

stateids don't suffice to sort out all the possible race conditions. See
the problems that we have in v4.0 with delegations being recalled by the
server before the client has processed the results of the OPEN call.

Cheers
  Trond

---

From: [email-masked]
Date: Mon, 9 Jul 2007 13:05:06 -0400

Trond,

> server before the client has processed the results of the OPEN call.

That depends on the details of how they're used.  We have experience
(i.e., running code) with FMP for MPFS that a single sequence count
*that is included in all the relevant operations, including callbacks*
is sufficient to sort this out if one pays close attention to all the
details.

The class of race you describe is handled at the client by observing
that the callback carries seqid "n", but the client hasn't processed
the reply to operation "n", and hence the client holds the callback
until operation "n" (the OPEN) is processed.  OTOH, if the callback
doesn't carry a seqid, the game's over ... and the fix is not to make
that design mistake ;-).

Thanks,
--David

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 09 Jul 2007 13:22:11 -0400

> doesn't carry a seqid, the game's over ... and the fix is not to make

OK. I see where you're coming from now. Basically you are extending the
v4.0 sequencing scheme to work with callbacks? I suppose that can work,
but it does seem a tad inelegant.

Why do you want an alternative to sessions?

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 9 Jul 2007 14:31:08 -0400

I'm getting confused here.  Originally, there was a 
proposal to add stateid's to to layouts.  Now it seems
that something entirely different is being proposed.

It sounds like David Is proposing a single global
(for every pair of client and meta-data server) sequence
id.  Is that in fact the case?  It doesn't sound like
the fact that stateid's have sequence values meets his
requirement to have a "a single sequence count *that is 
included in all the relevant operations, including callbacks*"
at least as part as the "single" part.  With stateid's
you still have the same issue with delegation recall
that Trond points out.  

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Monday, July 09, 2007 1:22 PM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [nfsv4] layout stateids and pNFS callbacks

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 09 Jul 2007 14:53:42 -0400

> you still have the same issue with delegation recall

I believe that David is proposing that the client supply a sequence id
(presumably modelled on the scheme used by LOCK in the v4.0 spec) in the
GETLAYOUT call, and that the server then uses that sequence id as part
of the stateid. That way the client always knows whether or not the
stateid refers to an outstanding call or a completed call. David is that
correct?

Trond

---

From: Benny Halevy <[email-masked]>
Date: Tue, 10 Jul 2007 10:54:43 +0300

> stateid refers to an outstanding call or a completed call. David is that

I won't speak for David, but what I had in mind is a layout stateid per
<mds, client, file> that's incremented every time the layout changes, i.e.
on each LAYOUTGET and each LAYOUTRETURN.  On CB_LAYOUTRECALL, the server
sends the last stateid it knows about for this file and the client must
wait on every outstanding LAYOUTGET and LAYOUTRETURN operation until
it gets this stateid back before starting to return layouts for this
recall.  Any further LAYOUTGETs should be rejected by the server if they
conflict with the recall, otherwise they can be served.  The recall is done
either by returning a NOMATCHINGLAYOUT error to CB_LAYOUTRECALL or by
a respective LAYOUTRETURN.

Again, I propose to make this explicit by e.g. quoting the layout stateid
the CB_LAYOUTRECALL was sent for otherwise it will be harder to handle an
inflight LAYOUTRETURN that hasn't been seen by the server before it sent
out the CB_LAYOUTRECALL and that will effectively complete the recall.

Note that the per-file layout stateid does not solve the problem for
recalling and returning layouts for FSID and ALL.  A per-client layout
stateid may help here but it can limit parallelism.

Benny

---

From: [email-masked]
Date: Sat, 14 Jul 2007 14:52:28 -0400

Sorry for the delay in getting back to this, but between Trond
and Benny, what's been discussed is a specific example of what
I had in mind - thanks.

> >> Why do you want an alternative to sessions?

(1) Make it possible for pNFS to work in v4.0 w/o sessions.  I'm
	hearing that sessions have turned out to be a significant
	implementation concern.
(2) Make the pNFS protocol simpler.  Right now, the sessions
	solution to a race condition involves identifying the
	outstanding operation(s), which is somewhat more involved
	than having the client and server track a per-layout sequence
	number.
(?3?) The result may be more robust.  With sessions, if the server
	misses an outstanding operation in the sessions state (lots
	of concurrency there, so lots of opportunities to get things
	subtly wrong), things go awry.  With a sequence number, the
	check is based on state instead of existence of outstanding
	operations, and there's much less potential concurrency
	involved.  The "?"s are because I'm not sure how strongly
	I want to lean on this argument, and it's primarily a
	consequence of (2) anyhow.

Thanks,
--David

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 16 Jul 2007 10:26:29 -0400

>	hearing that sessions have turned out to be a significant

V4.0.5?  Is there going to be a spec for this and if so, who is going to
write it?  And if not, how are you going to get inter-operable
implementations?  If you have implementation concerns about sessions,
then you will really have them about a protocol without a spec.

If the result is a better v4.1, I'm perfectly OK with this change, but I
think trying to optimize V4.1 for V4.0.5 is a real mistake.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Saturday, July 14, 2007 2:52 PM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: [nfsv4] layout stateids and pNFS callbacks

Sorry for the delay in getting back to this, but between Trond and
Benny, what's been discussed is a specific example of what I had in mind
- thanks.

Thanks,
--David

> >> I'm getting confused here.  Originally, there was a proposal to add
> >> stateid's to to layouts.  Now it seems that something entirely 
> >> different is being proposed.
> >>
> >> It sounds like David Is proposing a single global (for every pair
> >> of client and meta-data server) sequence id.  Is that in fact the 
> >> case?  It doesn't sound like the fact that stateid's have sequence 
> >> values meets his requirement to have a "a single sequence count 
> >> *that is included in all the relevant operations, including 
> >> callbacks*"
> >> at least as part as the "single" part.  With stateid's you still 
> >> have the same issue with delegation recall that Trond points out.
> > 

> >> [email-masked] https://www1.ietf.org/mailman/listinfo/nfsv4
> > 
> >

---

From: "Muntz, Daniel" <[email-masked]>
Date: Mon, 16 Jul 2007 10:01:54 -0700

If someone is thinking of doing pNFS for v3/v4.0, sign me up.  I bet
there are a few others with similar interests.

  -Dan

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Saturday, July 14, 2007 2:52 PM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: [nfsv4] layout stateids and pNFS callbacks

Thanks,
--David

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Mon, 16 Jul 2007 13:41:21 -0400

> think trying to optimize V4.1 for V4.0.5 is a real mistake.

If I remember correctly, the main reason that pNFS requires Sessions is to
handle the races between LAYOUTGET/RETURN etc. Perhaps what David is
suggesting it that NFSv4.1 pNFS with his proposed stateid changes would not
require sessions, and therefore be an independent 4.1 feature.

-->Andy

---

From: Robert Gordon <[email-masked]>
Date: Mon, 16 Jul 2007 13:06:57 -0500

> 	hearing that sessions have turned out to be a significant

Perhaps the approach should be to identify those implementation
concerns and address those issues, rather than having the appearance
(perceived or real) of back tracking.

Robert.

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 16 Jul 2007 13:19:16 -0500

On Jul 16, 2007, at 12:41 PM, William A. (Andy) Adamson wrote:

>
>
> On 7/16/07, Noveck, Dave <[email-masked]> wrote: > (1) Make  
> it possible for pNFS to work in v4.0 w/o sessions.  I'm
> If I remember correctly, the main reason that pNFS requires  
> Sessions is to handle the races between LAYOUTGET/RETURN etc.  
> Perhaps what David is suggesting it that NFSv4.1 pNFS with his  
> proposed stateid changes would not require sessions, and therefore  
> be an independent 4.1 feature.

I am having a difficult time understanding what "an independent 4.1  
feature"
would be.

NFSv4.1 requires Sessions.  That has been decided long ago and
the main reason for its mandatory nature for 4.1 was the variety
of issues we would have in defining features in two flavors "with  
sessions"
and "without sessions".  We deemed it as too much work and complexity
for the flexibility in implementation.

As with any of the features defined in the NFSv4.1 Internet Draft, if
there is concern or experience about the unneeded or untenable
complexity of implementation, they need to be discussed here.
The issues may be real and may or should require a change in the
specification.  It may be that there is a misunderstanding of what
has been included in the Internet Draft and we just need to clarify
the language.

Please bring the issues forward and participate in the discussion.
I really don't want any surprises as we move forward.

Spencer

> > > I believe that David is proposing that the client supply a  
> sequence
> id
> > > (presumably modelled on the scheme used by LOCK in the v4.0  
> spec) in
> the
> > > stateid refers to an outstanding call or a completed call.  
> David is
> that
> > > correct?
> >
> > <mds, client, file> that's incremented every time the layout  
> changes,
> i.e.
> > conflict with the recall, otherwise they can be served.  The  
> recall is
> done
> > the CB_LAYOUTRECALL was sent for otherwise it will be harder to  
> handle
> an
> > >> From: Trond Myklebust [mailto:[email-masked] ]

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 16 Jul 2007 13:23:47 -0500

> 	hearing that sessions have turned out to be a significant

You need to clarify this David.  Was this a typo and you meant "in  
v4.1 w/o sessions"
or did you mean using NFSv4.0 as a new files based layout type?

> 	than having the client and server track a per-layout sequence

We will continue the stateid proposal for layouts and see if it
can be adapted to provide what is mentioned.  However, Sessions
is still going to be a mandatory to implement feature unless
there is a good reason not to and then the proposals should be
in the form of what issues there are with Sessions and if/how
they can be addressed.

Before we go off to far in the deep end, it may be best to give
David an opportunity to respond.

Spencer

---

From: "Muntz, Daniel" <[email-masked]>
Date: Mon, 16 Jul 2007 11:52:51 -0700

Perhaps what Andy is suggesting is that pNFS could be a feature,
independent of 4.1.  I think there might even have been
research/prototyping done in this area, including distributed metadata
service.
 
  -Dan

________________________________

From: William A. (Andy) Adamson [mailto:[email-masked]] 
Sent: Monday, July 16, 2007 10:41 AM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]; [email-masked];
[email-masked]
Subject: Re: [nfsv4] layout stateids and pNFS callbacks

	>       hearing that sessions have turned out to be a
significant
	
	V4.0.5?  Is there going to be a spec for this and if so, who is
going to 
	write it?  And if not, how are you going to get inter-operable
	implementations?  If you have implementation concerns about
sessions,
	then you will really have them about a protocol without a spec.
	
	If the result is a better v4.1, I'm perfectly OK with this
change, but I
	think trying to optimize V4.1 for V4.0.5 is a real mistake.

If I remember correctly, the main reason that pNFS requires Sessions is
to handle the races between LAYOUTGET/RETURN etc. Perhaps what David is
suggesting it that NFSv4.1 pNFS with his proposed stateid changes would
not require sessions, and therefore be an independent 4.1 feature.

-->Andy

	-----Original Message-----
	From: [email-masked] [mailto:[email-masked]]
	Sent: Saturday, July 14, 2007 2:52 PM
	To: [email-masked]; [email-masked]
	Cc: [email-masked]; [email-masked]
	Subject: RE: [nfsv4] layout stateids and pNFS callbacks
	
	Sorry for the delay in getting back to this, but between Trond
and
	Benny, what's been discussed is a specific example of what I had
in mind
	- thanks. 

	
	(1) Make it possible for pNFS to work in v4.0 w/o sessions.  I'm
	        hearing that sessions have turned out to be a
significant 
	        implementation concern.
	(2) Make the pNFS protocol simpler.  Right now, the sessions
	        solution to a race condition involves identifying the
	        outstanding operation(s), which is somewhat more
involved 
	        than having the client and server track a per-layout
sequence
	        number.
	(?3?) The result may be more robust.  With sessions, if the
server
	        misses an outstanding operation in the sessions state
(lots 
	        of concurrency there, so lots of opportunities to get
things
	        subtly wrong), things go awry.  With a sequence number,
the
	        check is based on state instead of existence of
outstanding
	        operations, and there's much less potential concurrency 
	        involved.  The "?"s are because I'm not sure how
strongly
	        I want to lean on this argument, and it's primarily a
	        consequence of (2) anyhow.
	
	Thanks,
	--David
	
	> >> I'm getting confused here.  Originally, there was a
proposal to add 
sequence
	id
spec) in
	the
	> > GETLAYOUT call, and that the server then uses that sequence
id as 
	part
David is
	that
	> > correct?
	>
	> I won't speak for David, but what I had in mind is a layout
stateid 
	per
changes,
	i.e.
	> on each LAYOUTGET and each LAYOUTRETURN.  On CB_LAYOUTRECALL,
the
	server
	> sends the last stateid it knows about for this file and the
client 
	must
	> recall.  Any further LAYOUTGETs should be rejected by the
server if 
	they
recall is
	done
	> Again, I propose to make this explicit by e.g. quoting the
layout
	stateid
	> the CB_LAYOUTRECALL was sent for otherwise it will be harder
to handle
	an
	> inflight LAYOUTRETURN that hasn't been seen by the server
before it
	sent

---

From: [email-masked]
Date: Mon, 16 Jul 2007 14:59:01 -0400

Spencer (and Dave),

> > going to write it?  And if not, how are you going to get
inter-operable
sessions,
> > If the result is a better v4.1, I'm perfectly OK with this change,  
> > but I think trying to optimize V4.1 for V4.0.5 is a real mistake.

The result is definitely a better v4.1 courtesy of (2), which says
that a sequence number is a simpler, better and (arguably) more
robust way to solve the problems:

The fact that this approach also functionally separates pNFS from
sessions, thereby making it possible to specify pNFS for v4.0 without
sessions (IMHO, sessions are and should be v4.1 only) is a bonus -
that's mostly a side effect of introducing orthogonality between
pNFS and sessions (a good idea in general, IMHO).  FWIW, I agree
with Dave Noveck's comment that in order to use pNFS interoperably
with v4.0, a specification for doing so is required - that would be
something to be done in the future if there's sufficient interest.

> feature" would be.

Distinct features are independent when there is an absence of
functional cross-coupling that requires one feature to understand
the internal state of another - as currently specified, pNFS requires
understanding the internal state of sessions.  Absence of functional
cross-coupling helps with implementation modularity, even when all
the features involved are mandatory to implement.  To the extent
that everyone already knows this, I apologize for the architectural
pedantry.

> sessions" and "without sessions".  We deemed it as too much work
> and complexity for the flexibility in implementation.

That's ok - I have no problem with this approach.

> As with any of the features defined in the NFSv4.1 Internet Draft,
> if there is concern or experience about the unneeded or untenable

IMHO, sessions are needed in v4.1 for reasons other than pNFS - or
to turn it around, if the only use of sessions were pNFS, (IMHO)
sessions would not be in v4.1.  I don't have any general issue with
sessions as a required v4.1 feature - the concern here starts from
our original decision to take advantage of sessions to remove some
pNFS race conditions because sessions were already in the spec.
Recent discussion suggests that using a sequence number is a better
way (IMHO) to deal with pNFS race conditions, and hence the concern
is not about whether sessions should be a required v4.1 feature,
but rather about whether pNFS should be obligated to use sessions
when there is a better solution possible.

Thanks,
--David