Subject: RE: LOOKUP vs OPEN in NFSV4

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 3 Jul 2001 04:59:47 -0700

Well, if you do a LOOKUP or OPEN, you (the client) has
to be interested in the result, while for some other
operations that change the current filehandle, this 
may not be the case.  So I would not extend this to
any other ops.

As far as the chances of making such a change, they 
aren't very great because most people within the 
working group are satisfied with the GETFH approach.
The size of the change isn't the issue.

-----Original Message-----
From: Som [mailto:[email-masked]]
Sent: Tuesday, July 03, 2001 12:06 AM
To: Noveck, Dave; [email-masked]
Subject: RE: LOOKUP vs OPEN in NFSV4

--- "Noveck, Dave" <[email-masked]> wrote:
> Som wrote:
> > 3. Why don't the structure OPEN4resok contain the
> OPEN
> > generated filehandle instead of client doing one
> more
> > GETFH and getting the same filehandle? It atleast
> > reduces one operation for each open! 
> 
> Actually, I think OPEN and LOOKUP should directly
> return
> the filehandle (and not just when the op succeeds),
> but 
> this is very much a minority opinion in the working
> group.
> 
> V4 only returns handles via GETFH.  Ops that produce
> filehandles only change the current filehandle and
> there
> is a value in having that kind of uniformity.
> 
> The reason I think that LOOKUP and OPEN should
> return
> filehandles has to do with the handling of
> multi-component
> lookups/opens.  If a lookup encounters a symlink (or
> some
> other error situation), you'd like to be able to
> restart 
> the process at the point of failure, which means
> that you 
> need the filehandle where you stopped.  A
> COMPOUNDEDed
> GETFH can't do it, since the error stops the
> request,
> preventing the GETFH from being executed.  This
> makes 
> multi-component lookup/open kind of useless in any 
> environment where symlink might be encountered.
> 
>  

So, now that we have a very good reason to return
filehandle as part of OPEN4resok structure , applying
the same consistency rule ( :-) ) , why not return
filehandles for every state modifying operations? I
mean whenever client has to do GETFH, why not just
return the result always in OPEN4resok strucutre? 

Since NFSV4 is a very new protocol from earlier
versions this change should be possible to do....this
also saves considerable amount of wire traffic...is
this too big a change for NFSV4?

thanks, som.

__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 3 Jul 2001 05:19:54 -0700

I was referring to using a COMPOUNDed GETFH in the
context of a multi-component lookup.  Your example
is the reasonable way to do a multi-component so
that you can recover.  But the issue I see is that 
it doesn't use the multi-component lookup feature
of LOOKUP.  So I conclude that feature really doesn't
have a reasonable use.

It seems to me the choices are:

     a) Get rid of multi-component LOOKUP/OPEN and tell
        people to use COMPOUNDed single-component operations
        instead.

     b) Return enough information to make the multi-component
        LOOKUP feature useful.

I guess c) would be to leave things as they are and tell
people why using it is a poor choice since COMPOUNDED 
single-component LOOKUP's are better, leaving the question
of why it is in the protocol as an exercise for the 
reader.

-----Original Message-----
From: Martin Pool [mailto:[email-masked]]
Sent: Tuesday, July 03, 2001 12:22 AM
To: Som
Cc: [email-masked]
Subject: Re: LOOKUP vs OPEN in NFSV4

On  2 Jul 2001, Som <[email-masked]> wrote:

> > The reason I think that LOOKUP and OPEN should return filehandles
> > has to do with the handling of multi-component lookups/opens.  If
> > a lookup encounters a symlink (or some other error situation),
> > you'd like to be able to restart the process at the point of
> > failure, which means that you need the filehandle where you
> > stopped.  A COMPOUNDEDed GETFH can't do it, since the error stops
> > the request, preventing the GETFH from being executed.  This makes
> > multi-component lookup/open kind of useless in any environment
> > where symlink might be encountered.

Actually I think a COMPOUNDed GETFH can do that perfectly well.  The
RFC has an example where you do

               PUTFH  (directory filehandle)
               LOOKUP "pub"
               GETFH
               LOOKUP "foo"
               GETFH
               LOOKUP "bar"
               GETFH

If any of these components are symlinks, then that operation will fail
and the client will have filehandles for all the previous directories.
This does mean you get filehandles for all previous directories as
well, but it's probably pretty useful to stick them into a cache
anyhow.

> So, now that we have a very good reason to return filehandle as part
> of OPEN4resok structure , applying the same consistency rule ( :-) )
> , why not return filehandles for every state modifying operations? I
> mean whenever client has to do GETFH, why not just return the result
> always in OPEN4resok strucutre?
> 
> Since NFSV4 is a very new protocol from earlier versions this change
> should be possible to do....this also saves considerable amount of
> wire traffic...is this too big a change for NFSV4?

I don't understand why you say it would generate more wire traffic.
Remember that the GETFH can be compounded with the OPEN.  Returning
filehandles on every call would add up to 132 bytes of filehandle data
to every single (state-modifying) response, whereas sending a GETFH
takes only one word more in the request, no more round trips, and
doesn't impose a burden on clients who don't need the information.  I
can imagine clients who keep track of files by name rather than fh,
and always send LOOKUP and never call GETFH.

-- 
Martin 
VA Linux Systems

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 3 Jul 2001 09:48:00 -0700

I agree that doing a LOOKUP and then doing a READ or
WRITE without doing an OPEN is Bad Thing.  The spec
seems to say as much, but then confuses the issue,
giving rise to the idea that there are two types of
filehandles (one produced by LOOKUP and the other
produced by OPEN), which I don't think is the case.

A handle returned by LOOKUP is not different in kind
from one returned by OPEN.  In fact, there is no 
problem caching a handle returned by OPEN by open 
and using it to satisfy a request that would otherwise
require a LOOKUP.  But the contrary case, taking a
handle gotten by LOOKUP and using it in place of doing
an OPEN does create difficulties, but not because the
handles are of different kinds, but because you lose
any guarantees the server might provide as to the handles'
continued validity (e.g. servers might prevent an open
file from being deleted).

This gets back to the "special" stateids and their (evil)
role in the protocol.  The state created by open is represented
by a stateid, so, in the interests of sanity, you should
only do READs and WRITEs to an open file using the handle
and stateid returned by the OPEN.  The spec talks about
how you should get the filehandle using OPEN, but it 
creates a mechanism, the special stateids, whose only real
use is in fact to allow you to do READs and WRITEs without
an OPEN, since if you did an OPEN, you would have both the
stateid and the correct filehandle, and wouldn't need the
special stateid's.

Also, as you indicate, the same issue applies to DAFS,
so I've added dafs-discussions to the cc-list.

-----Original Message-----
From: Som [mailto:[email-masked]]
Sent: Tuesday, July 03, 2001 12:46 AM
To: Martin Pool
Cc: [email-masked]
Subject: Re: LOOKUP vs OPEN in NFSV4

--- Martin Pool <[email-masked]> wrote:
> > 2. With the introduction of OPEN/CLOSE in NFSV4,
its
> a
> > stateful server...Doesn't it make a very complex
> NFSV4
> > implementation if the same server has to handle
> > stateless requests ( e.g. clients using LOOKUP
> > returned filehandle ) from the same NFSV4 clients?
> 
> Certainly having a stateful server is more complex
> than not.  Of
> course in earlier versions much of this complexity
> was present, but
> split into rpc.lockd and rpc.statd, which certainly
> made it more
> confused at the network level.  Once the server is
> stateful, I don't
> think adding support for lightweight stateless
> operations makes it
> much worse.  Servers which don't support locking can
> ignore much of
> this stuff.

I haven't see a network filesystem protocol where its
READ/WRITE routines can be called with two type of
filehandles ( one returned by LOOKUP style stateless
filehandle and the other one returned by OPEN style
stateful filehandle )....Probably NFSV4 derivative
DAFS is the only other protocol which has the same
feature...

I agree with the usage example given by you and
Dave...LOOKUP can be used to get file attributes etc
and that's a good example. But LOOKUP can be misused
to READ/WRITE files confusing the stateful semantics
of servers.

thanks, som.

=====
"Chess holds its master in its own bonds,
shackling the mind and brain so that the 
inner freedom of the very strongest must 
suffer"-Albert Einstein

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 8 Jul 2001 04:58:42 -0700

So this is a motivation for READ with stateid of
all zeroes.

That leaves WRITE with stateid of all zeroes and
READ with stateid of all ones to be accounted for.

-----Original Message-----
From: Brent [mailto:[email-masked]]
Sent: Saturday, July 07, 2001 6:19 PM
To: [email-masked]
Subject: Re: LOOKUP vs OPEN in NFSV4

"Noveck, Dave" wrote:
> Unfortunately, the spec currently allows you to
> do READ and WRITE with "special" stateids, all
> zeros and all ones.  It's not clear to me why
> these exist.  Unless there's a good reason for
> them, I would hope these get deleted.  I think
> this issue is on Spencer's to-do list.

Well, one of the goals of the protocol was that it
be possible to reach out over the Internet and grab a 
file from a server with a minimum of interaction.

For instance, with the HTTP protocol you can issue
a "GET" request and get a file in a single turnaround
(once the TCP connection is established).  Whereas with
NFS versions 2 and 3 you had that tacky MOUNT protocol
and even with WebNFS you still had to issue a separate
LOOKUP request to retrieve a filehandle before you could
issue a READ.

So, we're speculating that v4 might have clients that
are not concerned about cache consistency or exclusive
access, e.g. an image viewer or MP3 player.

Currently, it's possible for those clients to set up
a connection and generate a single compound request
that can lookup the file,  grab its attributes, and its
data, in a single request.

	Brent

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 8 Jul 2001 06:38:22 -0700

This response is restricted to the NFSv4-related stuff
so I'm taken dafs-discussions out of the cc-list.

Som wrote: 
> Noveck, Dave wrote:

<Stuff deleted>

> >> On NFSV4 front:
> >> How does NFSV4 server handles retransmission of
> >> non-idempotent requests by client? Will server
> >> erroneously re-execute the retransmitted
> >> non-idempotent request?
> > 
> > 
> > Let's separate the spec from the implementations.
> > 
> > Implementations maintain a response case organized
> > by RPC xid and sending IP address which can detect
> > retransmitted requests and avoid re-executing them.
> > However the spec does not require this or even mention
> > it.
> > 
> > The problem is that because such response caches are 
> > of bounded size, there is no guarantee that when the
> > retransmitted request is received, the corresponding
> > entry will still be in the response cache.  It may have
> > been displaced by a stream of other requests.  
> > 
> > Also, response caches are generally not maintained across 
> > server reboot so that a retranmission of a request that
> > was successfully executed and committed to stable storage,
> > will not be recognized as a retransmission if a server reboot
> > has intervened, and will be erroneously re-executed.
> > 
> > These weaknesses have led to some specific mechanisms to
> > deal with particularly pressing instances of this general
> > problem of non-idempotent re-execution.  In V3, a special
> > mechanism to implement exclusive open was included because
> > the response cache could not be relied upon.  Similarly, in
> > V4, locking requests have sequence numbers to prevent 
> > erroneous re-execution when the response cache is not
> > adequate to the purpose.
> > 
> > I think a possible line of development would be to use
> > sequence numbers generally and not just for specific
> > requests (operations actually in NFS-v4).  The problem
> > that this poses is avoiding over-restrictive serialization.
> > NFS-v4 limits serialization by making the unit of sequencing
> > something that inherently tends to issue sequenced requests,
> > i.e. a process.  The alternative to that is to allow streams
> > of sequenced requests to be freely created once you have a
> > safe method of doing that.  I think some of the ideas Neil 
> >
> > Brown was presenting were headed in that direction.   

> [som-start] I have not followed those discussions, may be I should 
> search thru' archive to find it.
>           If possible please post the URL here.

Here's the actual start of the thread:
http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2001/0013.html

But I was thinking mainly of the discussion in and following

http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2001/0064.html

> However, I am not able to understand the problems of defining sequence 
> numbers as part of NFS
> protocol itself....sequence numbers, if defined for the protocol will 
> become part of every NFS verbs...
> one should not attach any other concept of process etc to that...may be 
> I am missing something here!

For a minute, let's put aside what should and shouldn't be done.

I was describing what is in fact done in NFS-v4 which is to apply
sequence numbers to a limited set of operations.  NFS-v4 deals with 
the fact that sequencing of requests may cause undesirable 
serialization, by associating separate sequence streams with
individual lockowners (which typically map to client processes).
Since the typical single process only issues a single request (of
the class that are sequenced) at a time, the serialization that 
sequencing requires is typically not onerous, since the normal 
process wasn't prepared to have more than one of these in flight
at a time anyway.

I take it you believe that that is not the right way to do things.
I've come, rather late, to that same conclusion.  But any change
that basic is not appropriate for NFS-v4 at this stage.  The
protocol is far enough along that we are limited to minor changes
within the same basic framework.  Such a change would have to 
wait for an NFS-v5, although it might be shoehorned into a minor
version if one were prepared to be sufficiently creative/kludgy.

> Keeping RPC xid in response cache is definitely trying to get around the 
> problem and
> not solving it completely...

Yes, but the history of NFS has been that NFS has generally been pretty
successful in not solving problems completely while other protocols
that did do that haven't been as successful.  (e.g. DFS).

That fact has made people in the group somewhat skeptical about
general solutions to difficult problems, and therefore somewhat
conservative, but not totally, since NFS-v4 is a stateful
protocol, unlike previous versions.

In the case of sequencing, it was recognized that for the new
stateful requests, the response cache would not be adequate, so
sequencing was added to the specific operations that needed it,
and as far as I know, the idea of a general solution to the 
retransmission problem wasn't considered.
  
> So why not fix the problem where it needs to be fixed?

Because it's too late now for V4 and when it wasn't too late,
people thought of solving the general retransmission problem
as too difficult.  So we opted for something very specific to
the issue at hand.

The experience with DAFS's handling of this issue has caused me 
to reconsider this view and think about how you could use the
infrastructure that V4 already has to deal with the general
retranmssion issue in the NFS context.

So here's a sketch of how that might be done in a hypothetical
NFS-v5.

     Take SETCLIENTID and SETCLIENTID_CONFIRM out of COMPOUND
     and make them separate procedures.

     Normally SETCLIENTID would be the first operation performed
     at connection, although exceptions could be made for Brent's
     desire for NFS doing HTTP-like access.

     SETCLIENTID would contain information to allow a negotiattion 
     of the number of simultaneous COMPOUND requests that a client
     might have outstanding at a given time.

     Each COMPOUND request would contain a client-id, a stream 
     number, and a sequence number within that stream.  A client
     would not be allowed to issue sequence S(n)+1 for a stream
     until it had received the response for S(n).

     The server, when it received a COMPOUND would check the 
     sequence number for the requests as a whole, just as it does
     for sequenced opeations now.  If it's a retransmission, the
     old response can be sent.  If it's neither the next number nor
     a retransmission, it's an error.

     To accommodate purely read-only use, zero client-id, zero as
     the stream and sequence would be allowed for idempotent requests.

     The sequnce number would be removed from the operations that 
     currently contain it.

     The stateid being changed on lock/unlock/etc would no longer
     be needed.

     Mechanisms to renegotiate parameters, deal with deadlocks
     do to no streams avaliable, etc, can be provided as separate
     procedures.  Since these are infrequent, any necessary
     protection with respect to retranmission can be handled via
     extra round trips using CONFIRM mechanisms.

This is not fully fleshed out, but I think it could be.  The result
would be something that had DAFS-like do-it-exactly-once semantics
(although I would make the non-volatile aspect server-optional) but 
had an NFS-style (rather than a DAFS-style) relationship to its 
transports, in particular supporting unreliable transports.