Subject: RE: LOOKUP vs OPEN in NFSV4

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 6 Jul 2001 21:19:12 -0700

Som wrote: 
> --- "Noveck, Dave" <[email-masked]> wrote:
> > Som wrote:
> > Another related issue is whether some of the infra-
> > structure that is used for stateful requests might
> > have
> > been used more universally.  For example, sequence
> > numbers and lockowners (which would have to be
> > generalized 
> > to be general request sequencing objects) might have
> > been 
> > made universal and done on a request level, rather
> > than on
> > an operation basis.  DAFS is an NFS-v4 variant that
> > has 
> > gone a considerable distance in this direction in
> > that
> > the requirement for specific stateful-request
> > sequencing
> > has been obviated by requring more quarantees from
> > the
> > transport and using sequence numbers in a general
> > header
> > for all requests, allowing the response cache to be
> > brought
> > within the protocol and made dependable.  My feeling
> > is that
> > this has been a successful approach.  One of the
> > interesting
> > questions for the future of NFS is whether similar
> > approaches
> > might be applied in the less constrained wider-area
> > environment,
> > but that would be something that would only be worth
> > considering for NFS-v5 and that's a long way off, if
> > it ever 
> > happens.	
> > 
> 
> Yes, request-id part of the DAFS protocol header is
> definitely something useful and several features can
> be developed using this. One of them could be ( in
> future ), DAFS working over unreliable media ( like
> WAN ). One possible DAFS enhancement: ability to
> retransmit non-idempotent requests ( use the same
> request-id so server knows its a retransmit ).

The issue is not reliability of the media.  DAFS
can be implemented on unreliable media (and no medium
is totally reliable).  The issue is the reliability
of the transport that DAFS plugs into.  So far DAFS
has been specified only for reliable connection-
oriented transports: VI and IB reliable connections.
There has been some discussion of adapting DAFS to
IB reliable datagrams, but so far as I know there
hasn't been any for transports that do not provide
reliabiable transmission.  As far as I am concerned,
this is a good thing.  

I believe it is worthwhile considering what might
be necessary to adapt DAFS to a broader set of 
environments but I would probably stop somewhere
around campus-wide or metropolitan area networks,
rather than going all the way to WAN's.  It is
tough try to be all things to all people and I think
has DAFS has benfited by not trying.  The main issues 
in adapting to networks larger than a single big room 
are security and tolerance of higher latency.  As far 
as reliability goes, making this the job of the 
transport is the right way to go.

The way that DAFS forms its request id's (stream id and
sequence) would allow you to do safe retransmission 
detection (as opposed to the NFS case discussed below),
but I just don't see the point of doing that, unless
unreliable transports have some enormous benefit to
compensate for the greater difficulties for the protocol.

> On NFSV4 front:
> How does NFSV4 server handles retransmission of
> non-idempotent requests by client? Will server
> erroneously re-execute the retransmitted
> non-idempotent request?

Let's separate the spec from the implementations.

Implementations maintain a response case organized
by RPC xid and sending IP address which can detect
retransmitted requests and avoid re-executing them.
However the spec does not require this or even mention
it.

The problem is that because such response caches are 
of bounded size, there is no guarantee that when the
retransmitted request is received, the corresponding
entry will still be in the response cache.  It may have
been displaced by a stream of other requests.  

Also, response caches are generally not maintained across 
server reboot so that a retranmission of a request that
was successfully executed and committed to stable storage,
will not be recognized as a retransmission if a server reboot
has intervened, and will be erroneously re-executed.

These weaknesses have led to some specific mechanisms to
deal with particularly pressing instances of this general
problem of non-idempotent re-execution.  In V3, a special
mechanism to implement exclusive open was included because
the response cache could not be relied upon.  Similarly, in
V4, locking requests have sequence numbers to prevent 
erroneous re-execution when the response cache is not
adequate to the purpose.

I think a possible line of development would be to use
sequence numbers generally and not just for specific
requests (operations actually in NFS-v4).  The problem
that this poses is avoiding over-restrictive serialization.
NFS-v4 limits serialization by making the unit of sequencing
something that inherently tends to issue sequenced requests,
i.e. a process.  The alternative to that is to allow streams
of sequenced requests to be freely created once you have a
safe method of doing that.  I think some of the ideas Neil 
Brown was presenting were headed in that direction.   

> thanks, som.

> Reference discussion:
> Something similar to this has been discussed in DAFS
> mailing list at
> http://groups.yahoo.com/group/dafs-discussions/message/248?threaded=1

---

From: Somenath Bandyopadhyay <[email-masked]>
Date: Fri, 06 Jul 2001 23:57:41 -0700

Dave, Thanks for your good comments.
          This time my comments are marked with [som].

> transport is the right way to go.

[som-start]

1. Its just my feeling that if DAFS succeeds in a big way in "local file 
sharing"
environment, and next in broader set of environemnts ( like MAN etc ), 
users
will like to extend it such a way that it can be used everywhere. So 
keeping the
specification flexible for such future use ( without complicating 
matters a lot for
the spec ) probably good for DAFS. Anyway, its just a feeling, I won't 
go further on
this.

2. Yes you are right about DAFS and reliable transport ( thanks for 
correcting me
on this, I should have used reliable transport instead of reliable media 
). But I don't
understand the usage of checksum in DAFS headers when it
assumes reliable transport. Only meaningful reason I heard so far is for 
debugging purpose. In that
case, at least keep checksums in optional part of the header ( if its 
not there, create one )...what do
you feel about this alternate header proposal?

http://groups.yahoo.com/group/dafs-discussions/message/260?threaded=1

[som-end]

> Brown was presenting were headed in that direction.   

[som-start] I have not followed those discussions, may be I should 
search thru' archive to find it.
          If possible please post the URL here.
However, I am not able to understand the problems of defining sequence 
numbers as part of NFS
protocol itself....sequence numbers, if defined for the protocol will 
become part of every NFS verbs...
one should not attach any other concept of process etc to that...may be 
I am missing something here!

Keeping RPC xid in response cache is definitely trying to get around the 
problem and
not solving it completely...
So why not fix the problem where it needs to be fixed?

[som-end]

thanks, som.