Subject: RE: append writes (con't)

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 19 Mar 2001 09:23:30 -0800

Mike Eisler wrote:
> Your advocacy of safe shared APPEND may be something that customers ask
> for, and it's probably the same set of customers that want true cache
> coherency; you know, the ones that want to mmap a the same file to
> multiple clients, and use spinlocks on the mapped memory. I tend to
> refer those customers to IBM/TransARC for a suitable product.

Is that because IBM/TransARC can support that, or because you'd like
to give them a little grief :-)

Can they really support what is essentially DSM?  That would be kind 
of neat, in a horrifying sort of way.

> >don't exist, but I don't know of them.  I'm not sure whether their 
> >existence or number is relevant.  The reasons that something is or isn't
> >in a protocol should have some ultimate reference to those that are to
> >use it but the relation is often not very direct.  
> 
> Agreed it is indirect. And aside from those ignorant of syslog
> capabilities, there are there customers asking for reliable multi-client
> logging over NFS?

No, but customers often want a network of computers to be more like
a single computer.  That's part of why NFS exists.

Some would like it to be exactly like a single computer.  I've worked
on the no-remote-access-to-memory SSI problem enough not to want to 
work it on anymore.

I think we should strive to make it as easy as we *reasonably* can
to take something written for multiple processes on a single machine
and run it on multiple machines.  Saying it is a simple matter of 
writing an open-source library or a simple matter of using a 
different set of routines only goes so far.  Users would prefer 
to have it just work, and I can't say I blame them.

We're really discussing what is reasonable to do.  I see a couple of
categories of things that are unreasonable:

     Things that are esoteric in the sense that they are rarely used
     even locally.  I would argue that most cases of read-write
     atomicity fall into this category, despite the fact that this
     behavior is specified by POSIX.

     Things that are just too hard to do given all the constraints
     on NFS implementations.  Anything that requires distributed
     shared memory support is certainly an example.

Clearly append writes don't fit in the first category.  The experience
in NFS Classic (ie v2 and v3) has been that they fit in the second
one, despite their apparent simplicity.

Because V4 has invested in a lot of stateful infrastructure, append
writes may be a lot more doable than they have been in the past.  We
shouldn't tie ourselves in knots trying to do this, since people have
learned to live without it, but if it can be reasonably done, I don't see
why we shouldn't allow it.  It isn't esoteric.

There's been a lot of discussion of this and related issues in terms of
"cache coherency".  I don't see cache coherency per se as the essence of
the problem.  You want to look at the semantics you support.  If you put 
aside the read-write atomicity semantic, as an implementation artefact,
then you are left with a number of ways that multiple processes writing
to a given file may co-ordinate their work, whether they are located
on the same client or diffrerent clients.  Locking is one way and V4's
approach is to support that and define caching so that caching does not
interfere with guarantees that locking would normally provide, without
trying to make the caches coherent.  Another way that applications have
is simply to append to the same file.  That works fine with two processes
on the same client and turns the file into garbage if the process are
on different clients.  Having it just work would certainly be nicer.

---

From: Trond Myklebust <[email-masked]>
Date: 19 Mar 2001 20:21:43 +0100

>>>>> " " == Dave Noveck <Noveck> writes:

     > There's been a lot of discussion of this and related issues in
     > terms of "cache coherency".  I don't see cache coherency per se
     > as the essence of the problem.  You want to look at the
     > semantics you support.  If you put aside the read-write
     > atomicity semantic, as an implementation artefact, then you are
     > left with a number of ways that multiple processes writing to a
     > given file may co-ordinate their work, whether they are located
     > on the same client or diffrerent clients.  Locking is one way
     > and V4's approach is to support that and define caching so that
     > caching does not interfere with guarantees that locking would
     > normally provide, without trying to make the caches coherent.
     > Another way that applications have is simply to append to the
     > same file.  That works fine with two processes on the same
     > client and turns the file into garbage if the process are on
     > different clients.  Having it just work would certainly be
     > nicer.

The cache coherency problem *is* the essence of the problem when you
try to mix read() and open(O_APPEND).

Say I open my file O_APPEND|O_RDWR, or say I start a process to run
'tail -f' on my logfile. How do I ensure that my client is in sync
with the server?
After all, even 'syslog' likes to compress its logs using the famous
'last message repeated N times'...

Cheers,
    Trond