Subject: RE: append writes (con't)

From: Mike Eisler <[email-masked]>
Date: Fri, 16 Mar 2001 14:37:59 -0800 (PST)

>> So there must be a customer somewhere that wants this if engineers are
>> spending their time on this difficult problem.  It would be nice to
>> know about the customers and what they are attempting to accomplish.
>
>People write log files for many reasons.  All OS's that I know of
>have an append mode for this very purpose.  It would be very inconvenient
>for each process to have it's own log due to the volume of files and
>the fact that you would have to post-process to get any coherent
>view.  This is why OS's typically allow multiple processes to write,
>in append mode, to the same file with a coherent result, on local
>file systems.  What are the customers that use this feature trying to
>accomplish?  Just get their work done without wasting time doing 
>something complex.  They prefer to have a facility that makes their
>life simple.

Some OS's have a feature like syslog.

>In a networked environment the same hold true.  To have to log 
>application data based on the machine on which the process executed
>is undesirable.  You can program around this and some people have
>but it is clearly not desirable and anybody who has to kludge
>around this would certainly choose to spend his time otherwise.

In a networked environemtn one can configure syslog to forward log requests
from one node to another. This deals with the issues of centralization, and
makes the issues of multi-client append moot. I.e. there is only one
syslog daemon writing to a file, so atomic append is irrelevant, whether
the file is local or NFS accessed.

This by the way is similar to the one of the old arguments for device
access via NFS, some folks wanted to access remote tape drives. Today
we have the rmt and ndmp protocols to address this. Others wanted
to access remote displays, and X was invented for that.

If there were a requirement for other needs for networked append, I
suspect protocols tailor to the application would be invented.

	-mre

---

From: RJ Atkinson <[email-masked]>
Date: Fri, 16 Mar 2001 18:29:33 -0500

At 17:16 16/03/01, Noveck, Dave wrote:

>have an append mode for this very purpose.  

Dave,

        Most server operations folks I know sleep Just Fine
at night by using BSD SYSLOG to handle log file transactions
over the network.  No, SYSLOG isn't perfect.  Once in a blue
moon, a network weirdness will mean that SYSLOG will lose
a log transaction or truncate one.  People are not terribly
upset about this AFAIK.

        Folks who want a transaction database system with lots
of 9s of reliability generally buy one and use something such 
as SQLnet to move those transactions reliably over the network.

        I don't see a big NFS feature gap any way it is sliced.  
I must be missing something here.

Ran
the token NFS user participant

---

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 17 Mar 2001 12:17:31 -0800

Spencer Shepler wrote: 
> On Fri, Noveck, Dave wrote:
> >
> So you don't have specific instances of customers that have lamented
> about the lack of append mode writes on a distributed filesystem?

No.  That's true of a lot of a lot of what's in V4.

I do know of customers that have lamented the lack of security (over-the-
wire and ACL's) and poor locking in previous versions of NFS.

On the other hand, I don't know of customers that are asking for COMPOUND,
the new attribute model, or delegations in NFS.  I'm not saying that they
don't exist, but I don't know of them.  I'm not sure whether their 
existence or number is relevant.  The reasons that something is or isn't
in a protocol should have some ultimate reference to those that are to
use it but the relation is often not very direct.  

> So let's talk about the client APIs for a moment.  For this to work,
> the NFS client agent will need to force each write() to the server for
> the append.  That is different than current behavior and it may be
> considered too much of a performance burden for applications that are
> not writing to a log that is shared amongst various systems.  So how
> does the application indicate its true desire about O_APPEND.

I don't think its a question of the applications desire but rather
the degree of sharing.  The server can deal with the performance
issue by granting delegations in the single-client case.  Thus would 
be a case in which delegations, while not strictly required, would
be needed to maintain performance in this particular case.

So, if v4.1 were to do this, I think the answer would be to say that 
the server SHOULD grant a write delegation if an OPEN for APPEND is done
and the file is not opened by any other client. 

> It seems much easier to write an open-source library that provides a
> distributed logging mechanism and corresponding aggregation
> methodology.  Maybe something like the syslog service.

If someone else does it, it's certainly easier. :-)

Since I've never actually done such an open-source library, I can't 
estimate its absolute level of difficulty.  My feeling is that if 
you want to do something useful and general you wind up dealing with 
many of the difficult naming and failure and recovery issues that 
distributed file systems have to do anyway.

Anyway, it's my impression that some of the recent infrastructural
devlopments in V4 make this feature attainable.  But if there is no
consensus that it's desirable, it isn't worth persuing.

---

From: Mike Eisler <[email-masked]>
Date: Sun, 18 Mar 2001 18:17:07 -0800 (PST)

>On the other hand, I don't know of customers that are asking for COMPOUND,

Customer's don't ask for compound, but they do ask for lower latency
file operations, especially those that have asked for ways to use NFS through
WANs.

>the new attribute model, or delegations in NFS.  I'm not saying that they

Customer's don't ask for an extensible attribute model, but they do
ask for more attributes, ACLs and PC-centric attributes for example.

As for delegations, they don't ask for them, but they do ask for
faster file operations, especially locking.

Your advocacy of safe shared APPEND may be something that customers ask
for, and it's probably the same set of customers that want true cache
coherency; you know, the ones that want to mmap a the same file to
multiple clients, and use spinlocks on the mapped memory. I tend to
refer those customers to IBM/TransARC for a suitable product.

>use it but the relation is often not very direct.  

Agreed it is indirect. And aside from those ignorant of syslog
capabilities, there are there customers asking for reliable multi-client
logging over NFS?

	-mre

---

From: Mike Eisler <[email-masked]>
Date: Mon, 19 Mar 2001 11:22:14 -0800 (PST)

> Mike Eisler wrote:
> Is that because IBM/TransARC can support that, or because you'd like
> to give them a little grief :-)

I believe the former to be true. As to whether more customers for IBM
is a problem, you'd have to ask Mr. Gerstner.

> Can they really support what is essentially DSM?  That would be kind 
> of neat, in a horrifying sort of way.

If they handle write sharing by acquiring tokens, and ping ponging
tokens between competing clients, yes of course. My understanding is
that they do just that; anything less isn't cache coherency.
 
> Because V4 has invested in a lot of stateful infrastructure, append
> writes may be a lot more doable than they have been in the past.  We
> shouldn't tie ourselves in knots trying to do this, since people have
> learned to live without it, but if it can be reasonably done, I don't see
> why we shouldn't allow it.  It isn't esoteric.

If it can be reasonably done. 

	-mre

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 19 Mar 2001 11:58:04 -0800

> is a problem, you'd have to ask Mr. Gerstner.

I suppose it depends on the customer :-)

> that they do just that; anything less isn't cache coherency.

No but you can have cache coherency through tokens and not have nearly
enough to allow mapping of the file with spinlocks in the mapped 
memory.

The case in which people read and write a shared distributed file
using the read and write calls is relatively well behaved as these
things go.  You know what you have to lock and only one region can
be locked at a time.  Things can go slowly when sharing, including
false sharing, goes against you, but you make progress.  If there 
is too much token ping-ponging, you call fall back to sending all
requests to the server.

When you add memory mapping, things get interesting in that you
have to be really tightly integrated with the VM and get tokens
based on the user access (read or write) to memory.

When you add spinlocks the fun just multiplies.  A user does a 
test-and-set so he gets exclusive access to the page and successfully
gets his lock.  Now, he does some stuff, and some other process
on a another client does a write which requires write access to
that page.  The first process loses access to the page, which is
exactly correct, except that if the second process should now
try to do a test-and-set on the spin lock held by the first
process it will fail repeatedly, and the first process is going 
have trouble getting access to the page to release his lock.
So to make this work you probably need some tight integration
with scheduling to notice and prevent (or quickly correct) those
kind of unpleasant situations.

That's why I think that Mr. Gerstner might be less than grateful
for the referral.