Subject: Re: [nfsv4] Sleepy clients

From: Mike Eisler <[email-masked]>
Date: Fri, 26 Aug 2005 08:32:01 -0700 (PDT)

> -----Original Message-----
> From: Jim Rees [mailto:[email-masked]] 
> Sent: Friday, August 26, 2005 6:10 AM
> To: [email-masked]
> Subject: Re: [nfsv4] Sleepy clients 
> 
> 
>   OP_ZZZ seems like the top of the very high slippery slop of a client
>   working in a 'disconnected' mode... There's a nasty tar pit at the
>   bottom of that slop..
> 
> I disagree.  I've been running disconnected afs on my laptop 
> for ten years
> and it makes me very happy.

I interpreted the tar pit comment as: "figure out what needs to
change (if anything), in order to make disconnected mode work better
in NFSv4, and then make a more formal proposal". Not a bad idea.

> However, I think it's important to put the burden of 
> disconnection on the
> client, not the server.  There should be no OP_ZZZ.  If the 
> client wants to

Servers should defer revocation of state after the lease expires until there
is a conflict regardless. OP_ZZZ is hint to those servers that don't, that maybe they
should. Maybe for the use case Brent C has, adding a hint operation that says:
"I tend to go to sleep a lot", would be better that an OP_ZZZ. Less chatty.

> disconnect, it must be prepared to suffer the consequences.  
> In most cases
> that will mean it wants to return all resources to the server and be
> prepared to reclaim them on reconnection.

It has to be prepared for the consequences with or without OP_ZZZ, because
the server certainly isn't going to hold off conflicts indefintely.
Indeed, with OP_ZZZ, just the opposite will happen: I can
see servers interpreting OP_ZZZ to say: "if I get a conflict, I'm not even
going to bother with recall to the zombie client; 
I'm going to revoke and grant the conflicting
state to the live clients." NFSv4 service is for living, not the undead. :)

So if we go the OP_ZZZ route, there may need to be an OP_AWAKE. But
OP_RENEW might be a good enought OP_AWAKE.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Jim Rees <[email-masked]>
Date: Fri, 26 Aug 2005 13:38:34 -0400

I believe the current protocol provides everything you need.  The main
things are ability for the client to return all state when it disconnects,
ability to reclaim state on reconnection, and ability to detect write
conflicts on reconnection.

That, and a sunny, optimistic outlook on life.

---

From: [email-masked]
Date: Sat, 27 Aug 2005 19:57:11 -0400 (EDT)

> That, and a sunny, optimistic outlook on life.

I guess I'm a little more pessimistic:-). For NFSv3, due to the stateless
server, there was no state to reclaim and, as such, no concern w.r.t.
being able to reclaim that state. For NFSv4, if another client has
acquired conflicting state (such as Open_share_Deny_both on a file) while the
client is asleep, then the client can't reclaim the state when it reconnects.
This seems to me to be a step backward from NFSv2,3 w.r.t. reclaiming state
upon reconnection.

Also, it seems to me that sleepy/disconnected clients might want to be
able to do more than simply detect write conflicts. A couple of thoughts:
1 - It seems to me that a client might want to prevent other clients from
    performing conflicting writes while it is asleep. (Open_share_deny_both
    would do that, if only the sleepy client could maintain the lease.)
OR
2 - A client might want to be able to modify a file on the server when it wakes
    up without destroying other changes done by other clients while it was
    asleep. In other words, versioning. (I believe this is what MIT's Cedar
    file system did and it was what old DEC OS's like VMS did. In case you
    aren't familiar with the concept, good old VMS would create a new version
    of a file each time it was modified. If you wanted to open version NNN,
    you opened "foo.dat;NNN". If you opened "foo.dat" you opened the newest
    version.) It is a simple way around the problem of conflicting writes,
    and allows the application/user to merge changes later.

Just my $0.00 worth. Have a good weekend, rick

---

From: Jim Rees <[email-masked]>
Date: Sat, 27 Aug 2005 21:59:31 -0400

That seems very anti-social to me.  Write conflicts are rare, and when they
do happen, I'd rather give priority to the connected clients.  How do you
know the disconnected client will ever come back?  What will you do if it
doesn't?

  2 - A client might want to be able to modify a file on the server when it wakes
      up without destroying other changes done by other clients while it was
      asleep. In other words, versioning.

It would be silly to just overwrite.  The disconnected client must detect
conflicts when it reconnects.  You don't need versions at the server, just
at the client, and you only need two, maybe three.  A linear versioning
system like VMS isn't really right, because the two versions both diverge in
parallel from a common ancestor.  Disconnected afs calls them by the
original name and the original name .something then lets the user decide
which to keep.

---

From: [email-masked]
Date: Mon, 29 Aug 2005 12:40:15 -0400 (EDT)

[Tom wrote]
> >What might be considered a more social variant would not stop the connected
> >clients from having access, but would put a "flashing red message" on their
> >screen "Change this file at your peril, since Fred has it on his laptop".
> 
> Isn't this an application-specific thing? For example, MS Office
> does exactly this, by writing your name and edit status into the
> file you're working on. Subsequent opens find this "metadata"
> and do just what you describe.

Yes, the "flashing red message", etc. is obviously application specific.
My assumption was that it would be the result of a "lock", similar to
an Open/share that would indicate to the application that it can't/shouldn't
access the file for now. (As for the "Fred" part, I was thinking vaguely along
the lines of the reply to NFS4ERR_DENIED for Lock, that would indicate who holds
the conflicting lock.)

It reminds me of all the Unix apps. that do strange contorted things with
"lock files" because the apps. don't use/trust file locking primitives.

> In other words, do we really want the filesystem doing this?

I think it "might" be nice if the underlying file system allowed some
kind of lock that would remain while the client is asleep, so that apps
or middleware on the clients can do this.

[Dave wrote]
> We could do such a thing within NFS, but do we really want to?
> My perspective that this sort of this is best left to
> a middleware solution. In fact, in VMS the versioning was
> handled the the RMS middleware, not the Files-11 filesystem.
> If you did straight SYS$QIO calls you had no explicit semantics
> of versions, it was just a weird part of the name.

[my stuff clipped]
> Disconnected operation is hard, often the merging and managing
> of data on reconnect is best left to middleware that has
> knowledge about file formats, logical records, or other bits
> that allow intelligent merging.

The merging is definitely application specific for the reasons you mention.
However, I'm not quite convinced w.r.t. the versioning itself. If several
clients try to write a file back to a server concurrently and the client
side is going to handle versioning, it seems they:
1 - Must recognize that other clients are also doing so and know to
    create a new/different version, instead of just re-writing the file.
    (Open/Deny_write might do the job here, assuming this is a fresh, after
     reconnect, open?)
2 - Must create the new version is such a way that no other client will
    create the same new version. (Sounds a bit like exclusive open?)
and handle it in an application (or client middleware) specific way...

OR

The clients Open/Write/WithVersioning and the "middleware" under NFS
on the server could create multiple versions, as required.

I honestly have no idea which of the above makes more sense? rick

---

From: [email-masked]
Date: Mon, 29 Aug 2005 14:54:13 -0400 (EDT)

[I wrote]
>   "lock files" because the apps. don't use/trust file locking primitives.
[Jim writes]
> Some of the contortions were only required in the old days before O_EXCL.

Sorry for not explaining it. I meant the above to be an analogy to what
can happen if application programmers have to come of with solutions to
things that the underlying file system doesn't handle.

[Jim writes]
> I disagree.  This would require changes in all clients, not just
> disconnected clients, and it's an unreasonable burden to impose on them.
> Disconnected clients should retain their own copy of the original file if
> they think they will need it when they reconnect.

Here, I was referring to multiple clients that have local copies of the
same file, that are writing them back to the server after reconnection.

Hope this clarifies what I was thinking, rick

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 29 Aug 2005 15:08:37 -0400

> One thing I'm not sure about is how an NFSv4 client detects that it
lost a
> lock when it re-acquires its state. 

It can find out if it's lock was lost.  If the stateid is still valid,
then
the lock was not lost.  If it isn't, then either there was a conflict
(which caused the lock to be lost) or the server dropped the lock 
for another reason (either it always does or there was a resource issue)
in which case, you have to assume that it was at least possible that 
someone else got an inconsistent lock. 

> You can infer that if the change
> attribute is different then the file was modified, but there is a
subtle
> difference between having a file change and having someone else break
your
> lock.  It may not matter.

If the guy with the presentation has deny-all, then the lock may have
been
broken because of somebody reading.  He might be quite willing to go on 
with his updates as long as nobody changed the file.  The difference
does
matter but most people, for opens, are really concerned with whether the
file changed.

For byte-range locks, you have to assume that if stateid is lost, you
have lost your lock.

-----Original Message-----
From: Brent Welch [mailto:[email-masked]] 
Sent: Monday, August 29, 2005 2:48 PM
To: Jim Rees
Cc: [email-masked]
Subject: Re: [nfsv4] Sleepy clients 

>>>Jim Rees said:
 > 
 >   It reminds me of all the Unix apps. that do strange contorted
things 
with
 > they think they will need it when they reconnect.

Yup - let's make sure we know what the AFS disconnected operation effort
has
learned in the last like 10 years of working on this.  The server just
has to provide enough help for clients to detect conflicts when they
reconnect.  This is the change attribute.  And the server has to be
forgiving with respect to lease revocation, which it sounds like most
servers are; they only revoke on conflict.  I think we're done here.

One thing I'm not sure about is how an NFSv4 client detects that it lost
a
lock when it re-acquires its state.  You can infer that if the change
attribute is different then the file was modified, but there is a subtle
difference between having a file change and having someone else break
your
lock.  It may not matter.

I suggest we see how far we can get with these two basic features
(change attribute and forgiving lease expirations) before we add
any more complexity to the server to support disconnected clients.

In other words, I have no mercy for the marketing guy with the
whole file lock that is on a flight to Europe.  His windoz app
will help him "Save As" the file and his colleagues can sort
things out later.  As a file server, I've done my duty if I've
allowed his client to detect the problem.

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]