Subject: [nfsv4] follow-up of xattr discussion

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Thu, 23 Jul 2015 11:42:04 +0200 (CEST)

Hi all, 

after watching IETF93 NFS session, I want to support XATTR activity
with use cases that we have. Typical complicated storage system has
much broader set ot attributes which can't be exposed through standard
attributes or not generic enough to enforce it. Never the less, we need
some mechanisms to set and get them.

Currently this is implemented as 'magic' files, (like a /proc file system),
where reading a special file gets the value and writing into the file updates
it. Some example:

file checksum:
$ cat '.(get)(big30g)(checksum)' 
ADLER32:e3630001
$

Get file locality on the system (DISK/TAPE/LOST)
$ cat '.(get)(big30g)(locality)'
ONLINE
$

List user defined attrubutes
$ cat '.(tags)()'          
.(tag)(OSMPriority)
.(tag)(OSMTemplate)
.(tag)(sGroup)
$

$ cat '.(tag)(sGroup)'
generated
$

One can say we use our file system as a data base. But our file system is a database and
we want to expose data base nature to the users.

The file 'magic' file semantics has down sides - this files not included in the listing,
have weird cache behaviour and nfs overhead to read/write them OPEN/LOOKUP/LAYOUTGET/READ/CLOSE...

Tigran.

---

From: Christoph Hellwig <[email-masked]>
Date: Thu, 23 Jul 2015 11:58:18 +0200

Hi Tigran,

magic values is exactly what user xattrs are NOT.  So it's an entirely
different use case than what Marc presented.

We also highly discourage this magic for Linux xattr implementations.

---

From: Benny Halevy <[email-masked]>
Date: Thu, 23 Jul 2015 13:30:38 +0300

If I understand Tigran correctly, he looks forward to use XATTRS over NFS
to store the application level metadata they currently store in their
"magic" files.

I think it's a valid use case since there are no functional side effects to
getting or setting values stored in the extended attributes although they
are not entirely semantic free; e.g. storage locality that is a system
level attribute whose content is generated by the server.

Benny
On Jul 23, 2015 12:58, "Christoph Hellwig" <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Thu, 23 Jul 2015 12:51:27 +0200 (CEST)

----- Original Message -----
> From: "Benny Halevy" <[email-masked]>
> To: "Christoph Hellwig" <[email-masked]>
> Cc: [email-masked], "Tigran Mkrtchyan" <[email-masked]>
> Sent: Thursday, July 23, 2015 12:30:38 PM
> Subject: Re: [nfsv4] follow-up of xattr discussion

> to store the application level metadata they currently store in their

Yes, this is what I meant.

> level attribute whose content is generated by the server.

Of cosre some of the attributes are system specific and some of them 'free-form'.
If we can get a standard notation of a quality-of-storage, then we may try to
push it into a standard attributes. We are currently working on defining this
QoS.

But user defined attributes are required as well. Scientific data always has some
metadata associated with it. On a 'classic' storage system, the metadata will be
'hard coded' into the filename:

mc10_7TeV.105200.T1_McAtNlo_Jimmy.merge.NTUP_SUSY.e598_s933_s946_r1831_r1700_p403_tid254878_00

Having this metadata stored in the filesystem (and make it possible to search data based on it)
will change the way how people do the work.

Tigran.

---

From: [email-masked] (J. Bruce Fields)
Date: Thu, 23 Jul 2015 11:38:41 -0400

> Yes, this is what I meant.

If I understand correctly, your very first example uses an xattr read to
calculate a checksum on the fly.  That's definitely *not* just passive
xattr storage, and I don't think it's going to work with the current
draft.

Almost every time xattrs come up there are people who say "I want
xattrs, because I can use them as a free-form interface to my
filesystem".  The same thing's happened to me elsewhere--no matter how
clear I try to be that we're only doing user xattrs, people come back to
me with "we'd love that!  We could use it to set striping hints!" (or
whatever).

Is there any way to eliminate this confusion?  Or do we
need to give up and figure out what's wanted and do it somehow?

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 23 Jul 2015 12:09:41 -0400

> me with "we'd love that!  We could use it to set striping hints!" (or

Agreed. It's popular because it is the easiest way to avoid the IETF process.

> need to give up and figure out what's wanted and do it somehow?

Once we start adding interfaces that provide arbitrary functionality,
then we're effectively giving up the ability to spec the NFSv4
protocol within the IETF. If people want the arbitrary functionality,
then please write a side band protocol for it, instead of turning the
NFSv4 protocol into something that cannot be standardised.

---

From: David Noveck <[email-masked]>
Date: Thu, 23 Jul 2015 13:15:22 -0400

Benny wrote:

> That's definitely *not* just passive

In fact, the draft says:

> It requires that the attribute key/value MUST not be interpreted by the NFS

> clients and servers

which sound to me like such uses for not allowed.  I'll be reviewing this
document in
a bit and will propose some toughening up of the language.

> NFSv4 protocol into something that cannot be standardised.

Agree!

As we think about moving xattr forward, we have to deal with the question of
which protocol version these attributes and operations are to be added to.
Until
we define NFSv4.2 as extensible, the only choice would be trying to do an
NFSv4.3,
which almost certainly years away.

So it look to me that we need to do two things beyond arriving at a
consensus that this
is basically a good thing to do.

   - Getting draft-ietf-nfsv4-minorversion2 through IESG review.  I assume
   Tom and Spencer are following up on that.
   - Getting draft-ietf-nfsv4-versioning through IESG review.  As of now,
   it's a ways away from that.

Because of some email address difficulties, I did not get
draft-ietf-versioning-01 out in time for it to be discussed in
Prague.  I still urge people to RTFD (i.e. Read That Fantastic Document :-) and
comment on the list.  I don't believe
our current versioning model is workable and if people don't like what I've
done, please tell me what you'd like changed
or publish your own approach.  I hope we can move forward with significant
changes to our approach to version
management.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 23 Jul 2015 14:45:47 -0400

On Thu, Jul 23, 2015 at 01:15:22PM -0400, David Noveck wrote:
> which sound to me like such uses for not allowed.  I'll be reviewing
> this document in a bit and will propose some toughening up of the
> language.

One of the practical difference is the caching.  A lot of the
filesystem-specific attributes look incompatible with client caching.

Looking at the "caching" section: it does say clients may cache reads,
and that the change attribute can be depended on to validate that cache,
but:

	https://tools.ietf.org/html/draft-naik-nfsv4-xattrs-02#section-6.2.2

	Such caching is write through in that modification to xattrs is
	always done by means of requests to the server and should not be
	only done locally. Due to the relative infrequency of xattr
	updates, it is suggested that all changes be propagated
	synchronously. The client MUST NOT maintain a cache of modified
	xattrs.

What's the rationale for that MUST NOT?

--b.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 23 Jul 2015 16:48:35 -0400

On Jul 23, 2015, at 1:15 PM, David Noveck <[email-masked]> wrote:

> So it look to me that we need to do two things beyond arriving at a consensus that this
> 	• Getting draft-ietf-nfsv4-minorversion2 through IESG review.  I assume Tom and Spencer are following up on that.
> 	• Getting draft-ietf-nfsv4-versioning through IESG review.  As of now, it's a ways away from that. 
> Because of some email address difficulties, I did not get draft-ietf-versioning-01 out in time for it to be discussed in 
> Prague.  I still urge people to RTFD (i.e. Read That Fantastic Document :-) and comment on the list.  I don't believe
> our current versioning model is workable and if people don't like what I've done, please tell me what you'd like changed
> or publish your own approach.  I hope we can move forward with significant changes to our approach to version 
> management.

This seems to me like a reasonable plan.

--
Chuck Lever

---

From: "Manoj Naik" <[email-masked]>
Date: Fri, 31 Jul 2015 08:22:39 -0700

"nfsv4" <[email-masked]> wrote on 07/23/2015 11:45:47 AM:

> From: "J. Bruce Fields" <[email-masked]>
> To: David Noveck <[email-masked]>
> Cc: Christoph Hellwig <[email-masked]>, "[email-masked]" <[email-masked]>
> Date: 07/23/2015 11:46 AM
> Sent by: "nfsv4" <[email-masked]>
> 

Same as the one for all other attributes (except size, time_modify, 
change): see https://tools.ietf.org/html/rfc5661#section-10.6

Manoj.

---

From: David Noveck <[email-masked]>
Date: Fri, 31 Jul 2015 14:20:27 -0400

The text in question reads as follows:

Other than those three attributes, the client MUST NOT maintain a
cache of modified attributes. Instead, attribute changes are
immediately sent to the server.

When those two sentences are read together, it seems to me that the
intention is/was to forbid writeback caching of attributes. With just the
first of them present, it might lead someone to suppose that a writethrough
cache of modified attributes is to be forbidden as well. I don't think that
is the intention.