Subject: [nfsv4] Linux IMA on NFS follow-up

From: Chuck Lever <[email-masked]>
Date: Tue, 28 Aug 2018 14:38:00 -0400

Hi Spencer D. !

During our meeting in Montreal last month, I presented a backgrounder
on supporting Linux IMA (file integrity measurement) on NFS.

A stumbling block to forward progress of the IMA-on-NFS specification
is the lack of any standard describing IMA. You recommended reaching
out to the Linux IMA community to inquire if they had plans for
standardizing IMA. I have done this:

   https://www.spinics.net/lists/linux-integrity/msg03399.html

And received no response. I conclude that there is no active effort
and perhaps no desire to standardize Linux IMA.

Therefore I propose the following changes to
draft-ietf-nfsv4-integrity-measurement:

 - Replace all specific mention of IMA with general discussion of
   purpose and use cases
 - Rename the new attributes with non-specific names; for example
   fattr4_signature_content, fattr4_signature_attributes, and
   fattr_signature_attrlist
 - REQUIRE that the material stored in the first two attributes is
   self-describing, such that the integrity measurement subsystems
   on NFS endpoints can properly interpret and utilize their content

Do you feel this is adequate for the purposes of standardization? Are
there other approaches you would like me to consider?

Comments/suggestions from the cheap seats (tm David Black) also welcome.

--
Chuck Lever

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 28 Aug 2018 14:13:51 -0500

Hi, Chuck,

You invited comments from the "cheap seats" in your note, but I think
that's actually me, and people who understand what's going on better than I
do would be in the orchestra pit seats. So I'd also invite comments from
people who understand what's going on better than I do.

Having said that ...

> and perhaps no desire to standardize Linux IMA.

I THINK what we're talking about here, is a stable reference for
draft-ietf-nfsv4-integrity-measurement to point to. Does that sound right?

I'd like to better understand what reference is available - whether it is

   - a versioned description with a stable URL
   - an unversioned description
   - "the code is in github, good luck"
   - or some variation along that spectrum

I think having a stable reference matters more than something being a
"standard", from our perspective.

>    on NFS endpoints can properly interpret and utilize their content

So, is the goal here to be resilient to changes in IMA, or to be resilient
to replacing IMA with something else?

> there other approaches you would like me to consider?

I'd only be looking for some way to achieve interoperation, so I'd be
listening for feedback from people who will implement this and deploy this
mechanism.

I hope that's helpful!

Spencer (D)

---

From: Chuck Lever <[email-masked]>
Date: Tue, 28 Aug 2018 16:05:34 -0400

> On Aug 28, 2018, at 3:13 PM, Spencer Dawkins at IETF <[email-masked]> wrote:
> 
> Hi, Chuck, 
> 
> You invited comments from the "cheap seats" in your note, but I think that's actually me, and people who understand what's going on better than I do would be in the orchestra pit seats. So I'd also invite comments from people who understand what's going on better than I do.

According to the RFC Editor, as AD you are the final
authority on what is an adequate set of citations
for the NFS specs we publish, so you get to be the
conductor today.

Or anyway, you get first crack at an opinion.

> I THINK what we're talking about here, is a stable reference for draft-ietf-nfsv4-integrity-measurement to point to. Does that sound right?

Yes, that was the problem described in Montreal.

> 	• a versioned description with a stable URL
> 	• an unversioned description
> 	• "the code is in github, good luck"
> 	• or some variation along that spectrum

There is a wiki with a white paper, and there is what
is implemented in Linux.

https://sourceforge.net/p/linux-ima/wiki/Home/

So, some from row B and some from row C. And some man
pages. It's a practical implementation, not something
that was conceived by standard and later implemented.

However, all the participating filesystem needs to do
is store some security metadata. So I would like the
NFS pieces to be entirely agnostic to the metadata
format, and instead focus on:

a) how to find this metadata
b) how large it can be
c) how to protect it in-transit
d) how to authorize access and updates to it

> I think having a stable reference matters more than something being a "standard", from our perspective. 

We could cite the URL where the white paper resides, as
an Informative reference, but I'm not certain that is a
"stable" URL. Maybe the wiki is stable enough?

> So, is the goal here to be resilient to changes in IMA, or to be resilient to replacing IMA with something else?

Mostly the former goal. The latter is hypothetical at this
point, but possible down the road.

> I'd only be looking for some way to achieve interoperation, so I'd be listening for feedback from people who will implement this and deploy this mechanism.

Fair enough.

The issue of interoperation is IMA's problem: they need to
be able to add/remove features while remaining compatible
with what's already in on-disk filesystems, as their vision
of IMA evolves. IOW they already have to solve this problem
with local filesystems.

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Tue, 28 Aug 2018 20:42:18 +0000

> of IMA evolves. IOW they already have to solve this problem

Is your intention to limit the discussion to IMA? What about filesystems
which support native integrity, such as ReFS, ZFS, etc? It seems that if
those are present, there should be a story for NFS to use them. Or if not,
then a portable, interoperable NFS-level definition. Hitching onto a
sidetracked train doesn't seem, to me, a recipe for success.

Just my view from another cheap seat.

Tom.

> -----Original Message-----
> From: nfsv4 <[email-masked]> On Behalf Of Chuck Lever
> Sent: Tuesday, August 28, 2018 4:06 PM
> To: Spencer Dawkins at IETF <[email-masked]>
> Cc: NFSv4 <[email-masked]>
> Subject: Re: [nfsv4] Linux IMA on NFS follow-up
> 
> 
> 
> > You invited comments from the "cheap seats" in your note, but I think that's
> actually me, and people who understand what's going on better than I do
> would be in the orchestra pit seats. So I'd also invite comments from people
> who understand what's going on better than I do.
> 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.spi
> nics.net%2Flists%2Flinux-
> integrity%2Fmsg03399.html&amp;data=02%7C01%7Cttalpey%40microsoft.co
> m%7C18f04391586c4fcee43308d60d21a54e%7C72f988bf86f141af91ab2d7cd
> 011db47%7C1%7C0%7C636710835491618174&amp;sdata=Up9RBEFvcwXzx9
> hcDCBnYO3U1usX4MEwHlOZGbFZ83s%3D&amp;reserved=0
> >>
> > I THINK what we're talking about here, is a stable reference for draft-ietf-
> nfsv4-integrity-measurement to point to. Does that sound right?
> 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsourcefor
> ge.net%2Fp%2Flinux-
> ima%2Fwiki%2FHome%2F&amp;data=02%7C01%7Cttalpey%40microsoft.com
> %7C18f04391586c4fcee43308d60d21a54e%7C72f988bf86f141af91ab2d7cd01
> 1db47%7C1%7C0%7C636710835491618174&amp;sdata=iIaZX2iTXbmTvIqGuz
> OTJ97Zm7Sz6Y6MaWezWZA9vj0%3D&amp;reserved=0
> 
> > I'd only be looking for some way to achieve interoperation, so I'd be listening
> for feedback from people who will implement this and deploy this mechanism.
> 
> Fair enough.
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf
> .org%2Fmailman%2Flistinfo%2Fnfsv4&amp;data=02%7C01%7Cttalpey%40micr
> osoft.com%7C18f04391586c4fcee43308d60d21a54e%7C72f988bf86f141af91a
> b2d7cd011db47%7C1%7C0%7C636710835491628183&amp;sdata=NQQdpaw
> GC19QWioBOPkj2O7SnLlvweZvYSNhsLEt1MU%3D&amp;reserved=0

---

From: Chuck Lever <[email-masked]>
Date: Tue, 28 Aug 2018 19:22:11 -0400

Hi Tom-

> On Aug 28, 2018, at 4:42 PM, Tom Talpey <[email-masked]> wrote:
> 
> which support native integrity, such as ReFS, ZFS, etc?

I think it is limited to integrity measurement. It is
different than "native integrity", where the data is
protected only while it's at rest on the NFS server.
Native integrity metadata is generated on the fly by 
the filesystem itself.

Native integrity also would not seem to be suitable for
protecting the data while in transit. NFS has a separate
mechanism for that too.

IMA provides provenance: the content hash is signed by a
software provider. The signature plus the content hash
guarantee that the content hasn't been altered since the
file was written by the software provider. It ensures
integrity from the "factory" all the way to the
execution environment of the end user.

So this is a piece of long-lived security metadata that
is generated independently of users or local storage
devices.

Our approach to this class of metadata has been to create
additional fattr4 attributes: ACLs, security labels, and
so on. Here we have something that has a definite purpose,
but no defined format.

> It seems that if
> then a portable, interoperable NFS-level definition.

The NFS protocol and NFS implementations do not have any
role in interpreting this metadata. All that is needed
is to store some additional bits that will be interpreted
elsewhere.

Seems like we've done this before: NFS security labels
are defined elsewhere, and are not interpreted by NFS
itself. User extended attributes, named streams are not
interpreted by NFS, and are separate from file content.

IMO it's valid to claim that interoperability in this
case is out of scope.

> Hitching onto a

--
Chuck Lever

---

From: "Quigley, David" <[email-masked]>
Date: Wed, 29 Aug 2018 04:31:07 +0000

To be fair IMA integrity and ZFS integrity aren't the same thing. ZFS integrity is to validate that the file is correct based on what you placed on disk at the block level. IMA's integrity is similar to a tripwire style integrity. Where you have essentially taken an integrity snapshot of the system so if a file is modified from what you believe should be there you can detect it. So there is no interoperability story to tell there. I've never heard of ReFS before so I don't know what kind of integrity that is whether it’s the IMA or ZFS type. I think the important takeaway here is that Integrity in the IMA sense is more in line with system integrity not traditional file integrity. 

-----Original Message-----
From: nfsv4 [mailto:[email-masked]] On Behalf Of Tom Talpey
Sent: Tuesday, August 28, 2018 2:42 PM
To: Chuck Lever <[email-masked]>; Spencer Dawkins at IETF <[email-masked]>
Cc: NFSv4 <[email-masked]>
Subject: Re: [nfsv4] Linux IMA on NFS follow-up

> The issue of interoperation is IMA's problem: they need to be able to 
> add/remove features while remaining compatible with what's already in 
> on-disk filesystems, as their vision of IMA evolves. IOW they already 
> have to solve this problem with local filesystems.

Is your intention to limit the discussion to IMA? What about filesystems which support native integrity, such as ReFS, ZFS, etc? It seems that if those are present, there should be a story for NFS to use them. Or if not, then a portable, interoperable NFS-level definition. Hitching onto a sidetracked train doesn't seem, to me, a recipe for success.

Tom.

> actually me, and people who understand what's going on better than I 
> do would be in the orchestra pit seats. So I'd also invite comments 
> from people who understand what's going on better than I do.
> 
> According to the RFC Editor, as AD you are the final authority on what 
> is an adequate set of citations for the NFS specs we publish, so you 
> get to be the conductor today.
> 
> >> On Tue, Aug 28, 2018 at 1:38 PM Chuck Lever 
> >> <[email-masked]>
> wrote:
> >> Hi Spencer D. !
> >>
> >> During our meeting in Montreal last month, I presented a 
> >> backgrounder on supporting Linux IMA (file integrity measurement) on NFS.
> >>
> >> A stumbling block to forward progress of the IMA-on-NFS 
> >> specification is the lack of any standard describing IMA. You 
> >> recommended reaching out to the Linux IMA community to inquire if 
> >> they had plans for standardizing IMA. I have done this:
> >>
> >>
> > draft-ietf-
> There is a wiki with a white paper, and there is what is implemented 
> in Linux.
> 
> So, some from row B and some from row C. And some man pages. It's a 
> practical implementation, not something that was conceived by standard 
> and later implemented.
> 
> However, all the participating filesystem needs to do is store some 
> security metadata. So I would like the NFS pieces to be entirely 
> agnostic to the metadata format, and instead focus on:
> 
> We could cite the URL where the white paper resides, as an Informative 
> reference, but I'm not certain that is a "stable" URL. Maybe the wiki 
> is stable enough?
> 
> 
> > So, is the goal here to be resilient to changes in IMA, or to be 
> > resilient to
> replacing IMA with something else?
> 
> Mostly the former goal. The latter is hypothetical at this point, but 
> possible down the road.
> 
> 
> >> Do you feel this is adequate for the purposes of standardization? 
> >> Are there other approaches you would like me to consider?
> >
> The issue of interoperation is IMA's problem: they need to be able to 
> add/remove features while remaining compatible with what's already in 
> on-disk filesystems, as their vision of IMA evolves. IOW they already 
> have to solve this problem with local filesystems.
> 
> 
> > I hope that's helpful!
> >
> > Spencer (D)
> >
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Tom Talpey <[email-masked]>
Date: Wed, 29 Aug 2018 07:48:43 -0400

> protecting the data while in transit. NFS has a separate

Ok, thanks for the clarification. Obviously, I haven't read the
draft.

NFS has protection for its messages, at least if krb5i is used,
but that's a hop-by-hop protection and quite different. Not at
issue, here, got it.

> is generated independently of users or local storage

Ok, that's clear, but it requires the upper layer to provide the
information. So NFS becomes basically a transport for the hash.
What upper layers are actually using such an interface? Are
they actually looking to use NFS?

Unless such an application is part of the discussion, I don't think
the working group should take this on.

Tom.

---

From: Chuck Lever <[email-masked]>
Date: Wed, 29 Aug 2018 11:18:07 -0400

> On Aug 29, 2018, at 7:48 AM, Tom Talpey <[email-masked]> wrote:
> 

It was a good question! And I'm happy to repeat some of the
draft and PowerPoint here for folks who missed Montreal.

> but that's a hop-by-hop protection and quite different. Not at

As David pointed out, perhaps different terminology would
help distinguish the purpose of this new mechanism from
typical at-rest integrity (a la ZFS) and in-transit integrity
(a la krb5i). I'll use the term "measurement metadata" for
the moment.

> What upper layers are actually using such an interface?

Let's consider an example of how IMA works today.

A software vendor wants to provide its own version of the
"ping" program. To avoid malicious tampering, it computes an
HMAC hash of the executable file, and signs the hash. That
signed hash accompanies the executable file to the vendor's
customers. This is the measurement metadata.

When a customer installs "ping" on a filesystem that supports
IMA, the measurement metadata is saved as an extended attri-
bute of the file storing the "ping" executable, similar to
any security label.

Before executing "ping", the customer's operating system
pulls the "ping" executable into its page cache. The IMA
module reads the measurement metadata. It verifies the signa-
ture against the software vendor's public certificate. It
verifies the hash by recomputing the HMAC on the copy of
"ping" in the page cache.

If all checks out, "ping" is allowed to execute normally. If
not, the operating system prevents execution.

Measurement metadata is typically provided by someone else,
not by local agents. And this metadata is frequently auth-
enticated.

> Are they actually looking to use NFS?

Cloud operators that support Linux provide the guest's root
filesystem, where most of the guest OS's executables reside,
using ext4 on a virtual disk (or via iSCSI). ext4 supports
IMA. Cloud operators want to use that to guarantee to
customers that they are getting OEM software components that
have not been tampered with.

Some operators would like to use NFS for guest root file-
systems. They'd like to have the same support for measure-
ment in that environment as they have with ext4.

Conversely, a cloud operator may want to provide its own
particular versions of some executables, in which case it
can hash and sign them and provide the customers with its
public cert for verification.

> the working group should take this on.

The application that uses measurement metadata is the IMA
subsystem, which resides in the kernel with the filesystem
implementation, to be clear. The measurement subsystem is a
security module that is separate from the filesystem. No
part of any filesystem implementation alters or interprets
measurement metadata.

If tripwire is similar to IMA, then perhaps we are really
talking about a class of applications. Each member of that
class might have their own metadata format, much like the
various parties that might want to write into an NFSv4
security label, which oddly enough is also an opaque array
of bytes that is not interpreted by NFS.

--
Chuck Lever

---

From: "Theodore Y. Ts'o" <[email-masked]>
Date: Wed, 29 Aug 2018 16:54:33 -0400

> of bytes that is not interpreted by NFS.

I don't know if the NFS working group wants to be quite this general,
but there is also fs-verity[1], which requires an opaque array of
bytes plus an opaque _stream_ of bytes.

[1] https://events.linuxfoundation.org/wp-content/uploads/2017/11/fs-verify_Mike-Halcrow_Eric-Biggers.pdf

(ignore the date in the URL; this was presented this week on Monday in
Vancouver, at the Linux Security Symposium 2018.)

fs-verity uses a Merkle Tree, and the difference from IMA is that if
it allows blocks to be verified as they are used.  This minimizes the
time between launching a gargantuan binary and when it can first start
executing.  It's also useful if you have a very large media file (say
a 4 gigabyte video) and you want to start viewing it right away
without waiting for the IMA hash calculation.

It also has the benefit that if there are large portions of the file
that never get used --- for example if this is an APK, and it includes
a dozen-plus translation files into different languages, many of those
languages will never be used on a particular device, so there's no
point trying to calculate a crypto checksum over parts of the file
which are irrelevant from security perspective.  (E.g., if you are
using the Brazillian translation, you really don't care if the French
translation has been corrupted.)

The flip side is that instead of knowing at file open time whether or
not the file is fully authenticated, the error is detected when the
specific portions of the file is read into memory.  So the error might
occur when an executed demand-pages in part of the binary (causing the
binary to abort mid-execution).

For NFS, the other problem with IMA which is solved by fs-verity is
that if the data file is really large (hundreds of megabytes), the
entire file will get sucked over the network when it is first opened,
and the IMA hash calculated and validated.  Once IMA hash is
validated, if there is memory pressure, those pages will get pushed
out of the page cache --- and then when the pages are subsequently
demand paged from a (potentially malicious) NFS server or
man-in-the-middle attacker, no data integrity check will be performed
at that point.

Fs-verity has been implemented for the ext4 and f2fs file systems, and
will be shipping as part of Android Pie.  I've also learned that
Fuschia had independently come up with the exact some mechanism,
for a similar although slightly different use case.

Cheers,

					- Ted

---

From: Chuck Lever <[email-masked]>
Date: Wed, 29 Aug 2018 17:12:19 -0400

> On Aug 29, 2018, at 4:54 PM, Theodore Y. Ts'o <[email-masked]> wrote:
> 

Yep, several of us have been following this work closely.
However, as Eric Biggers pointed out when he posted his
fs-verity patches last week, fs-verity does not replace
IMA, it augments it.

The plan is to use fs-verity or something like it, once a
measurement mechanism is supported by NFS, to help with
performance issues you describe below.

But we do have a problem with stuffing the Merkel tree
into the file's byte stream and emulating the EOF. Instead
we would like NFS to transport the Merkle tree in a named
attribute. Named attributes are already supported by NFSv4
and can handle arbitrarily large blobs of data independently
of the file's regular content.

The proposed user space interface design is not going to work
for NFS. A real API (like an ioctl) is a much better approach
as it appropriately separates the presentation of the Merkle
tree from the underlying transport and storage mechanisms
that will necessarily be different depending on the filesystem
implementation.

--
Chuck Lever

---

From: Benjamin Kaduk <[email-masked]>
Date: Wed, 29 Aug 2018 16:18:48 -0500

Sniping from the back of the third balcony (and trimming heavily)...

> HMAC hash of the executable file, and signs the hash. That

(Not sure why HMAC would be needed, since that's a keyed hash and how do
you distribute the key, but the rest holds just fine with an unkeyed hash.)

> public cert for verification.

But how are these certs/trust anchors distributed?  If they also come over
NFS, then there's not much added value.

> of bytes that is not interpreted by NFS.

I think the original posting was probably in the right ballpark in that we
just need a self-describing container for "this sort of data" (though we of
course need to pay attention to whether it's fixed-length vs. streaming, if
we actually need two of them, etc.).  Once it's self-describing (i.e., with
an OID tag and an explicit length so unrecognized types can be skipped),
then the stability of the reference becomes a problem for whoever is
issuing the OID and less of something that we need to worry about.

-Ben

---

From: Chuck Lever <[email-masked]>
Date: Wed, 29 Aug 2018 17:26:57 -0400

> On Aug 29, 2018, at 5:18 PM, Benjamin Kaduk <[email-masked]> wrote:
> 

Trust anchors are distributed independently of the protected
file content. For sensitive deployments, they are pre-loaded
into a local Trusted Computing Module.

--
Chuck Lever

---

From: "Theodore Y. Ts'o" <[email-masked]>
Date: Thu, 30 Aug 2018 02:49:18 -0400

On Wed, Aug 29, 2018 at 05:12:19PM -0400, Chuck Lever wrote:
> into the file's byte stream and emulating the EOF.
>
> Instead we would like NFS to transport the Merkle tree in a named
> attribute. Named attributes are already supported by NFSv4 and can
> handle arbitrarily large blobs of data independently of the file's
> regular content.

The kernel/userspace interface is an interesting problem.  Technically
it doesn't have anything to do with NFSv4 the _protocol_ but userspace
interface has a lot to do with how difficult it is to implement the
protocol and interface that with the userspace applications that are
trying to set up the Merkle tree.

POSIX doesn't specify an interface to write to an named attribute (aka
alternate data streams).  Yes, Solaris has a proposed interface which
is wildly popular with rootkit authors.  But no other Unix or Linux
compatible OS supports it.  Worse, most of Linux's file systems don't
support alternate data streams.

So it's all very well and good to say that from a NFSv4 transport
perspective, we should use named attributes.  But from a client
perspective, we don't have any way way to write named attributes in
Linux, because the VFS doesn't support it.  And for a Linux NFS server
implementation which is going to write the file to an existing file
system (e.g., ext4, xfs, btrfs), there is no support for a named
attribute, so it's not clear how an NFS server implementation would
support this.

> implementation.w

Well, we do have a real API; and it is an ioctl.  It's just that the
ioctl says, "treat the last 4k of the file as a fs-verity header; this
will tell you where the data fork ends, and the Merkle tree starts,
and move the Merkle tree to a hidden storage (which is not defined),
and truncate it so that file ends at the end of the file."

It just so happens that the Merkle tree storage uses the logical to
physical block map for ext4 and f2fs, so it looks like we're just
adjusting i_size.  But from a *conceptual* point of view, that ioctl
could also result in an implementation creating an Alternate Data
Stream in NTFS, and copying that data to that stream, and then
truncating the file.

This was certainly an expedient way to do this, because it meant we
didn't need to define a new VFS / Linux System call interface for
Alternate Data Streams or NFS Named Attributes.  Personally, I have
zero interest in trying to create such an userspace interface.  If you
want to try, there's an active thread going on right now about the
wisdom (or foolishness) of creating syscall interface for ADS/Named
Attribute.   I don't think it has much chance of success.

	       	     	      	       - Ted

---

From: Chuck Lever <[email-masked]>
Date: Thu, 30 Aug 2018 13:31:42 -0400

> On Aug 30, 2018, at 2:49 AM, Theodore Y. Ts'o <[email-masked]> wrote:
> 

We seem to be talking past each other.

There are three architectural problems to solve:

1. How Merkle trees are presented to user space (user
space API)

2. How Merkle trees are transported from storage initiator
to storage target and back (storage protocol)

3. How Merkle trees are stored on durable storage (on-disk
filesystem)

For expediency the current fs-verity proposal has ignored 2.
(because local-only filesystems don't have this issue) and
combined 1. and 3. Fine for a prototype.

A generic implementation of fs-verity, one that can be
implemented by a broad variety of local and remote filesystems,
has to grapple with and separate all 3 of these issues, because
they might be handled in three distinct hardware domains for
some filesystems.

The NFSv4 _protocol_ has a named attribute facility. That's not
to say we want to expose that facility to the NFS client's user
space or even to the NFS client's VFS. And we are not claiming
that the server's exported filesystem has to use named
attributes either.

None of these layers has to know exactly how the Merkle tree
gets from the initiator to the target and back.

A named attribute protocol element is quite useful for
solving 2. for NFSv4, because it is already defined and has
the required semantics. To put it plainly, we will not
have to extend the NFSv4 protocol to be able to transport
Merkle trees using a named attribute.

Please don't confuse the use of NFSv4 named attributes for
this special purpose with the need to implement full
stack named attribute support in Linux. And also note that
this is only one proposal. We have other choices for 2.

> Alternate Data Streams or NFS Named Attributes.

For better or worse, some non-Linux operating systems already
have support for named attributes. NFS has to play well with
them as well as it does with Linux. It's not fair to bring
fs-verity to the NFS community and ask that non-Linux NFS
implementations support something that works nicely only
for ext4 on Linux.

Again, how NFS gets the Merkle tree from point A to point B
can and should be opaque to applications and the VFS. The
filesystem storage implementation details should also be
opaque. Therefore the user space API should be abstract; ie
it should not reflect or expose the specifics of either 2.
or 3 to applications.

> Personally, I have

Yes, I'm watching that discussion. It looks fair and cons-
tructive and likely to terminate with EWONTIMPLEMENT.

--
Chuck Lever

---

From: "Quigley, David" <[email-masked]>
Date: Thu, 30 Aug 2018 18:11:42 +0000

The only comment I would add here is that named attributes are essentially directories and files with the data for the named attribute being the file data. This was one of the reasons we dismissed this faculty outright for MAC labeling because its essentially turtles the entire way down. If these are directories and files then they themselves would need MAC labels of their own. The same might be said of the Integrity data. If these are files and directories wouldn't you want an IMA measurement of them as well? At what point would you stop? 

Dave

-----Original Message-----
From: nfsv4 [mailto:[email-masked]] On Behalf Of Chuck Lever
Sent: Thursday, August 30, 2018 11:32 AM
To: Theodore Y. Ts'o <[email-masked]>
Cc: NFSv4 <[email-masked]>
Subject: Re: [nfsv4] Linux IMA on NFS follow-up

>> But we do have a problem with stuffing the Merkel tree into the 
>> file's byte stream and emulating the EOF.
>> 

1. How Merkle trees are presented to user space (user space API)

2. How Merkle trees are transported from storage initiator to storage target and back (storage protocol)

For expediency the current fs-verity proposal has ignored 2.
(because local-only filesystems don't have this issue) and combined 1. and 3. Fine for a prototype.

A generic implementation of fs-verity, one that can be implemented by a broad variety of local and remote filesystems, has to grapple with and separate all 3 of these issues, because they might be handled in three distinct hardware domains for some filesystems.

The NFSv4 _protocol_ has a named attribute facility. That's not to say we want to expose that facility to the NFS client's user space or even to the NFS client's VFS. And we are not claiming that the server's exported filesystem has to use named attributes either.

None of these layers has to know exactly how the Merkle tree gets from the initiator to the target and back.

A named attribute protocol element is quite useful for solving 2. for NFSv4, because it is already defined and has the required semantics. To put it plainly, we will not have to extend the NFSv4 protocol to be able to transport Merkle trees using a named attribute.

Please don't confuse the use of NFSv4 named attributes for this special purpose with the need to implement full stack named attribute support in Linux. And also note that this is only one proposal. We have other choices for 2.

>> The proposed user space interface design is not going to work for 
>> NFS. A real API (like an ioctl) is a much better approach as it 
>> appropriately separates the presentation of the Merkle tree from the 
>> underlying transport and storage mechanisms that will necessarily be 
>> different depending on the filesystem implementation.w
> 

> Alternate Data Streams or NFS Named Attributes.

For better or worse, some non-Linux operating systems already have support for named attributes. NFS has to play well with them as well as it does with Linux. It's not fair to bring fs-verity to the NFS community and ask that non-Linux NFS implementations support something that works nicely only for ext4 on Linux.

Again, how NFS gets the Merkle tree from point A to point B can and should be opaque to applications and the VFS. The filesystem storage implementation details should also be opaque. Therefore the user space API should be abstract; ie it should not reflect or expose the specifics of either 2.
or 3 to applications.

> Attribute.   I don't think it has much chance of success.

Yes, I'm watching that discussion. It looks fair and cons- tructive and likely to terminate with EWONTIMPLEMENT.

--
Chuck Lever

---

From: "Theodore Y. Ts'o" <[email-masked]>
Date: Thu, 30 Aug 2018 15:37:41 -0400

> 1. How Merkle trees are presented to user space (user

"Present" really isn't the right word here, since userspace never has
to be "presented the Merkle tree".  The file system *uses* the Merkle
tree to validate the integrity of the pages.

I think what you mean here is how is the Merkle tree presented (I
would use the word transferred) *from* userspace *to* the kernel.
Without an Named Attribute interface in the Linux system call
interface (which I have zero interest in proposing and implementing),
there are basically only two options:

1.  Passing the Merkle tree as a single in-memory buffer which sent to
the kernel via a pointer.

2.  Passing the Merkle tree in the "primary" data stream (e.g., after
the file data), and then triggering a request to move the Merkle tree
to the file system specific storage.

The problem with (1) is that the Merkle tree can be very large ---
approximately file size divided by 127.  So for a 500 megabyte file,
that's roughly 4 megs of Merkle tree.  That's really awkward for (1),
which is why (2) makes a lot more sense --- and why we did it that
way.

> to storage tharget and back (storage protocol)
> filesystem)t

>From my perspective, both of these are file system-specific
implementation details which can be different for different file
systems.  It may be that what we are doing is more convenient for
local disk formats, but the API described above is the only thing we
could have done in the absense of an alternate data stream API for
Linux.

What we *could* do is require userspace issue an ioctl stating, "we're
about to start writing the fs-verity metadata", and the local OS
system call interface continue to be to write the data at the
beginning of the next block.  But the local NFS client on Linux would
then know that the writes should be going to a NFSv4 named attribute
--- and what that named attribute would be named, would properly be
the concern of the NFSv4 IETF working group.

So what I'm proposing is that today, userspace does this:

1)  Write the file data
2)  Write the fs-verity metadata starting at the next 4k boundary
3)  Issue an ioctl that converts the file into a read-only fs-verity
    protected file.

We could add to this interface a requirement that userspace do a step
an extra step between steps 1 and 2:

1.5) Issue an ioctl saying, "we're about to start writing the
     fs-verity metadata".

Where NFSv4 decides to store the fs-verity metadata, and how the NFSv4
client transfer the fs-verity metadata to the NFSv4 server is NFSv4's
business, and I wouldn't presume to dictate that to the NFSv4
implementors or working group.  :-)

Does that seem like workable solution that would allow the NFSv4
working group to proceed?  And Chuck, maybe we should move discussion
about the local OS API details to a Linux mailing list, such as the
linux-fscrypt mailing list, since it's not likely to be of interest to
most of the other NFSv4 wg members?

Cheers,

					- Ted

---

From: Chuck Lever <[email-masked]>
Date: Fri, 31 Aug 2018 11:38:00 -0400

> On Aug 30, 2018, at 3:37 PM, Theodore Y. Ts'o <[email-masked]> wrote:
> 

I agree that 1. is also not adequate for NFS.

For example, we have an RPC message size limit, usually around
1MB, thus a single RPC message would not be capable of trans-
porting a sizable Merkle tree.

> implementors or working group.  :-)

Agreed, the goal is to provide a well-constructed API here
so that filesystem implementers have the flexibility they
want.

> working group to proceed?

I was thinking about something not much different to what you
proposed above. The exact steps depend on whether your ioctl
operates on fd's or pathnames.

1)  Open soon-to-be-protected file for writing (say, fd 3)
2)  Write data content into fd 3
3)  Issue a special ioctl on fd 3 that returns fd 4
4)  Close fd 3
5)  Write fs-verity metadata to fd 4
6)  Close fd 4
6a) Side effect of close: the file is now immutable and
protected by the fs-verity metadata (or, could issue an
ioctl to do this).

This further hides the implementation of the storage mechanism
from user space. Rather, it looks like a very familiar API
abstraction that may or may not be the same underlying storage
container. And we get to continue to use write(2) to set up
the fs-verity metadata.

> And Chuck, maybe we should move discussion

I'd like to give my colleagues another moment or two to
comment, then we can concern ourselves with implementation
details on another mailing list.

--
Chuck Lever

---

From: "Theodore Y. Ts'o" <[email-masked]>
Date: Fri, 31 Aug 2018 17:35:52 -0400

> protected by the fs-verity metadata (or, could issue an

There are a bunch of things that makes this awkward; and they are all
implementation details.  In order to set up an alternate file
descriptor, you're going to have to fake up a pseudo-inode.  This is
actually a lot more complex than you might think, since it means
faking up a path structure containing a dentry which needs to be at
least someone real to keep the dcache code from getting very unhappy.
A new inode means you also need to deal with SElinux that will be very
curious what the SELinux context should be for that new inode (and not
just for SELinux but for any other LSM).

A new file descriptor raises all sorts of new questions about what
should happen if various fcntl operations are set on it; or mmap(2),
dup(2), fchown(2) or fchmod(2), etc.  Or what should be returned if
stat(2) is called on the new file descriptor?

Using a new inode associated with the fd 4 also makes life much harder
for the existing file systems that have implemented fs-verity, since a
new inode means the writes will go into the page cache in a separate
address space.  And while we could just make fd 4 a dup of fd 3, that
makes certian assumptions about the f_offset of the fd, that might be
different implementation (such as NFS might choose).

So I would greatly prefer using two ioctl's as opposed to the two fd
solution; but this is really not really relevant to the NFSv4 protocol
--- it really is a OS interface issue.

						- Ted