Subject: [nfsv4] Possible change to Flex Files layout

From: Rick Macklem <[email-masked]>
Date: Tue, 8 Mar 2016 22:39:14 -0500 (EST)

Hi,

I am building an NFSv4.1 pNFS for FreeBSD using Flex Files and
I've run into a case where it would be nice to make a small
addition to the layout.

I have a case where the files are replicated over K of the N
servers in the cluster. Which K of the N is on a per file basis.
(I am using GlusterFS as a backend and this is its replication
 strategy, if I understand it correctly.)

When there are replicas, the files are basically mirrored, but
writes only need to be done to one of the replicas/mirrors.
(Again, if I understand what GlusterFS does.)

-- So, what do others think of adding a boolean for the layout
   that says "only one mirror needs to be written VS all mirrors
   need to be written".

I can live without this, but I think I would find it useful and
I don't see a problem with adding it.

Thanks for your consideration of this, rick

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 9 Mar 2016 09:04:25 -0500

Hi Rick,

> Thanks for your consideration of this, rick

I don't see any major problems with this request. We have the
ffl_flags field in the layout, and so adding a flag for this would be
trivial, and could be done without breaking existing clients. Those
existing implementations would presumably be inefficient, since they
would ignore the flag, and continue to write to all mirrors, but they
would not break.

One question I have, though, is how this would affect the
ffds_efficiency? The current expectation for that field is that it
applies only to the mirrored read case, where you want to select one
of many mirrors. Wouldn't we also want to extend that scope of that
field so that it can apply to writes?
Either way I think we'd need a proposal for how we might modify the
description in https://tools.ietf.org/html/draft-ietf-nfsv4-flex-files-06#section-8.1

Cheers
  Trond

---

From: Rick Macklem <[email-masked]>
Date: Wed, 9 Mar 2016 22:46:18 -0500 (EST)

> field so that it can apply to writes?
Although I don't have a strong opinion, I just stated it would be used
for writes the same as reads for this case in the diff. (At least that
was what I tried to say;-)

I also stated that the storage device had to make sure that all mirrors
were updated before replying successfully to the write (but they don't
have to have the data committed to stable storage until the associated
Commit). Does this sound reasonable?

> description in
> https://tools.ietf.org/html/draft-ietf-nfsv4-flex-files-06#section-8.1
> 
I've attached a diff of the .txt version as a first stab at the change.

Thanks for your comments, rick

>   Trond
--- draft-ietf-nfsv4-flex-files-06.txt.sav	2016-03-09 18:17:00.253928000 -0500
+++ draft-ietf-nfsv4-flex-files-06.txt	2016-03-09 19:34:51.670735000 -0500
@@ -765,6 +765,7 @@ Internet-Draft              Flex File La
    <CODE BEGINS>
 
    /// const FF_FLAGS_NO_LAYOUTCOMMIT   = 1;
+   /// const FF_FLAGS_WRITE_ONE_MIRROR  = 2;
 
    /// typedef uint32_t            ff_flags4;
    ///
@@ -928,6 +929,8 @@ Internet-Draft              Flex File La
    tight coupling of the storage devices.  FF_FLAGS_NO_LAYOUTCOMMIT, can
    be set to indicate that the client is not required to send
    LAYOUTCOMMIT to the metadata server.
+   FF_FLAGS_WRITE_ONE_MIRROR can be set to indicate that the client only
+   needs to update one of the mirrors (see Section 8.2).
 
 5.2.  Interactions Between Devices and Layouts
 
@@ -1072,7 +1075,10 @@ Internet-Draft              Flex File La
    informed of via the layout.  If a layout segments is being resilvered
    to a storage device, that mirrored copy will not be in the layout.
    Thus the metadata server MUST update that copy until the client is
-   presented it in a layout.  Also, if the client is writing to the
+   presented it in a layout.
+   If the FF_FLAGS_WRITE_ONE_MIRROR is set in ffl_flags, the client need
+   only update one of the mirrors (see Section 8.2).
+   Also, if the client is writing to the
    layout segments via the metadata server, e.g., using an earlier
    version of the protocol, then the metadata server MUST update all
    copies of the mirror.  As seen in Section 8.3, during the
@@ -1103,7 +1109,8 @@ Internet-Draft              Flex File La
 
 8.2.  Writing to Mirrors
 
-   The client is responsible for updating all mirrored copies of the
+   If the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is not set,
+   the client is responsible for updating all mirrored copies of the
    layout segments that it is given in the layout.  If all but one copy
    is updated successfully and the last one provides an error, then the
    client needs to return the layout to the metadata server with an
@@ -1126,6 +1133,17 @@ Internet-Draft              Flex File La
    one.  If it has updates that were not committed, it MUST resend those
    updates to all mirrors.
 
+   If the FF_FLAGS_WRITE_ONE_MIRROR flag in ffl_flags is set, the client only
+   needs to update one of the copies of the layout segment.  For this case, the
+   storage device MUST ensure that all copies of the mirror are updated when
+   any one of the mirrors is updated.  All mirrors must be updated before
+   the storage device sends a successful reply for the write, but the data
+   does not need to be on stable storage until the associated Commit has
+   been done by the client.
+   The client may choose any one of
+   the mirrors and may use ffds_efficiency in the same manner as for reading
+   when making this choice.
+
 8.3.  Metadata Server Resilvering of the File
 
    The metadata server may elect to create a new mirror of the layout

---

From: Rick Macklem <[email-masked]>
Date: Wed, 9 Mar 2016 23:20:17 -0500 (EST)

> https://tools.ietf.org/html/draft-ietf-nfsv4-flex-files-06#section-8.1
More re propagation of a write to the mirrors:

Obviously, without a "write one mirror" flag, the client knows when all
mirrors are written, since the client does it.

In the "diff" I posted a few minutes ago...
- It stated that the write would be propagated to the mirrors before the
  write reply was sent.

- The "softer" version of this would be to require the write to be propagated
  to all mirrors by the time the Commit reply is sent to the client.

The "softer" version may be preferable, since it doesn't put as strong a requirement
on the server, but still allows the client to ensure the write is propagated
by doing a Commit when necessary.

If both seem appropriate, an additional FF_FLAGS_xx could be introduced to indicate
to the client which of the above the server provides when FF_FLAGS_WRITE_ONE_MIRROR
is set.

So, what do others think w.r.t. this?

Thanks, rick

---

From: Benny Halevy <[email-masked]>
Date: Thu, 10 Mar 2016 10:45:45 +0200

Hi Rick,

If the DS that the client wrote to responds with STABLE* then file data
have to be
on stable storage under any failure scenario.  Bare in mind that the client
will mark
the dirty page as clean in this case and is free to drop it out of its data
cache so it won't be able
to rewrite it in a server restart case. Therefore, I wouldn't allow data on
mirrors to be unstable
in this case.

Benny

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>

---

From: Rick Macklem <[email-masked]>
Date: Thu, 10 Mar 2016 20:23:01 -0500 (EST)

Benny Halevy wrote:
> 
> 
> On Thu, Mar 10, 2016 at 5:46 AM, Rick Macklem < [email-masked] > wrote:
> 
> 
> 
> Trond Myklebust wrote:
> > Hi Rick,
> > 
> > On Tue, Mar 8, 2016 at 10:39 PM, Rick Macklem < [email-masked] >
> > wrote:
> > > Hi,
> > > 
> If the DS that the client wrote to responds with STABLE* then file data have
> to be
To me, requiring the DS to reply STABLE to the write (which implies written to
stable storage on all mirrors) seems to be a very strong requirement.

Now, if you are saying that the DS may reply STABLE to indicate that all mirrors
have been updated, including to stable storage or the DS may reply UNSTABLE to
indicate that the mirrors may not be updated until the Commit, then I would
be fine with that.

After all, write sharing is a rare case and the client can use Commit to ensure
the update is done on all mirrors when that rare case is handled, I'll admit I
like the idea of that option being available. (I'd guess that most clients would
do the Commit before unlocking an exclusive byte range lock that overlaps the
byte range written, since write-sharing clients will normally use byte-range locking
to handle concurrency.)
--> Without the FF_FLAGS_WRITE_ONE_MIRROR flag, the client will need to write to
    all mirrors (and do Commits if the write reply is UNSTABLE), so I don't see
    the UNSTABLE write to one mirror too "soft" a requirement.

To be honest, at this time I have no idea if the NFSv3 server built into GlusterFS
does STABLE or UNSTABLE writes and what guarantees it provides w.r.t. when replicas
are updated, so I am trying to define this in such a way that I can support it.
(Btw, I don't really need this change. The default GlusterFS setup doesn't do
 replication and, therefore, won't use mirrors. I just thought it would be a nice
 feature to support.)

rick

---

From: Benny Halevy <[email-masked]>
Date: Fri, 11 Mar 2016 09:57:30 +0200

> > If the DS that the client wrote to responds with STABLE* then file data
have
> stable storage on all mirrors) seems to be a very strong requirement.

It is indeed. Similarly, think of a server with mirrored write cache. If it
replied STABLE before the mirrored copy was protected by stable storage
(and that includes battery backup of the cache) and crashed while losing
the user data or redundant data, then you'd expect it to return EIO when
accessing these areas that are potentially corrupt.  Returning stale data
post STABLE write or COMMIT can seriously mess up applications that rely on
stable storage semantics.

> indicate that the mirrors may not be updated until the Commit, then I
would

This is trivially possible when the client asks for UNSTABLE however (and
sadly IMO) the protocol does not allow the server to demote a write from
stable to unstable.

> byte range written, since write-sharing clients will normally use
byte-range locking

Agreed.

>     the UNSTABLE write to one mirror too "soft" a requirement.

I'd consider leaving that semantic choice to the admin as an export flag,
based on the application, as the behavior you described is reasonable and
practical for e.g. interactive usage, similar to other server tweaks that
allow divergence from the strict protocol (and that better be clearly
documented of course)

Benny

---

From: Rick Macklem <[email-masked]>
Date: Fri, 11 Mar 2016 21:06:06 -0500 (EST)

Benny Halevy wrote:
> 
> 
> 
> On Mar 11, 2016 3:23 AM, "Rick Macklem" < [email-masked] > wrote:
> > 
> > Benny Halevy wrote:
> > > 
> > > 
> replied STABLE before the mirrored copy was protected by stable storage (and
> that includes battery backup of the cache) and crashed while losing the user
> data or redundant data, then you'd expect it to return EIO when accessing
> these areas that are potentially corrupt. Returning stale data post STABLE
> write or COMMIT can seriously mess up applications that rely on stable
> storage semantics.
> 
I wasn't questioning the case where the DS chooses to reply STABLE. When it
does so, it must have committed to stable storage (and this applies to all
mirrors when writing to the metadata server or a DS when FF_FLAGS_WRITE_ONE_MIRROR
is set).

The case I meant was where the client specifies UNSTABLE.
I thought you were saying that the DS would be required to do this as a
STABLE write and reply STABLE. This seemed an unreasonably strong requirement
for the server.

> sadly IMO) the protocol does not allow the server to demote a write from
Sure, but I assumed the client did the write to the DS as UNSTABLE. Sorry
about the confusion.

> allow divergence from the strict protocol (and that better be clearly
Sure, although I tend to think the client`s choice of STABLE vs UNSTABLE in
the write request could be sufficient, since it is the client that needs
to redo writes for the UNSTABLE case and it is the client that will know
when any exclusive byte range lock is released.

Btw, I think the current draft needs to clarify what the semantics are for
the same question, but when applied to writes to the metadata server.
Section 8 Pg 19:
- clearly states that writes to the metadata server must be done on all
  mirrors.
  - What it doesn't seem to say is how quickly this needs to be done?
    (Which was the question that I had for the case where a write to a
     single DS will write to all mirrors.)

If we assume that the STABLE/UNSTABLE variants are adequately handled by
the STABLE/UNSTABLE in the write reply, there is still the question of
when all mirrors must be updated for the UNSTABLE case (2 possiblilities
I can think of. Are there others):
1 - All mirrors must be updated before the Write reply is sent to the client,
    and to stable storage iff STABLE is replied.
OR
2 - All mirrors must be updated before the Write reply is sent to the client,
    including to stable storage, if the write reply specifies STABLE.
    or
    All mirrors must be updated before the Commit reply is sent to the client,
    including to stable storage, if the write reply specifies UNSTABLE.

I am now leaning towards (2) for both the writes to the metadata server or
the write done to one DS iff the new flag FF_FLAGS_WRITE_ONE_MIRROR is added
to the spec. and is set.
I lean towards (2) because updating a lot of mirrors could take a while and
the client can choose by specifying STABLE vs UNSTABLE whether it requires the
server to update all mirrors right away or can wait until it does a Commit.

So, what do others think, rick

---

From: Benny Halevy <[email-masked]>
Date: Sat, 12 Mar 2016 09:57:27 +0200

On Mar 12, 2016 04:06, "Rick Macklem" <[email-masked]> wrote:
> > > > On Thu, Mar 10, 2016 at 5:46 AM, Rick Macklem < [email-masked]
>
> > > > wrote:
> > > >
> > > >
> > > >
> > > > Trond Myklebust wrote:
> > > > > Hi Rick,
> > > > >
> > > > the dirty page as clean in this case and is free to drop it out of
its
> > > > data
> > > > to rewrite it in a server restart case. Therefore, I wouldn't allow
data
> > > > on
> > > > mirrors to be unstable
> > > > in this case.
> > > >
> > > To me, requiring the DS to reply STABLE to the write (which implies
written
> > > to
(and
> STABLE write and reply STABLE. This seemed an unreasonably strong
requirement

OK, that makes more sense.
Still, the DS that was target to write better respond with UNSTABLE in this
case to ensure that the client will issue COMMIT before dropping the dirty
data out of its cache and before returning success to the application's
respective system call (a-la sync, flush, or close)

>
> > >
> > > Now, if you are saying that the DS may reply STABLE to indicate that
all
> > > mirrors
> > > have been updated, including to stable storage or the DS may reply
UNSTABLE
> > > to
> > > After all, write sharing is a rare case and the client can use Commit
to
> > > ensure
> > > like the idea of that option being available. (I'd guess that most
clients
> > > would
> > > do the Commit before unlocking an exclusive byte range lock that
overlaps
> > > the
> > > --> Without the FF_FLAGS_WRITE_ONE_MIRROR flag, the client will need
to
> > > write to

If that is the case then no export option is needed.

> So, what do others think, rick

Aside for the stability of written data there is the requirement for
visibility.
That is, a clustered server needs to mimic as much as possible the
functional behavior of a monolithic server.  With that in mind, the newly
written data is visible to all clients in a monolithic server regardless if
its stable_how value. The application level relies on that for
close-to-open semantics.  This implies that in a clustered server
configuration with multiple DSs, the MDS must make sure all mirrored copies
will return the new data before the WRITE reply is sent.

There could be multiple ways to ensure that. One is to actually write the
new data to all mirrors. But for example, in a shared storage architecture
the MDS might only need to invalidate the caches in the DSs.

Benny

>
> > Benny
> >
> > >
> > > To be honest, at this time I have no idea if the NFSv3 server built
into
> > > GlusterFS
> > > does STABLE or UNSTABLE writes and what guarantees it provides w.r.t.
when
> > > replicas
> > > are updated, so I am trying to define this in such a way that I can
support
> > > it.
> > > (Btw, I don't really need this change. The default GlusterFS setup
doesn't
> > > do
> > > replication and, therefore, won't use mirrors. I just thought it
would be a
> > > nice

---

From: David Noveck <[email-masked]>
Date: Sat, 12 Mar 2016 10:34:08 -0500

visibility.

Or at least potential requirements as to visibility.

> That is, a clustered server needs to mimic as much as possible the
functional behavior of a monolithic server.

True, but a lot of the deployments will not be clustered servers.  In fact,
in many cases in which one is replicating data, the two replicas will be
far enough apart that a natural disaster is unlikely to simultaneously
affect both.

In any case, pNFS does not strive to precisely emulate clustered server
semantics.  Instead, it seeks to provide many of the benefits of clustered
operation, while allowing requirements that depend on the clients being
aware of the distributed  nature of the arrangement.  Use of LAYOUTCOMMIT
is an example.

>With that in mind, the newly written data is visible to all clients in a
monolithic server regardless if its stable_how value. The application level
relies on that for close-to-open semantics.

True.

> This implies that in a clustered server configuration with multiple DSs,
the MDS must make sure all mirrored copies will return the new data before
the WRITE reply is sent.

It would imply that if an MDS with multiple DS's were a clustered server
configuration.

Since it isn't, I'd like us to consider how we might avoid a requirement to
immediately propagate writes to all replicas.  Given that there may be good
reasons to have some of these hundreds or thousands of miles away from the
others, I think we want to avoid a situation in which the client writes to
the closest replica but has to wait for all  replicas to receive and reply
to the write.  This adds a latency of 16μs/mi based on the distance to the
farthest replica.

I'm thinking we need something like the following to deal with this issue:

When stable writes are done to the MDS or to a single replica (if allowed
by the appropriate  layout flag), it is the responsibility of the receiving
node to propagate the written data stably, before replying to the client.

In the corresponding cases in which unstable writes are done, the receiving
node does not have any such obligation, although it may choose to
asynchronously propagate the updates.  However, once a COMMIT is replied
to, all replicas must reflect the writes that have been done, and these
data must have been committed to stable storage on all replicas.

In order to avoid situations in which stale data is read from replicas to
which writes have not been propagated:

   - A client which has outstanding unstable writes made to single node
   (MDS or DS) MUST do all reads from that same node.
   - When writes are flushed to the server, for example to implement,
   close-to-open semantics, a COMMIT must be done by the client to ensure that
   up-to-date written data will be available irrespective of the particular
   replica read.

On Sat, Mar 12, 2016 at 2:57 AM, Benny Halevy <[email-masked]> wrote:

> > > > > On Thu, Mar 10, 2016 at 5:46 AM, Rick Macklem <
> [email-masked] >
> > > > > wrote:
> > > > >
> > > > >
> > > > >
> > > > > Trond Myklebust wrote:
> > > > > > Hi Rick,
> > > > > >
> > > > > to rewrite it in a server restart case. Therefore, I wouldn't
> allow data
> > > > > on
> > > > > mirrors to be unstable
> > > > > in this case.
> > > > >
> Still, the DS that was target to write better respond with UNSTABLE in
> this case to ensure that the client will issue COMMIT before dropping the
> dirty data out of its cache and before returning success to the
> application's respective system call (a-la sync, flush, or close)
>
> >
> > > >
> > > > After all, write sharing is a rare case and the client can use
> Commit to
> > > > ensure
> > > > the update is done on all mirrors when that rare case is handled,
> I'll
> > > > admit I
> > > > does STABLE or UNSTABLE writes and what guarantees it provides
> w.r.t. when
> > > > replicas

---

From: Matt Benjamin <[email-masked]>
Date: Sat, 12 Mar 2016 20:12:57 -0500 (EST)

This approach appears consistent with the intent of dispersed DS architecture, as I understand it.  It seems helpful to formalize it.

Matt

----- Original Message -----
> From: "David Noveck" <[email-masked]>
> To: "Benny Halevy" <[email-masked]>
> Cc: "Rick Macklem" <[email-masked]>, "nfsv4" <[email-masked]>
> Sent: Saturday, March 12, 2016 10:34:08 AM
> Subject: Re: [nfsv4] Possible change to Flex Files layout
> 
> 
> 
> True, but a lot of the deployments will not be clustered servers. In fact, in
> many cases in which one is replicating data, the two replicas will be far
> enough apart that a natural disaster is unlikely to simultaneously affect
> both.
> 
> aware of the distributed nature of the arrangement. Use of LAYOUTCOMMIT is
> an example.
> 
> the closest replica but has to wait for all replicas to receive and reply to
> the write. This adds a latency of 16 μs/mi based on the distance to the
> farthest replica.
> 
> When stable writes are done to the MDS or to a single replica (if allowed by
> the appropriate layout flag), it is the responsibility of the receiving node
> to propagate the written data stably, before replying to the client.
> 
> 
> 
> asynchronously propagate the updates. However, once a COMMIT is replied to,
> all replicas must reflect the writes that have been done, and these data
> must have been committed to stable storage on all replicas.
> 
> 
> 
>     * A client which has outstanding unstable writes made to single node (MDS
>     or DS) MUST do all reads from that same node.
>     * When writes are flushed to the server, for example to implement,
>     close-to-open semantics, a COMMIT must be done by the client to ensure
>     that up-to-date written data will be available irrespective of the
>     particular replica read.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Sat, Mar 12, 2016 at 2:57 AM, Benny Halevy < [email-masked] > wrote:
> 
> 
> 
> 
> 
> 
> On Mar 12, 2016 04:06, "Rick Macklem" < [email-masked] > wrote:
> > 
> > Benny Halevy wrote:
> > > 
> > > 
> > > 
> That is, a clustered server needs to mimic as much as possible the functional
> behavior of a monolithic server. With that in mind, the newly written data
> is visible to all clients in a monolithic server regardless if its
> stable_how value. The application level relies on that for close-to-open
> semantics. This implies that in a clustered server configuration with
> multiple DSs, the MDS must make sure all mirrored copies will return the new
> data before the WRITE reply is sent.
> 
> There could be multiple ways to ensure that. One is to actually write the new
> data to all mirrors. But for example, in a shared storage architecture the
> MDS might only need to invalidate the caches in the DSs.
> 
> Benny
> 
> 
> > 
> > > Benny
> > > 
> > > > 
> > > > replication and, therefore, won't use mirrors. I just thought it would
> > > > be a
> > > > nice
> > > > feature to support.)
> > > > 
> > > > rick
> > > > 
> > > > > Benny
> > > > > 
> > > > > 
> > > > > 

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309

---

From: Rick Macklem <[email-masked]>
Date: Sat, 12 Mar 2016 20:58:21 -0500 (EST)

David Noveck wrote:
>    that
Yes. And unsurprisingly David has expressed it in a much clearer way than I did.

It might be nice to mention that the server still has the option of making the
data visible on all replicas (and committed to stable storage on all replicas)
for the case of the unstable write, indicated by returning STABLE (actually FILE_SYNC
or DATA_SYNC) in the write reply.

rick

---

From: Rick Macklem <[email-masked]>
Date: Sat, 12 Mar 2016 21:21:32 -0500 (EST)

> to the write.  This adds a latency of 16μs/mi based on the distance to the
We could define an additional flag FF_FLAGS_REPLICATION_PROPAGATED_ON_COMMIT
(there must be a better/shorter name for this, but you get the idea;-)
so that the server can indicate to the clients that it does this.
Without the flag, the replication (but not necessarily commit to stable storage)
would be guaranteed to be completed when the Write RPC replies as Benny suggested
would be the case for a clustered server configuration.

rick

---

From: Benny Halevy <[email-masked]>
Date: Tue, 15 Mar 2016 08:40:52 +0200

On Sat, Mar 12, 2016 at 5:34 PM, David Noveck <[email-masked]> wrote:

> True, but a lot of the deployments will not be clustered servers.  In
> fact, in many cases in which one is replicating data, the two replicas will
> be far enough apart that a natural disaster is unlikely to simultaneously
> affect both.
>
That's true, though there is no provisioning at the moment for high-latency
mirroring such that
data is propagated to remote DSs asynchronously.

> Since it isn't, I'd like us to consider how we might avoid a requirement
> to immediately propagate writes to all replicas.  Given that there may be
> good reasons to have some of these hundreds or thousands of miles away from
> the others, I think we want to avoid a situation in which the client writes
> to the closest replica but has to wait for all  replicas to receive and
> reply to the write.  This adds a latency of 16μs/mi based on
> the distance to the farthest replica.
>
> In the corresponding cases in which unstable writes are done, the
> receiving node does not have any such obligation, although it may choose to
The behavior with regards to visibility semantics would lay somewhere
in between the state where the writing client has dirty data in its page
cache and the state where
the data is stable and visible too all clients.

In the former case, the data is visible to processes on that client but to
no other client.
In the latter case, once the data is written unstablely to the MDS or DS,
still all the procs on this client can see the new data but other clients
may or may not see it, depending what DS they choose to read from.
After COMMIT, all clients must be able to see the new data.

How about caching semantics in this "twilight zone"?
Can a client that have read old data from a yet-to-be-updated DS and cached
it determine that the data
is stale post COMMIT?  Specifically, what is the mechanism for representing
that in the change attribute?

Benny

>> > > > > on stable storage under any failure scenario. Bare in mind that
>> the
>> > > > > client
>> > > > > will mark
>> > > > > the dirty page as clean in this case and is free to drop it out
>> of its
>> > > > > data
>> > > > Now, if you are saying that the DS may reply STABLE to indicate
>> that all
>> > > > mirrors
>> > > > --> Without the FF_FLAGS_WRITE_ONE_MIRROR flag, the client will
>> need to
>> > > > write to

---

From: Rick Macklem <[email-masked]>
Date: Tue, 15 Mar 2016 20:58:57 -0400 (EDT)

Benny Halevy wrote:
> 
> 
> On Sat, Mar 12, 2016 at 5:34 PM, David Noveck < [email-masked] > wrote:
> 
> 
> 
> 
> 
> In the former case, the data is visible to processes on that client but to no
> other client.
Good point. I would have said that the "change" attribute must change once the
data is visible on all DS mirrors. If all clients see the new data after the
writer does a COMMIT then that is when the change attribute should be updated
to tell clients that their cached data is stale.
--> This implies that the MDS needs to know when all mirrors are updated.

This leads me to a question w.r.t. the current draft...
    How does the current draft address this? It requires the client
    to write to all mirrors, but doesn't seem to require any action done to
    the MDS by the client?
    --> So how does the MDS know when to update Change for a case such as
        NFSv3 DS servers being written?

rick

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 15 Mar 2016 21:26:03 -0400

>         NFSv3 DS servers being written?

How is that in scope for a pNFS layout document? Like all other such
drafts, it is entirely up to the MDS how it implements the control
channel.