Subject: Re: [nfsv4] Possible change to Flex Files layout

From: Thomas Haynes <[email-masked]>
Date: Tue, 18 Jul 2017 09:16:13 +0000

On Mar 12, 2016, at 4:34 PM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:

> Aside for the stability of written data there is the requirement for visibility.

Or at least potential requirements as to visibility.

> That is, a clustered server needs to mimic as much as possible the functional behavior of a monolithic server.

True, but a lot of the deployments will not be clustered servers.  In fact, in many cases in which one is replicating data, the two replicas will be far enough apart that a natural disaster is unlikely to simultaneously affect both.

In any case, pNFS does not strive to precisely emulate clustered server semantics.  Instead, it seeks to provide many of the benefits of clustered operation, while allowing requirements that depend on the clients being aware of the distributed  nature of the arrangement.  Use of LAYOUTCOMMIT is an example.

>With that in mind, the newly written data is visible to all clients in a monolithic server regardless if its stable_how value. The application level relies on that for close-to-open semantics.

True.

> This implies that in a clustered server configuration with multiple DSs, the MDS must make sure all mirrored copies will return the new data before the WRITE reply is sent.

It would imply that if an MDS with multiple DS's were a clustered server configuration.

Since it isn't, I'd like us to consider how we might avoid a requirement to immediately propagate writes to all replicas.  Given that there may be good reasons to have some of these hundreds or thousands of miles away from the others, I think we want to avoid a situation in which the client writes to the closest replica but has to wait for all  replicas to receive and reply to the write.  This adds a latency of 16μs/mi based on the distance to the farthest replica.

I'm thinking we need something like the following to deal with this issue:

When stable writes are done to the MDS or to a single replica (if allowed by the appropriate  layout flag), it is the responsibility of the receiving node to propagate the written data stably, before replying to the client.

In the corresponding cases in which unstable writes are done, the receiving node does not have any such obligation, although it may choose to asynchronously propagate the updates.  However, once a COMMIT is replied to, all replicas must reflect the writes that have been done, and these data must have been committed to stable storage on all replicas.

In order to avoid situations in which stale data is read from replicas to which writes have not been propagated:

  *   A client which has outstanding unstable writes made to single node (MDS or DS) MUST do all reads from that same node.
  *   When writes are flushed to the server, for example to implement, close-to-open semantics, a COMMIT must be done by the client to ensure that up-to-date written data will be available irrespective of the particular replica read.

Better late than never, draft-11 will encompass Rick’s change for FF_FLAGS_WRITE_ONE_MIRROR, your change above, and my change for error handling in this case.

---

From: David Noveck <[email-masked]>
Date: Tue, 18 Jul 2017 08:24:35 -0400

Cool.

On Tue, Jul 18, 2017 at 5:16 AM, Thomas Haynes <[email-masked]>
wrote:

>
> On Mar 12, 2016, at 4:34 PM, David Noveck <[email-masked]> wrote:
>
> > That is, a clustered server needs to mimic as much as possible the
> functional behavior of a monolithic server.
>
> True, but a lot of the deployments will not be clustered servers.  In
> fact, in many cases in which one is replicating data, the two replicas will
> be far enough apart that a natural disaster is unlikely to simultaneously
> affect both.
>
> In any case, pNFS does not strive to precisely emulate clustered server
> semantics.  Instead, it seeks to provide many of the benefits of clustered
> operation, while allowing requirements that depend on the clients being
> aware of the distributed  nature of the arrangement.  Use of LAYOUTCOMMIT
> is an example.
>
> >With that in mind, the newly written data is visible to all clients in a
> monolithic server regardless if its stable_how value. The application level
> relies on that for close-to-open semantics.
>
> True.
>
> > This implies that in a clustered server configuration with multiple DSs,
> the MDS must make sure all mirrored copies will return the new data before
> the WRITE reply is sent.
>
> Since it isn't, I'd like us to consider how we might avoid a requirement
> to immediately propagate writes to all replicas.  Given that there may be
> good reasons to have some of these hundreds or thousands of miles away from
> the others, I think we want to avoid a situation in which the client writes
> to the closest replica but has to wait for all  replicas to receive and
> reply to the write.  This adds a latency of 16μs/mi based on
> the distance to the farthest replica.
>
> When stable writes are done to the MDS or to a single replica (if allowed
> by the appropriate  layout flag), it is the responsibility of the receiving
> node to propagate the written data stably, before replying to the client.
>
> In the corresponding cases in which unstable writes are done, the
> receiving node does not have any such obligation, although it may choose to
> asynchronously propagate the updates.  However, once a COMMIT is replied
> to, all replicas must reflect the writes that have been done, and these
> data must have been committed to stable storage on all replicas.
>
>    - A client which has outstanding unstable writes made to single node
>    (MDS or DS) MUST do all reads from that same node.
>    - When writes are flushed to the server, for example to implement,
>    close-to-open semantics, a COMMIT must be done by the client to ensure that
>    up-to-date written data will be available irrespective of the particular
>    replica read.
>
>
>
>
>
> Better late than never, draft-11 will encompass Rick’s change
> for FF_FLAGS_WRITE_ONE_MIRROR, your change above, and my change for error
> handling in this case.
>
>
>
>
>