Subject: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

From: Benny Halevy <[email-masked]>
Date: Wed, 06 Jul 2011 18:37:51 +0300

In the recent Bakeathon in Ann Arbor we noted the following interesting behavior:

The server provided a layout w/o NFL4_UFLG_COMMIT_THRU_MDS.

The client sent WRITEs to the DSs with stable=FILE_SYNC4,
expecting the DS to take all the way to the MDS to sync both data
and metadata.

However, the data server currently does not support that and naively
returned a successful WRITE reply with committed==DATA_SYNC4

Although this currently strictly violates the protocol regarding the WRITE
operation, the client turned around and did the right thing which is sending
LAYOUTCOMMIT to the MDS, the exact thing it would have done if it sent
a UNSTABLE or DATA_SYNC4 WRITE that returned DATA_SYNC4.

Now, the question is should we just allow NFSv4.1 Data Server
to return DATA_SYNC4 for WRITEs with stable=FILE_SYNC4, a feature
that would help implementing pNFS servers using logic that the pNFS
clients need to implement anyway for the common use case?

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 06 Jul 2011 12:10:51 -0400

> clients need to implement anyway for the common use case?

No.
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 6 Jul 2011 16:51:20 +0000

I agree with Trond.  The DS should behave correctly.
It is nice of the client to do what it did but should
not count on it.

Spencer

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Benny Halevy
> Sent: Wednesday, July 06, 2011 8:38 AM
> To: NFSv4
> Subject: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)
> 
> The client sent WRITEs to the DSs with stable=FILE_SYNC4, expecting the DS
> to take all the way to the MDS to sync both data and metadata.
> 
> operation, the client turned around and did the right thing which is
> sending LAYOUTCOMMIT to the MDS, the exact thing it would have done if it
> sent a UNSTABLE or DATA_SYNC4 WRITE that returned DATA_SYNC4.
> 
> Now, the question is should we just allow NFSv4.1 Data Server to return
> DATA_SYNC4 for WRITEs with stable=FILE_SYNC4, a feature that would help
> implementing pNFS servers using logic that the pNFS clients need to
> implement anyway for the common use case?
> 
> Benny
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Thu, 07 Jul 2011 07:02:25 +0300

>> clients need to implement anyway for the common use case?

"No", we should not, or "No" the client does not have to implement
that logic anyway?, or "No" I just want to make the server's life harder
because <...>

I don't want to put words in your mouth, care to elaborate?

---

From: Benny Halevy <[email-masked]>
Date: Thu, 07 Jul 2011 07:18:34 +0300

This approach is somewhat inconsistent with section 13.6
where we say:

   GETATTR and SETATTR MUST be directed to the metadata server.  In the
   case of a SETATTR of the size attribute, the control protocol is
   responsible for propagating size updates/truncations to the data
   servers.  In the case of extending WRITEs to the data servers, the
   new size must be visible on the metadata server once a LAYOUTCOMMIT
   has completed (see Section 12.5.4.2).

This text acknowledges the difficulty of the data server with propagating
extending WRITE-induced size changes back to the MDS over the control protocol.
With the errata, where the client sends an extending WRITE with FILE_SYNC4
there will be no LAYOUTCOMMIT, hence the conflict.

If the data server returned FILE_SYNC4 I can live with the assumption that the
file metadata is updated globally, however, if the server implementation does
not do that, we can achieve the same goal using building blocks we already
have. So I ask again, why raise the bar for server implementations when
data servers are already a special case in the protocol and clients come ready
out of the box with functionality that renders this hard-to-implement feature
redundant?

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 07 Jul 2011 00:21:32 -0400

> I don't want to put words in your mouth, care to elaborate?

No, the server should not be allowed to override a client request just
because someone can't be arsed to make it comply. The semantics for
WRITE are already defined, and have been for several decades.

If you want to define new semantics for WRITE that change the meaning of
the 'stable' flag, then define a new WRITE_MINUS operation. Just don't
expect an implementation of that for our client.

pNFS is already full enough with bizarre protocol deviations (witness
the stateid sequence crap that occurs on layout recalls), I see no
reason to agree to yet another.

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 07 Jul 2011 00:29:44 -0400

RFC5661, section 18.32.3

   The client specifies with the stable parameter the method of how the
   data is to be processed by the server.  If stable is FILE_SYNC4, the
   server MUST commit the data written plus all file system metadata to
   stable storage before returning results.  This corresponds to the
   NFSv2 protocol semantics.  Any other behavior constitutes a protocol
   violation.

This has defined WRITE behaviour since the 'stable' flag was introduced
in RFC1813.

Trond

---

From: Benny Halevy <[email-masked]>
Date: Thu, 07 Jul 2011 07:42:12 +0300

But pNFS presents a different model of a clustered server.
Assuming we will one day standardize the back-end control protocol
one would be able to mix and match metadata servers and data servers
from different vendors.  In such a world, I would like to allow
the implementation of the simplest possible standalone data servers
that will be responsible only for the data and metadata they store locally.
The MDS is still responsible for maintaining the global state and directing
the state on the DSs for the purpose of implementing file-wide changes (e.g.
truncate) and enforcing global policies (such as security or fencing).
This model is inspired by the OSD model that has proven to be very highly scalable.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 07 Jul 2011 09:33:59 -0400

> that will be responsible only for the data and metadata they store locally.

How does FILE_SYNC4 break the above model? It doesn't make the data
server responsible for global file state, it makes it responsible for
_propagating_ global file state changes such as a size change.
The DS will usually want to do that sort of propagation anyway in order
to avoid inconsistencies between global and local file state in case of
an MDS crash (relying on layoutcommit is insufficient, since the client
can crash too).

Trond

NetApp

---

From: Benny Halevy <[email-masked]>
Date: Thu, 07 Jul 2011 16:49:00 +0300

> _propagating_ global file state changes such as a size change.

I'm not saying that FILE_SYNC4 breaks the model, I'm just worried about the
implementation complexity of having to have a metadata update channel from the
DS back to the MDS.  And the opportunity to relax that constraint by using a
practically mandatory-to-implement (to get decent performance) client feature.

> an MDS crash (relying on layoutcommit is insufficient, since the client

In case the MDS crashes and the client did not send a reclaiming LAYOUTCOMMIT
and the lease time is over I can see it recovering the global file state
by polling the affected DSs.  I'm not sure that having the DSs pushing that
state to the MDS as a very helpful thing.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 07 Jul 2011 09:51:17 -0400

> practically mandatory-to-implement (to get decent performance) client feature.

SETATTR is hard?

NetApp

---

From: Benny Halevy <[email-masked]>
Date: Thu, 07 Jul 2011 16:57:38 +0300

>> practically mandatory-to-implement (to get decent performance) client feature.

It should be a "conditional" SETATTR that allows only extending the file
since the client may have been writing to multiple DSs in parallel and
the SETATTRs may arrive at the MDS out of order.  Not to say, the possibility
of adding a truncate to the mix, (though for which I presume the MDS
will recall outstanding WRITE layouts to make sure WRITEs will not race
with the truncating SETATTR)

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 07 Jul 2011 10:00:39 -0400

On Thu, 2011-07-07 at 16:57 +0300, Benny Halevy wrote: 
> the SETATTRs may arrive at the MDS out of order.

I repeat: this is hard? That's all the layoutcommit does too.

>   Not to say, the possibility

Nope. That's only blocks and objects. No reason for a 'files' server to
recall the layout in this case.

NetApp

---

From: Benny Halevy <[email-masked]>
Date: Thu, 07 Jul 2011 17:16:01 +0300

On 2011-07-07 17:00, Trond Myklebust wrote:
> Nope. That's only blocks and objects. No reashoon for a 'files' server to

So how would you solve the following race:

1. DS recv from client:		WRITE (offset=1000, length=1, stable=FILE_SYNC4)
2. MDS recv from client:	SETATTR (length=0)
3. DS send to MDS:		DS_SETATTR (length=1001)
4. MDS send to DS:		SETATTR (length=0)
5. DS recv from MDS:		SETATTR (length=0)
6. MDS recv from DS:		DS_SETATTR (length=1001)

With no way to properly serialize the operations we may end up
with different lengths on the MDS and DS.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 07 Jul 2011 10:27:11 -0400

> with different lengths on the MDS and DS.

The above is an ordering problem. Serialisation is not necessary to
solve it.

If the data server keeps a local change attribute and sends that as part
of the DS_SETATTR request and the SETATTR reply, then the MDS can
trivially work out the order of the operations above.

NetApp

---

From: "Benny Halevy" <[email-masked]>
Date: Fri, 8 Jul 2011 04:55:10 +0000

I'm pretty sure we'll find more and more corner cases requiring special care which will only be revealed in full once we specify the standard back end control protocol hence I'm pushing on a minimalistic approach allowing to implement effective servers and clients with the minimum amount of effort.
This made NFSv3 so successfully ubiqtous. Being open, simple, and effective.  The fight we had around the forgetful client model and the need for layoutcommit, while annoying, it symbolizes a drive for minimalism - let's see how we can achieve our goal with the least amount of messages. Not that I think its necessarily effective in all cases, that's a different issue, but merely going through the exercise helped to shake out quite a few issues with the protocol.
-----Original Message-----
From: Trond Myklebust <[email-masked]>
Date: Thu, 07 Jul 2011 10:27:11 
To: Benny Halevy<[email-masked]>
Cc: NFSv4<[email-masked]>
Subject: Re: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

NetApp

---

From: Boaz Harrosh <[email-masked]>
Date: Fri, 8 Jul 2011 11:35:16 -0700

> reason to agree to yet another.

Right so now you want to break perfectly working & simple generic code
and have the code play STD lawyer. I mean by accident the code was behaving
"properly"? No it is no accident it is just the simple thing to do.

Now you want to put new, more code that does:
	Remember what I sent ; // I did not care before
	See what the server returned;
	Send layout commit only if Server is Behaving;
	Other wise trash the data and return error to the Application.

	Today we just have one stage:
	Send layout commit if return != FILE_SYNC4

So you are the one introducing "bizarre".

When writing through MDS the full (Since NFSv2) backward compatible
semantics are preserved to the full. But a DS write is not an NFS write. A DS write
is a special new pNFS operation, that draws from the mechanics of the old operation.
For example: The DS filehandles need not be persistent since you only allowed to use
the one returned from layout_get after an open. Actually the MDS can revoke them.
Two DS write/reads are done without set/get attrs.. etc Just see all the if DS places
in the code.

The solution to above problem is simple. The new Errata should say:

	DS writes can only be UNSTABLE or DATA_SYNC4. FILE_SYNC4 is only sent to MDS.

	(FILE_SYNC4 Has no meaning for DS because stripping may write to many DSs
	 Which one will you send it to)

How you guys keep insisting on breaking the files_layout to the point that it will
never fly, never cease to amaze me. You go to new heights every time. Now you are
willing to put in more complicated and error prone code, just to make sure every
body's life is more complicated.

Free Life
Boaz

---

From: Suchit Kaura <[email-masked]>
Date: Fri, 8 Jul 2011 11:53:07 -0700

I agree with Benny and Boaz, we spent a long time coming to agreement on how the client  can work with a symmetric file layout based server and an asymmetric one at the IETF meeting.

A very big goal for pNFS is to enable a common client that works with different servers. If Linux client works only with Netapp server and other companies need to change 
it to work with their servers we are back to the way it was prior to having a standard?

I think it is a better model if communication between DFS and MDFS is not forced on a write, in a way it defeats the whole purpose of pNFS if on every write DFS needs to send message to MDFS.

Regards,
Suchit

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Boaz Harrosh
Sent: Friday, July 08, 2011 11:35 AM
To: Trond Myklebust
Cc: NFSv4
Subject: Re: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

On 07/06/2011 09:21 PM, Trond Myklebust wrote:
>> that logic anyway?, or "No" I just want to make the server's life 
>> harder because <...>
>>
> If you want to define new semantics for WRITE that change the meaning 
> of the 'stable' flag, then define a new WRITE_MINUS operation. Just 
> don't expect an implementation of that for our client.
> 

Right so now you want to break perfectly working & simple generic code and have the code play STD lawyer. I mean by accident the code was behaving "properly"? No it is no accident it is just the simple thing to do.

When writing through MDS the full (Since NFSv2) backward compatible semantics are preserved to the full. But a DS write is not an NFS write. A DS write is a special new pNFS operation, that draws from the mechanics of the old operation.
For example: The DS filehandles need not be persistent since you only allowed to use the one returned from layout_get after an open. Actually the MDS can revoke them.
Two DS write/reads are done without set/get attrs.. etc Just see all the if DS places in the code.

How you guys keep insisting on breaking the files_layout to the point that it will never fly, never cease to amaze me. You go to new heights every time. Now you are willing to put in more complicated and error prone code, just to make sure every body's life is more complicated.

Free Life
Boaz

---

From: "Myklebust, Trond" <[email-masked]>
Date: Fri, 8 Jul 2011 12:07:15 -0700

> From: Suchit Kaura [mailto:[email-masked]]
> Sent: Friday, July 08, 2011 2:53 PM
> To: Boaz Harrosh; Myklebust, Trond
> Cc: NFSv4
> Subject: RE: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)
> 
> I agree with Benny and Boaz, we spent a long time coming to agreement
> on how the client  can work with a symmetric file layout based server
> and an asymmetric one at the IETF meeting.

...and the consensus agreement at the NetApp-hosted IETF meeting that
_all_ of you attended was that the client should not have to do anything
more if the FILE_SYNC4 mode was used. No COMMIT is required, and no
LAYOUTCOMMIT is required.

Now you're all coming back 6 months later and complaining because our
client implements exactly what you agreed to?

Sorry, but it's too late to bitch now. The release has already occurred
and the cat can't be stuffed back into the bag.

---

From: Suchit Kaura <[email-masked]>
Date: Fri, 8 Jul 2011 13:29:21 -0700

I don't think that was the agreement, it was explicitly said that we will see DATA_SYNC to DFS as the general case. In fact there was suggestion of how we could determine server type and change behavior on whether to do layout commit or not but that was resolved with client will send DATA_SYNC to the DFS as the general case.
I think we all understand the general motivation on always sending FILE_SYNC4 to data server. It just seems against the basic principle of standard and open client.

Also, please be careful with the idioms, we all know some, I do not want this to turn into a mud-slinging match. I have a lot of respect for you personally. 

Regards,
Suchit

-----Original Message-----
From: Myklebust, Trond [mailto:[email-masked]] 
Sent: Friday, July 08, 2011 12:07 PM
To: Suchit Kaura; Boaz Harrosh
Cc: NFSv4
Subject: RE: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

> and an asymmetric one at the IETF meeting.

...and the consensus agreement at the NetApp-hosted IETF meeting that _all_ of you attended was that the client should not have to do anything more if the FILE_SYNC4 mode was used. No COMMIT is required, and no LAYOUTCOMMIT is required.

Now you're all coming back 6 months later and complaining because our client implements exactly what you agreed to?

Sorry, but it's too late to bitch now. The release has already occurred and the cat can't be stuffed back into the bag.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Fri, 8 Jul 2011 14:03:07 -0700

> Sent: Friday, July 08, 2011 4:29 PM
> To: Myklebust, Trond; Boaz Harrosh
> Cc: NFSv4
> I don't think that was the agreement, it was explicitly said that we
> will see DATA_SYNC to DFS as the general case. In fact there was
> suggestion of how we could determine server type and change behavior
on
> whether to do layout commit or not but that was resolved with client
> will send DATA_SYNC to the DFS as the general case.
> I think we all understand the general motivation on always sending
> FILE_SYNC4 to data server. It just seems against the basic principle
of
> standard and open client.
> 

You're rewriting history to suit your argument:

   http://www.ietf.org/mail-archive/web/nfsv4/current/msg09050.html

See the bit that states:
Note that a client can always demand FILE_SYNC4 or DATA_SYNC4 in WRITE's
arguments.  Also note that specifying these stability levels may
adversely
impact performance.

This is what you and everybody else agreed to. It is what we have coded
to we're not changing that.

Trond

---

From: Suchit Kaura <[email-masked]>
Date: Fri, 8 Jul 2011 14:14:33 -0700

> Also note that specifying these stability levels may adversely impact performance.
And you are making sure that this happens for servers other than Netapp, that was not the agreement we had. History by definition cannot be rewritten, there were more than just I in that forum and it seems like you are the only one claiming consensus on this aspect, the rest of us seem to remember history as it is. I understand once again your motivations, just don't see how something with the intent and history of Linux allows such plagiarism by one company.

Regards,
Suchit
-----Original Message-----
From: Myklebust, Trond [mailto:[email-masked]] 
Sent: Friday, July 08, 2011 2:03 PM
To: Suchit Kaura; Boaz Harrosh
Cc: NFSv4
Subject: RE: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

on
of

See the bit that states:
Note that a client can always demand FILE_SYNC4 or DATA_SYNC4 in WRITE's arguments.  Also note that specifying these stability levels may adversely impact performance.

This is what you and everybody else agreed to. It is what we have coded to we're not changing that.

Trond

---

From: "Myklebust, Trond" <[email-masked]>
Date: Fri, 8 Jul 2011 14:25:15 -0700

> Sent: Friday, July 08, 2011 5:15 PM
> > Also note that specifying these stability levels may adversely
impact
> performance.
> And you are making sure that this happens for servers other than
> Netapp, that was not the agreement we had. History by definition
cannot
> be rewritten, there were more than just I in that forum and it seems
> like you are the only one claiming consensus on this aspect, the rest
> of us seem to remember history as it is. I understand once again your
> motivations, just don't see how something with the intent and history
> of Linux allows such plagiarism by one company.
>

Thanks...
 
Now go back and read the link that I sent you, and that was sent out on
this very mailing list, and that is archived there, in google and that
is part of the RFC5661 errata. If you didn't think that reflected the
consensus, then you've had ample time to tell us all. As I said earlier,
it is too late to bitch now.

Trond

> Note that a client can always demand FILE_SYNC4 or DATA_SYNC4 in
> WRITE's arguments.  Also note that specifying these stability levels
> may adversely impact performance.
> 
>

---

From: Boaz Harrosh <[email-masked]>
Date: Fri, 8 Jul 2011 14:34:12 -0700

> client implements exactly what you agreed to?

No! actually the client is implementing what I said. It currently receives a
DATA_SYNC4 and goes on to send a layout_commit to MDS.

That is until one of you guys will send a patch to break that. But right now
it does the good thing

> and the cat can't be stuffed back into the bag.

No bitching. Just describing the good behavior we have now and having the STD
up to speed with current details.

Boaz

---

From: "Welch, Brent" <[email-masked]>
Date: Fri, 8 Jul 2011 23:36:46 +0000

I'm catching up on this thread and wanted to make sure we can understand what we agree on, if anything.

The IETF meeting that produced the minutes cited here:
http://www.ietf.org/mail-archive/web/nfsv4/current/msg09050.html

lay out a behavior matrix for client and server with 6 rows representing the cross product of setting 
  NFS4_UFLG_COMMIT_THRU_MDS
and the three return codes from the Data Server
  DATA_SYNC4
  FILE_SYNC4
  UNSTABLE4

The table defines what the client's behavior is with respect to COMMIT and LAYOUTCOMMIT.

That thread does not discuss why the client should choose DATA_SYNC4, FILE_SYNC4, or UNSTABLE4 when sending WRITE to the Data server.  Which brings us to the topic of this thread: It is observed that clients sometimes send FILE_SYNC4 to the Data server.  What does this mean, and what should the client and server do about it, if anything.  (If this thread is about something different - "sorry")

My understanding is:

A. Generally the client sends UNSTABLE4 for the stable_how Write parameter.  However, there are cases (like O_SYNC) where the Linux client sends FILE_SYNC4 to the data servers.  This is historical from 4.0 (pre pNFS) and sort of falls out of the current code.  I'm not reading Linux code right now, so I could be wrong here.  The big picture as I understand it is that the client does try for UNSTABLE4 writes when it can, but uses FILE_SYNC4 in other cases.  There is debate about the meaning of FILE_SYNC4 when sent to a Data Server.

B. The client currently allows the file server to return DATA_SYNC4 as the stable_how committed return parameter, even though the client passed in FILE_SYNC4.  Basically, there is logic in the client to react to DATA_SYNC4 results no matter what it asked for.  There is debate about this client behavior because the NFS4.1 spec in section 18.32.3 says that the server must return >= the stability.  In short, the spec says that servers shouldn't return DATA_SYNC4 when asked for FILE_SYNC4.

My summary of the positions are:

P1. "Don't send FILE_SYNC4 to data servers".  There is an argument that this is either (a) meaningless or (b) non-optimal. I don't feel strongly, but the fact that there are now Data servers and Metadata Servers, then it makes sense that Data servers can certainly deal with DATA_SYNC4 but may or may not be able to deal with FILE_SYNC4.

=> One possible resolution to this is to amend the Spec to state that a Data server is allowed to return DATA_SYNC4 in response to a write that asked for FILE_SYNC4.  This would imply to the client that it still needs to commit metadata using COMMIT or LAYOUTCOMMIT (depending on the NFS4_UFLG_COMMIT_THRU_MDS flag).

=> Another solution is to have Data servers return an error when given FILE_SYNC4, which will motivate client implementations not to send that flag.

=> Another solution is to modify the client to flip FILE_SYNC4 into DATA_SYNC4 inside the pNFS layout driver.

P2. "Client's should error out if they get DATA_SYNC4 when they asked for FILE_SYNC4".  The argument here is that this is what the spec says, or rather the spec says this shouldn't happen, so the client may want to bail out if it does.

I think the same three possible solutions apply to this position as well.

My personal opinion is that sending DATA_SYNC4 to Data Servers and FILE_SYNC4 to Metadata servers is consistent with the protocol spec as it stands, and in the spirit of what pNFS is trying to do by separating data and metadata.

If you go this route, then it seems like the spec should be amended to call out this distinction for clarity.  Specifically, I would be OK with saying "MUST NOT send FILE_SYNC4 to data servers".  This removes two of the 6 rows from the matrix in the original decision.  I suppose another possibility is that the NFS server can either honor FILE_SYNC4 or return an error.  So, the client would have to deal with the different cases, which is more code.  I doubt you want that.

I think the trick of having the data server return DATA_SYNC4 to FILE_SYNC4 is a cute trick, but it does violate the WRITE command semantics, which I think is what Trond is objecting to.  If you allow the DATA_SYNC4 return to FILE_SYNC4, then you are basically admitting that the client should probably have sent DATA_SYNC4, unless it is lucky enough to talk to a data server that knows how to deal with FILE_SYNC4.  This approach is more flexible, and it sounds like it is what is implemented today (or, recently).  But, it also feels like this approach is exploiting a quirk of the current implementation and it might be better to beef things up (Spec and Implementation) so they agree and so that the important cases are handled well.

In short, sending DATA_SYNC4 to Data servers and FILE_SYNC4 to Metadata servers seems consistent and reasonable, and matches the spec.
	
	Brent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Myklebust, Trond
Sent: Friday, July 08, 2011 2:25 PM
To: Suchit Kaura; Harrosh, Boaz
Cc: NFSv4
Subject: Re: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

impact
cannot

Thanks...

Trond

---

From: "Myklebust, Trond" <[email-masked]>
Date: Fri, 8 Jul 2011 16:58:38 -0700

> From: Welch, Brent [mailto:[email-masked]]
> Sent: Friday, July 08, 2011 7:37 PM
> To: Myklebust, Trond; Suchit Kaura; Harrosh, Boaz
> Cc: NFSv4
> I'm catching up on this thread and wanted to make sure we can
> understand what we agree on, if anything.
> 
> lay out a behavior matrix for client and server with 6 rows
> representing the cross product of setting
> That thread does not discuss why the client should choose DATA_SYNC4,
> FILE_SYNC4, or UNSTABLE4 when sending WRITE to the Data server.  Which
> brings us to the topic of this thread: It is observed that clients
> sometimes send FILE_SYNC4 to the Data server.  What does this mean,
and
> what should the client and server do about it, if anything.  (If this
> thread is about something different - "sorry")
> 
> My understanding is:
> 
> A. Generally the client sends UNSTABLE4 for the stable_how Write
> parameter.  However, there are cases (like O_SYNC) where the Linux
> client sends FILE_SYNC4 to the data servers.  This is historical from
> 4.0 (pre pNFS) and sort of falls out of the current code.  I'm not
> reading Linux code right now, so I could be wrong here.  The big
> picture as I understand it is that the client does try for UNSTABLE4
> writes when it can, but uses FILE_SYNC4 in other cases.  There is
> debate about the meaning of FILE_SYNC4 when sent to a Data Server.

Correction: there seems to be a myth running around that clients are
obliged to use FILE_SYNC4 when dealing with O_SYNC. This is not the
case.

The only reason why a client might prefer FILE_SYNC4 in the case of
O_SYNC is if the write() syscall results in a request that can be
satisfied by a single WRITE rpc request. In that case, we do use
FILE_SYNC4 since UNSTABLE4 forces you to send a COMMIT and a
LAYOUTCOMMIT.

> B. The client currently allows the file server to return DATA_SYNC4 as
> the stable_how committed return parameter, even though the client
> passed in FILE_SYNC4.  Basically, there is logic in the client to
react
> to DATA_SYNC4 results no matter what it asked for.  There is debate
> about this client behavior because the NFS4.1 spec in section 18.32.3
> says that the server must return >= the stability.  In short, the spec
> says that servers shouldn't return DATA_SYNC4 when asked for
> FILE_SYNC4.
> 
> P1. "Don't send FILE_SYNC4 to data servers".  There is an argument
that
> this is either (a) meaningless or (b) non-optimal. I don't feel
> strongly, but the fact that there are now Data servers and Metadata
> Servers, then it makes sense that Data servers can certainly deal with
> DATA_SYNC4 but may or may not be able to deal with FILE_SYNC4.
> 
> => One possible resolution to this is to amend the Spec to state that
a
> Data server is allowed to return DATA_SYNC4 in response to a write
that
> asked for FILE_SYNC4.  This would imply to the client that it still
> needs to commit metadata using COMMIT or LAYOUTCOMMIT (depending on
the
> NFS4_UFLG_COMMIT_THRU_MDS flag).
>
> => Another solution is to have Data servers return an error when given
> FILE_SYNC4, which will motivate client implementations not to send
that
> flag.
> 
> => Another solution is to modify the client to flip FILE_SYNC4 into
> DATA_SYNC4 inside the pNFS layout driver.
> 
> P2. "Client's should error out if they get DATA_SYNC4 when they asked
> for FILE_SYNC4".  The argument here is that this is what the spec
says,
> or rather the spec says this shouldn't happen, so the client may want
> to bail out if it does.
> 
> My personal opinion is that sending DATA_SYNC4 to Data Servers and
> FILE_SYNC4 to Metadata servers is consistent with the protocol spec as
> it stands, and in the spirit of what pNFS is trying to do by
separating
> data and metadata.
> 
> If you go this route, then it seems like the spec should be amended to
> call out this distinction for clarity.  Specifically, I would be OK
> with saying "MUST NOT send FILE_SYNC4 to data servers".  This removes
> two of the 6 rows from the matrix in the original decision.  I suppose
> another possibility is that the NFS server can either honor FILE_SYNC4
> or return an error.  So, the client would have to deal with the
> different cases, which is more code.  I doubt you want that.
> 
> I think the trick of having the data server return DATA_SYNC4 to
> FILE_SYNC4 is a cute trick, but it does violate the WRITE command
> semantics, which I think is what Trond is objecting to.  If you allow
> the DATA_SYNC4 return to FILE_SYNC4, then you are basically admitting
> that the client should probably have sent DATA_SYNC4, unless it is
> lucky enough to talk to a data server that knows how to deal with
> FILE_SYNC4.  This approach is more flexible, and it sounds like it is
> what is implemented today (or, recently).  But, it also feels like
this
> approach is exploiting a quirk of the current implementation and it
> might be better to beef things up (Spec and Implementation) so they
> agree and so that the important cases are handled well.
> 
> In short, sending DATA_SYNC4 to Data servers and FILE_SYNC4 to
Metadata
> servers seems consistent and reasonable, and matches the spec.

In that case, the solution on the client would have to be to ignore pNFS
for all single WRITE requests. There is already a way to achieve this:
use the mdsthreshold attribute.

Trond

---

From: Suchit Kaura <[email-masked]>
Date: Fri, 8 Jul 2011 18:36:46 -0700

Brent,

I think your proposal sounds very reasonable. If we do not send FILE_SYNC4 to data servers then there is no issue as we will not have to for every write send an update to MDS.
It will help with achieving the goal of pNFS to have more parallelism for writes.

Regards,
Suchit

-----Original Message-----
From: Welch, Brent [mailto:[email-masked]] 
Sent: Friday, July 08, 2011 4:37 PM
To: Myklebust, Trond; Suchit Kaura; Harrosh, Boaz
Cc: NFSv4
Subject: RE: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

My understanding is:

	
	Brent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Myklebust, Trond
Sent: Friday, July 08, 2011 2:25 PM
To: Suchit Kaura; Harrosh, Boaz
Cc: NFSv4
Subject: Re: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

impact
cannot

Thanks...
 
Now go back and read the link that I sent you, and that was sent out on this very mailing list, and that is archived there, in google and that is part of the RFC5661 errata. If you didn't think that reflected the consensus, then you've had ample time to tell us all. As I said earlier, it is too late to bitch now.

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 08 Jul 2011 21:49:21 -0400

> It will help with achieving the goal of pNFS to have more parallelism for writes.

I don't.

Show that there are real generic performance benefits and I'll accept
the optimisations. Until then this is all hearsay, and I don't give a
damn.

Trond

NetApp