Subject: Re: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

From: "Matt W. Benjamin" <[email-masked]>
Date: Fri, 8 Jul 2011 14:53:53 -0400 (EDT)

Hi,

This certainly seems consistent with the goals I thought we were striving for, such as strong independence of the data servers.  To be honest, this thread has increased my worry that folks are placing design obligations on a 'control protocol' which -should- in principle be soluble with layouts alone in a valid pnfs architecture.  

Matt

----- "Boaz Harrosh" <[email-masked]> wrote:

> 
> When writing through MDS the full (Since NFSv2) backward compatible
> semantics are preserved to the full. But a DS write is not an NFS
> write. A DS write
> is a special new pNFS operation, that draws from the mechanics of the
> old operation.
> For example: The DS filehandles need not be persistent since you only
> allowed to use
> the one returned from layout_get after an open. Actually the MDS can
> revoke them.
> Two DS write/reads are done without set/get attrs.. etc Just see all
> the if DS places
> in the code.
> 
> The solution to above problem is simple. The new Errata should say:
> 
> 	DS writes can only be UNSTABLE or DATA_SYNC4. FILE_SYNC4 is only sent
> to MDS.
> 
> 	(FILE_SYNC4 Has no meaning for DS because stripping may write to many
> DSs
> 	 Which one will you send it to)
> 
> How you guys keep insisting on breaking the files_layout to the point
> that it will
> never fly, never cease to amaze me. You go to new heights every time.
> Now you are
> willing to put in more complicated and error prone code, just to make
> sure every
> body's life is more complicated.
> 
> Free Life
> Boaz
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

-- 

Matt Benjamin

The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: Boaz Harrosh <[email-masked]>
Date: Fri, 8 Jul 2011 14:45:07 -0700

On 07/08/2011 11:53 AM, Matt W. Benjamin wrote:
> Hi,
> 
> This certainly seems consistent with the goals I thought we were
> striving for, such as strong independence of the data servers.  To be
> honest, this thread has increased my worry that folks are placing
> design obligations on a 'control protocol' which -should- in
> principle be soluble with layouts alone in a valid pnfs architecture.
> 

currently the code does the right thing. When the guys will send in
a patch that breaks it. We can just make sure that the server keeps
doing the same. Now in that case the client is suppose to do error
handling, and refusing to do the right thing and just send a layout_commit
already, it will have to rewrite to MDS, before returning an error to
the application. We should watch like an hawk the new implementation.

Now what will happen is that the sync IO will suck even more (Written
twice on the wire) and someone is bound to complain. But the servers
should do what they must do. Eventually it will be *reverted* to the
right way ;-)

Boaz
>> When writing through MDS the full (Since NFSv2) backward
>> compatible semantics are preserved to the full. But a DS write is
>> not an NFS write. A DS write is a special new pNFS operation, that
>> draws from the mechanics of the old operation. For example: The DS
>> filehandles need not be persistent since you only allowed to use 
>> the one returned from layout_get after an open. Actually the MDS
>> can revoke them. Two DS write/reads are done without set/get
>> attrs.. etc Just see all the if DS places in the code.
>> 
>> DS writes can only be UNSTABLE or DATA_SYNC4. FILE_SYNC4 is only
>> sent to MDS.
>> 
>> (FILE_SYNC4 Has no meaning for DS because stripping may write to
>> many DSs Which one will you send it to)
>> 
>> How you guys keep insisting on breaking the files_layout to the
>> point that it will never fly, never cease to amaze me. You go to
>> new heights every time. Now you are willing to put in more
>> complicated and error prone code, just to make sure every body's
>> life is more complicated.
>> 
>> Free Life Boaz _______________________________________________ 
>> nfsv4 mailing list [email-masked]

---

From: "Myklebust, Trond" <[email-masked]>
Date: Fri, 8 Jul 2011 15:27:16 -0700

> -----Original Message-----
> From: Boaz Harrosh [mailto:[email-masked]]
> Sent: Friday, July 08, 2011 5:45 PM
> To: Matt W. Benjamin
> Cc: NFSv4; Myklebust, Trond
> Subject: Re: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)
> 

We code to a spec, not to particular implementations. If we fail to code to that spec, then we will fix it, but otherwise, it is up to the servers to make sure that they also conform to that spec.

The point of using FILE_SYNC4 is to avoid a bunch of extra round trips to the DS and MDS when all the client wants to do is send a single WRITE, otherwise, we use FILE_UNSTABLE4. If I was really optimizing for NetApp filers, as Suchit appears to believe, then I'd always send FILE_UNSTABLE4 and force the client to do a COMMIT, and a LAYOUTCOMMIT...

Duh...

---

From: Sandeep Joshi <[email-masked]>
Date: Fri, 8 Jul 2011 17:06:16 -0700

It will benefit all servers, won't it?

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Myklebust, Trond
Sent: Friday, July 08, 2011 3:27 PM
To: Boaz Harrosh; Matt W. Benjamin
Cc: NFSv4
Subject: Re: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

Duh...

---

From: "Myklebust, Trond" <[email-masked]>
Date: Fri, 8 Jul 2011 17:18:06 -0700

> From: Sandeep Joshi [mailto:[email-masked]]
> Sent: Friday, July 08, 2011 8:06 PM
> To: Myklebust, Trond; Boaz Harrosh; Matt W. Benjamin
> Cc: NFSv4
> Subject: RE: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)
> 

The intention was that the current code (which is the _same_ as the
ordinary NFSv3 and NFSv4.0 code) would benefit all servers. It is
usually easier to optimize write+commit on the server rather than on the
client. As I said in my reply to Brent, I'd expect servers that don't
like FILE_SYNC4 to the DS to use mdsthreshold to tell the client when it
is in everyone's interest to send the short writes to the MDS.

DATA_SYNC is almost always a lose: it requires an extra _synchronous_
COMMIT (if sent to the MDS) or LAYOUTCOMMIT (if sent to the DS) in order
to be effective. Why would I want to use that?

Trond

> We code to a spec, not to particular implementations. If we fail to
> code to that spec, then we will fix it, but otherwise, it is up to the
> servers to make sure that they also conform to that spec.
> 
> The point of using FILE_SYNC4 is to avoid a bunch of extra round trips
> to the DS and MDS when all the client wants to do is send a single
> WRITE, otherwise, we use FILE_UNSTABLE4. If I was really optimizing
for
> NetApp filers, as Suchit appears to believe, then I'd always send
> FILE_UNSTABLE4 and force the client to do a COMMIT, and a
> LAYOUTCOMMIT...
> 
> Duh...
> 
>

---

From: "Welch, Brent" <[email-masked]>
Date: Sat, 9 Jul 2011 01:25:14 +0000

It is helpful to clarify the small write vs. large write cases.  My understanding is that the rationale for sending FILE_SYNC4 to the data servers is for small writes that are going to be covered by a single WRITE operation.  This will reduce the client-to-server RPCs to one, but it implies some out-of-band communication between the DS and the MDS to ensure, for example, that a GETATTR to the MDS will see the correct file size.  So, really there is some second message.  With DATA_SYNC4, then yes, there is an additional client-server RPC in the LAYOUTCOMMIT to the MDS.  But, that message eliminates the need for the out-of-band DS to MDS communication.  Seems like 6 on one hand, half a dozen on the other.  This leaves the "send the data to the MDS instead of using pNFS".  This may also result in two messages as the MDS has to relay the write command to the DS, assuming that the Data Server is the true home for the data.

So, my impression is that if the data for the small write lives on the DS, and the MDS manages the metadata, then there is fundamentally a message to the MDS and a message to the DS.  With DATA_SYNC4 and LAYOUTCOMMIT, those two messages originate from the client and go to two different servers.  With FILE_SYNC4 there must be a second message from the DS to the MDS.  With writing to the MDS, then there is a message from the MDS to the DS.  It isn't clear that any of these three methods is strictly superior when handling a small write.

I think it is most important to clarify how COMMIT and LAYOUTCOMMIT work, and I think there is agreement on that.  Optimizing small writes is interesting, but it isn't a "no brainer" that DATA_SYNC4 and LAYOUTCOMMIT is somehow the worst choice.  I think, again, it is more important that we start with correct implementations, and then folks can compare how the servers perform.  It may be that we evolve some additional hint similar (but I think different) than NFS4_UFLG_COMMIT_THRU_MDS.  The "I like LAYOUTCOMMIT" flag is almost the opposite of NFS4_UFLG_COMMIT_THRU_MDS, and could perhaps be interpreted that way.  Is that worth exploring?

	Brent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Myklebust, Trond
Sent: Friday, July 08, 2011 5:18 PM
To: Sandeep Joshi; Harrosh, Boaz; Matt W. Benjamin
Cc: NFSv4
Subject: Re: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

Trond

for

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 08 Jul 2011 21:47:18 -0400

> I think it is most important to clarify how COMMIT and LAYOUTCOMMIT work, and I think there is agreement on that.  Optimizing small writes is interesting, but it isn't a "no brainer" that DATA_SYNC…

        Optimizing small writes is interesting, but it isn't a "no
        brainer" that DATA_SYNC4 and LAYOUTCOMMIT is somehow the worst
        choice.

Neither is it a no brainer that write through MDS is worse.

I'm not accepting _any_ optimisations until we have real performance
data.
When people have real data and are ready to publish the performance
improvements shown by specific optimisations, and we will judge those
optimisations on their merits. In the mean time, the defaults for pNFS
are the same as those in the no-pNFS case.

Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Suchit Kaura <[email-masked]>
Date: Fri, 8 Jul 2011 18:55:38 -0700

I think it is important to consider the case where a single file is spread across many data servers. Every write to each data server sending a message to metadata server vs the model where we have many messages to data servers followed by one message to metadata server from the client. 

I think there was a proposal from Netapp server folks to remove the layout commit altogether in the IETF meeting we created this errata. I understand the motivation for symmetric servers and would want to find a way to make sure that the protocol is optimized for that, I just think we should keep asymmetric servers in mind and have the  layout commit sent to mdfs. In any case I would like to find clarity on details. 

Trond, my apologies again if I have understood the reason for this change incorrectly. I hope we can figure out a way to make it a standard client that works across all servers efficiently. It will help promote pNFS and in some ways without that all of us lose...

Regards,
Suchit 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Welch, Brent
Sent: Friday, July 08, 2011 6:25 PM
To: Myklebust, Trond; Sandeep Joshi; Harrosh, Boaz; Matt W. Benjamin
Cc: NFSv4
Subject: Re: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

	Brent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Myklebust, Trond
Sent: Friday, July 08, 2011 5:18 PM
To: Sandeep Joshi; Harrosh, Boaz; Matt W. Benjamin
Cc: NFSv4
Subject: Re: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

> It will benefit all servers, won't it?

The intention was that the current code (which is the _same_ as the ordinary NFSv3 and NFSv4.0 code) would benefit all servers. It is usually easier to optimize write+commit on the server rather than on the client. As I said in my reply to Brent, I'd expect servers that don't like FILE_SYNC4 to the DS to use mdsthreshold to tell the client when it is in everyone's interest to send the short writes to the MDS.

DATA_SYNC is almost always a lose: it requires an extra _synchronous_ COMMIT (if sent to the MDS) or LAYOUTCOMMIT (if sent to the DS) in order to be effective. Why would I want to use that?

Trond

> > handling, and refusing to do the right thing and just send a 
> > layout_commit already, it will have to rewrite to MDS, before 
> > returning an error
to
for

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 08 Jul 2011 22:05:30 -0400

> Trond, my apologies again if I have understood the reason for this change incorrectly. I hope we can figure out a way to make it a standard client that works across all servers efficiently. It will…

Prove that using DATA_SYNC4 is better overall than using the current
scheme (and not _only_ for the bluearc server), and I'm willing to
change the behaviour. However I'm seriously sick and tired of people
telling us how to optimise the client without any hard proof to back up
their assertions.

Numbers talk, everything else walks...

Trond

NetApp

---

From: Suchit Kaura <[email-masked]>
Date: Fri, 8 Jul 2011 19:20:25 -0700

I think that is a reasonable way to think ( a difficult task for us for anything other than BA servers). 

Thus, I figure it would help if you clarify the current optimal client behavior or the one that will come next push :-)

Does the client ever send anything other than FILE_SYNC4 to the data server now? If so in which cases is that done from the client. I hope this is a reasonable request, if we can understand what is the default current behavior we can design tests for different cases and study performance for them as you suggest. It can help generate data for the systems we have access to and that can be shared. 

Regards,
Suchit

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Friday, July 08, 2011 7:06 PM
To: Suchit Kaura
Cc: Welch, Brent; Sandeep Joshi; Harrosh, Boaz; Matt W. Benjamin; NFSv4
Subject: RE: [nfsv4] FILE_SYNC4 to the DS (Errata ID: 2751)

> Trond, my apologies again if I have understood the reason for this change incorrectly. I hope we can figure out a way to make it a standard client that works across all servers efficiently. It will…

Prove that using DATA_SYNC4 is better overall than using the current scheme (and not _only_ for the bluearc server), and I'm willing to change the behaviour. However I'm seriously sick and tired of people telling us how to optimise the client without any hard proof to back up their assertions.

Trond

> > > > striving for, such as strong independence of the data servers.  
> > > > To
> > be
> > > > honest, this thread has increased my worry that folks are 
> > > > placing design obligations on a 'control protocol' which 
> > > > -should- in principle be soluble with layouts alone in a valid 
> > > > pnfs
> > architecture.
> > > >
> > >
> > > currently the code does the right thing. When the guys will send 
> > > in a patch that breaks it. We can just make sure that the server 
> > > keeps doing the same. Now in that case the client is suppose to do 
> > > error handling, and refusing to do the right thing and just send a 
> > > twice on the wire) and someone is bound to complain. But the 
> > > servers should do what they must do. Eventually it will be 
> > > *reverted* to the right way ;-)
> > 
> > code to that spec, then we will fix it, but otherwise, it is up to 
> > the servers to make sure that they also conform to that spec.
> > 
> > The point of using FILE_SYNC4 is to avoid a bunch of extra round 
> > trips to the DS and MDS when all the client wants to do is send a 
> > single WRITE, otherwise, we use FILE_UNSTABLE4. If I was really 
> > optimizing
> for

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 08 Jul 2011 22:31:47 -0400

> Does the client ever send anything other than FILE_SYNC4 to the data server now? If so in which cases is that done from the client. I hope this is a reasonable request, if we can understand what is…

As I said previously, the _default_ client behaviour is to use UNSTABLE.
The exception is if the client finds out that it only needs to send a
single WRITE request in order to satisfy an O_SYNC write() syscall, an
fsync() or a close() syscall.
IOW: this is an optimisation that we have found to be useful for the
non-pNFS case. Until someone proves otherwise for the general case, then
we will continue to do this for the pNFS case too.

I'm happy to accept further server hints as to what _it_ believes is
optimal, but for the moment, the only hints I have available to me on
the client are those given by the mdsthreshold attribute. Those
thresholds do not yet have an effect on our client, but I'm willing to
make implementing them a priority if we all agree that they can help to
solve this problem.

For the long term, if we need more hints in order to be effective, then
let's start working now on what we need in order to improve NFSv4.2.

Trond

NetApp

---

From: Benny Halevy <[email-masked]>
Date: Sun, 10 Jul 2011 14:08:47 +0300

> is in everyone's interest to send the short writes to the MDS.

mdsthreshold (threshold4_write_iosize in particular) has no notion of
synchronous vs. asynchronous writes.  Your suggestion makes sense for
single FILE_SYNC4 writes but not necessarily for a bunch of small async
WRITEs optionally followed by a COMMIT (if UNSTABLE) and a LAYOUTCOMMIT.

Benny

>>> layout_commit

---

From: Benny Halevy <[email-masked]>
Date: Sun, 10 Jul 2011 14:16:07 +0300

> Numbers talk, everything else walks...

Note that in my original proposal I requested for no changes in the
client but rather for a relaxation of the protocol requirements for the
data server to allow the server to opt for the async path.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Sun, 10 Jul 2011 15:15:53 -0400

> WRITEs optionally followed by a COMMIT (if UNSTABLE) and a LAYOUTCOMMIT.

OK. I've read the above paragraph over and over again and it still makes
absolutely no sense to me.

Why would any sane client split a write up into smaller writes?

Why would any sane client use a synchronous RPC call if it can send an
asynchronous one?

Finally, when does our client send out multiple FILE_SYNC4 writes
instead of using UNSTABLE+COMMIT? Please don't answer O_SYNC, 'cos that
is utter bullcrap that someone has somehow thrown out into the ether
without checking the code.

NetApp

---

From: Benny Halevy <[email-masked]>
Date: Mon, 11 Jul 2011 15:52:08 +0300

> Why would any sane client split a write up into smaller writes?

Sparse, discontiguous writes for example.

> Why would any sane client use a synchronous RPC call if it can send an

True, no reason to do that.

> without checking the code.

O_SYNC!
(Just kiddin' ;-))

At any rate, I don't care specifically about the linux client but
at any client vs. any server.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 11 Jul 2011 14:18:39 -0400

> Sparse, discontiguous writes for example.

OK... There is plenty of space to expand mdsthreshold. If the current
definition does not communicate your server's preferences accurately to
the client, then you are free to propose new hint definitions in 4.2.

In the meantime, 4.1 is stuck with what is already defined in rfc5661.

Trond

NetApp