Subject: [nfsv4] Comment/request for nfsv4.2

From: "Ramanathan, Raghavendran" <[email-masked]>
Date: Thu, 18 Sep 2014 21:18:30 +0000

Hi,

This is a feature request in nfsv4.2.

Along the lines of application hints, may nfsv4.2 also provide server side hints ? I am particularly targeting the NFS4ERR_DELAY error:

In case of delay error, nfsv4.2 server should reply with a string - that gives a hint on why the response is delayed:

For any operation(using GETFH as sample below):

   union GETFH4res switch (nfsstat4 status) {
    case NFS4_OK:
           GETFH4resok     resok4;
    case NFS4ERR_DELAY:

           opaque          delay_why<NFS4_OPAQUE_LIMIT>;
    default:
           void;
   };

The opaque string "delay_why" should be a useful message that explains or hints at the exact problem the server is facing. It need not display the internal or implementation details, but it must give a clear indication of the problem the server is facing. The client may propagate this string to the application, which may choose to display it or discard it.  However, since the client might be re-trying without returning to the application, this message might be simply displayed by the client, instead of the application.  The client must not attempt to parse the string.

Rational:

When a nfs access simply hangs as below, forcing the user to send interrupt:
proto-howler(US):cthon04:ls
^C
In case of DELAY error, we cannot debug even with a packet trace, which has only requests and replies with the DELAY error.

When the above change is implemented, we get a clue using either the command line itself or the packet traces.
It is easier if the user gets a clue as below:
proto-howler(US):cthon04:ls
Nfs server 10.73.28.30 is experiencing delay in accessing user information from NIS, retrying....
...

NIS can be Files/DNS/Ldap(or anything else that may come up)
User information may be anything: group, user, hostname lookup.

Benefit: Easy to debug when we know the problem underlying the given hanging command.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 18 Sep 2014 17:36:04 -0400

On Thu, Sep 18, 2014 at 5:18 PM, Ramanathan, Raghavendran
<[email-masked]> wrote:
> Along the lines of application hints, may nfsv4.2 also provide server side
> hints ? I am particularly targeting the NFS4ERR_DELAY error:
>
>
>
> In case of delay error, nfsv4.2 server should reply with a string – that
> gives a hint on why the response is delayed:
>
>
>
> The opaque string “delay_why” should be a useful message that explains or
> hints at the exact problem the server is facing. It need not display the
> internal or implementation details, but it must give a clear indication of
> the problem the server is facing. The client may propagate this string to
> the application, which may choose to display it or discard it.  However,
> since the client might be re-trying without returning to the application,
> this message might be simply displayed by the client, instead of the
> application.  The client must not attempt to parse the string.
>
>
>
>
>
>
>
> Rational:
>
>
>
> In case of DELAY error, we cannot debug even with a packet trace, which has
> only requests and replies with the DELAY error.
>
>
>
>
>
> Nfs server 10.73.28.30 is experiencing delay in accessing user information
> from NIS, retrying….
>
> …
>
>
>

Can't you use the COMPOUND4res "tag" field for this? As far as I know,
there is nothing preventing a NFSv4.x server from using that to add
debugging information to the wire traces.

Cheers
  Trond

---

From: "Ramanathan, Raghavendran" <[email-masked]>
Date: Thu, 18 Sep 2014 21:41:52 +0000

Trond,

Thank you for the response. As far I know, the tag field is set by the client, and nfs server simply copies it into the response. I was not aware that this field can be used for this return hint. I am open to it though.

Thanks,
rraghav

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, September 18, 2014 2:36 PM
To: Ramanathan, Raghavendran
Cc: [email-masked]
Subject: Re: [nfsv4] Comment/request for nfsv4.2

On Thu, Sep 18, 2014 at 5:18 PM, Ramanathan, Raghavendran <[email-masked]> wrote:
> Along the lines of application hints, may nfsv4.2 also provide server 
> side hints ? I am particularly targeting the NFS4ERR_DELAY error:
>
>
>
> In case of delay error, nfsv4.2 server should reply with a string – 
> that gives a hint on why the response is delayed:
>
>
>
> The opaque string “delay_why” should be a useful message that explains 
> or hints at the exact problem the server is facing. It need not 
> display the internal or implementation details, but it must give a 
> clear indication of the problem the server is facing. The client may 
> propagate this string to the application, which may choose to display 
> it or discard it.  However, since the client might be re-trying 
> without returning to the application, this message might be simply 
> displayed by the client, instead of the application.  The client must not attempt to parse the string.
>
>
>
>
>
>
>
> Rational:
>
>
>
> In case of DELAY error, we cannot debug even with a packet trace, 
> which has only requests and replies with the DELAY error.
>
>
>
>
>
> When the above change is implemented, we get a clue using either the 
> command line itself or the packet traces.
>
> Nfs server 10.73.28.30 is experiencing delay in accessing user 
> information from NIS, retrying….
>
> …
>
>
>

Can't you use the COMPOUND4res "tag" field for this? As far as I know, there is nothing preventing a NFSv4.x server from using that to add debugging information to the wire traces.

Cheers
  Trond

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 18 Sep 2014 17:48:02 -0400

> Thank you for the response. As far I know, the tag field is set by the client, and nfs server simply copies it into the response. I was not aware that this field can be used for this return hint. I…

The text states:

   The definition of the "tag" in the request is left to the
   implementor.  It may be used to summarize the content of the Compound
   request for the benefit of packet-sniffers and engineers debugging
   implementations.  However, the value of "tag" in the response SHOULD
   be the same value as provided in the request.  This applies to the
   tag field of the CB_COMPOUND procedure as well.

Note the use of the "SHOULD" above, which means that you can deviate
from this rule if you have a good reason for it.
I would think that if you simply append your info to whatever tag the
client sends (so that the COMPOUND4res tag contains both pieces of
information), then you will have followed the spirit of the above
rule.

Cheers
  Trond

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 18 Sep 2014 22:14:32 -0400 (EDT)

On Thu, 18 Sep 2014, Trond Myklebust wrote:

> > Thank you for the response. As far I know, the tag field is set by the
> > client, and nfs server simply copies it into the response. I was not
> > aware that this field can be used for this return hint. I am open to
> > it though.
> >
> > Thanks,
> > rraghav
>
> The text states:
>

You'd still have to get buy-in from everybody if you want your client to
be useful with someone else's server.

On the other hand, it seems like having the kernel blat a string onto the
user's tty is a rather large hole in the abstraction barriers, and not
necessarily a good idea.  Such a string might also include terminal
control characters, which can be extremely not-nice to the uesr.  Who
would filter such things out?

All in all, it is far from clear to me that this would be a good idea.

-Ben

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 18 Sep 2014 23:33:27 -0400

> be useful with someone else's server.

Buy in for what? The client can already stuff whatever it wants into
the tag field as long as it is utf-8 encoded. The question here is
about whether or not the server can do the same, to which the answer
is that RFC3530 and RFC5661 do not appear to forbid it.

> would filter such things out?

As the quoted paragraph above implies, COMPOUND tags essentially exist
to allow you to add debugging info to a packet trace. I'm not aware of
anyone currently logging directly to a userspace tty. Try doing so
with RPC calls maxing out the bandwidth on a 40GigE connection, and
I'm sure you'll find that control characters are the least of your
problems.
However if you're sifting through a packet trace and trying to figure
out why the server was doling out NFS4ERR_DELAYs like there was no
tomorrow, then perhaps this might help?

Either way, using tags like this shouldn't require any changes to the
NFS protocol, or to existing compliant implementations.

Trond

---

From: "Ramanathan, Raghavendran" <[email-masked]>
Date: Fri, 19 Sep 2014 04:00:52 +0000

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, September 18, 2014 8:33 PM
To: Benjamin Kaduk
Cc: Ramanathan, Raghavendran; [email-masked]
Subject: Re: [nfsv4] Comment/request for nfsv4.2

On Thu, Sep 18, 2014 at 10:14 PM, Benjamin Kaduk <[email-masked]> wrote:
>> > Thank you for the response. As far I know, the tag field is set by 
>> > the client, and nfs server simply copies it into the response. I 
>> > was not aware that this field can be used for this return hint. I 
>> > am open to it though.
>> >
>> > Thanks,
>> > rraghav
>>
>> The text states:
>>
> You'd still have to get buy-in from everybody if you want your client 
> to be useful with someone else's server.

Buy in for what? The client can already stuff whatever it wants into the tag field as long as it is utf-8 encoded. The question here is about whether or not the server can do the same, to which the answer is that RFC3530 and RFC5661 do not appear to forbid it.

> On the other hand, it seems like having the kernel blat a string onto 
> the user's tty is a rather large hole in the abstraction barriers, and 
> not necessarily a good idea.  Such a string might also include 
> terminal control characters, which can be extremely not-nice to the 
> uesr.  Who would filter such things out?

As the quoted paragraph above implies, COMPOUND tags essentially exist to allow you to add debugging info to a packet trace. I'm not aware of anyone currently logging directly to a userspace tty. Try doing so with RPC calls maxing out the bandwidth on a 40GigE connection, and I'm sure you'll find that control characters are the least of your problems.
However if you're sifting through a packet trace and trying to figure out why the server was doling out NFS4ERR_DELAYs like there was no tomorrow, then perhaps this might help?

RR>> The packet trace analysis is where I needed this information. Pushing it to application is secondary - so that the user may get an idea of what is going on. But like you pointed out there are plenty of snags in pushing the message to the application.
Thanks,
rraghav

Either way, using tags like this shouldn't require any changes to the NFS protocol, or to existing compliant implementations.

Trond

---

From: Benjamin Kaduk <[email-masked]>
Date: Fri, 19 Sep 2014 00:04:00 -0400 (EDT)

Sorry, it appears that no part of my intent was correctly understood.  Let
me try again.

> is that RFC3530 and RFC5661 do not appear to forbid it.

The original proposal involved a requirement that *all* severs insert a
descriptive string when sending NFS4ERR_DELAYs.  In particular the phrase
"it must give a clear indication of the problem the server is facing",
though I suppose the surroundings were only "should", so maybe it was not
intended as a hard requirement.

My point was: why require all servers to return a string if you're only
interested in your client to your server?  If you control both sides,
there are lots of ways to get information across.  A standard mechanism is
most useful for interoperability of different implementations.

In order to get interoperable functionality for including descriptive
strings with NFS4ERR_DELAYs by using the reply tag, the OP would need to
convince other server implementors to use the tags in the same way as
their server/client.  That is the "buy-in" that I claimed would be needed.

> As the quoted paragraph above implies, COMPOUND tags essentially exist

This paragraph of mine was not referring directly to the case where tags
would be used, rather it was referring to the ideas expressed in the
original post.

> I'm sure you'll find that control characters are the least of your

The original proposal was only for NFS4ERR_DELAY returns.  Are you
claiming that such returns happen "all the time" when maxing out the
bandwidth on a 40GigE connection?  If not, I fail to see the relevance of
the amount of traffic which is possible.

The original proposal had:

% The opaque string "delay_why" should be a useful message that explains
% or hints at the exact problem the server is facing. It need not display
% the internal or implementation details, but it must give a clear
% indication of the problem the server is facing. The client may propagate
% this string to the application, which may choose to display it or discard
% it.  However, since the client might be re-trying without returning to the
% application, this message might be simply displayed by the client, instead
% of the application.  The client must not attempt to parse the string.

This bit about "propagate to the application" seems unlikely, as it would
require some out-of-band indication of extended status about file i/o,
i.e., new system calls.  Propagating the string to the end-user is
"easier", since the process performing the i/o (may) have a connected tty,
which can be written to.  The client gets to choose whether to drop the
string, or pass it on, but is forbidden from parsing it.  Any client that
chooses to pass it on would effectively be allowing a remote party to
write arbitrary data to the tty in question, since parsing is forbidden.

My take: that's probably a bad idea.

> tomorrow, then perhaps this might help?

Hmm, maybe you were claiming that the delay return happened "all the
time".  In any case, the client could certainly use rate-limiting to
decide when to discard and when to pass on.

The general sense of my message was that if the original idea of the
original post was probably a bad idea, using a different mechanism to
achieve the same goal is probably still a bad idea.

> NFS protocol, or to existing compliant implementations.

I don't disagree; see my previous paragraph.

-Ben

---

From: Benjamin Kaduk <[email-masked]>
Date: Fri, 19 Sep 2014 00:14:14 -0400 (EDT)

On Fri, 19 Sep 2014, Ramanathan, Raghavendran wrote:

> RR>> The packet trace analysis is where I needed this information.
> Pushing it to application is secondary - so that the user may get an
> idea of what is going on. But like you pointed out there are plenty of
> snags in pushing the message to the application.

Thanks for the clarification.  It seems that maybe I payed insufficient
attention to the "rationale" section of the original post and focused only
on the technical proposal.

-Ben