Subject: Re: [nfsv4] Detecting loss of connection with RPC-over-RDMA

From: Tom Talpey <[email-masked]>
Date: Tue, 2 Feb 2016 10:52:04 -0500

On 1/31/2016 4:19 PM, William Allen Simpson wrote:
> On 1/27/16 6:28 PM, David Noveck wrote:
>> Since the client-sde transport presumably has logic to hold the
>> request pending if
>> there are zero credits available, it is probably a better use of
>> resources for him to reserve
>> the last credit for use by ping-type functionality, i.e.something that
>> will be responded to
>> right away if the server is neither dead nor a zombie.
>>
> True.  The client could refrain from using the last credit.
>
> But a better idea might be to measure anticipated round-trip time,
> same as we do with TCP.  We have well-known theory and algorithms.
>
> Or we could do both....  Refrain from using the last credit until
> the SRTT from the longest outstanding transaction has expired.

Catching up a bit here, I will say in response to Chuck's original
message that I think a transport-level ping using RDMA Write or RDMA
Read is problematic from a basic layering perspective, because even
if it doesn't fail, it conveys no information about the upper layer
status. There are a number of quirks in using zero-length or zero-valued
handles, as well, that make the approach far from universal.

>
>>  > would be appropriate for documentation as
>>> part of the RPC-over-RDMA standards.
>>
>> It doesn't seem like this fits in RFC5666bis if there is no current
>> implementation.
>>
> Agreed.  Let's stick to only what is currently done.

Right, it would be a new extension and therefore not a matter for
RFC5666bis.

>> I think this might make sense as an extension to Version One.  You
>> could define a new RDMA_PING
>> header type and describe how the client could deal with servers that
>> either do or don't support the extension.
>>
> No, a new field value for pinging must be Version Two.  There's no
> known response in Version One to a rejected value other than error
> and drop connection.

I look on this differently. If it's a requirement that each RPC
transport provide a keepalive mechanism, then let's approach it as
such. In the absence of this, an NFS-layer approach should be
considered. But neither question is fundamentally RDMA-specific.

Tom.

---

From: David Noveck <[email-masked]>
Date: Tue, 2 Feb 2016 11:57:24 -0500

> But neither question is fundamentally RDMA-specific.

Not fundamentally, but there is a necessary connection with RDMA.

RDMA transports have the property, which no others do, so far as I know,
that sending a simple keep-alive, or are-you-alive, can result in connection
loss.  Somehow, the RPC-over-RDMA protocol needs to deal with that
issue.

On Tue, Feb 2, 2016 at 10:52 AM, Tom Talpey <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Chuck Lever <[email-masked]>
Date: Tue, 2 Feb 2016 11:59:19 -0500

> On Feb 2, 2016, at 10:52 AM, Tom Talpey <[email-masked]> wrote:
> 

The RDMA Write mechanism was indeed proposed as a connection
keep-alive probe. Using upper level pings with a minute-long
timeout for this purpose has proven problematic on both
TCP and RDMA transports.

There are several NFS mechanisms that might be well-served by
faster detection of connection loss:

- restoring workloads faster after network partition or
server restart

- reconnecting sooner so a larger population of open and
locked files can be recovered during a time-limited grace
period

- reconnecting sooner so NFSv4.1 servers can terminate
their grace period sooner

- reconnecting to a replicated service sooner

I'm not aware of a requirement for RPC transports to provide
keep-alive and dead connection detection, but certainly NFS
would benefit if there was such a requirement.

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Tue, 2 Feb 2016 14:22:29 -0500

> faster detection of connection loss:

Fast-fail is definitely a desirable behavior in many situations,
and I agree. However, I don't think connection loss is the cause,
rather it's one way to detect failure. The issue in my view is
that it's *only* one aspect, and shouldn't be oversold/promoted
as the solution. One rather obvious thing it can't detect is
failure of the server itself, or its backing storage or filesystem.

issue.

I disagree that this is at all unique to RDMA. TCP connections can
silently fail, and a keepalive (or any other message) will result in
a timeout or a prompt RST, killing the connection. And of course
when iWARP is the RDMA transport, being layered on TCP means it
will have the exact same root cause.

It is true that RDMA connections can fail for other reasons, for
example invalid memory handles, base/bounds violations, etc. But
these would occur only as a result of bugs in upper layers. A
keepalive design should never even consider such a condition.

Bottom line, I believe an NFS-layer keepalive, perhaps using the
NFS4_NULL procedure, may be a worthwhile undertaking. But a
transport-only approach is flawed by design and should not be
part of the architecture.

Tom.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 4 Feb 2016 10:41:43 -0500

> On Feb 2, 2016, at 2:22 PM, Tom Talpey <[email-masked]> wrote:
> 

Exactly, connection loss detection is part of higher
level failure recovery. The problem we are trying to
mitigate is that it can sometimes take unreasonably long
for the transport layer to see that a connection has gone
stale.

And I agree, this appears to be an issue with every
connection-oriented transport we are familiar with.

> will have the exact same root cause.

A TCP RST occurs only after the client attempts to retransmit
_and_ the other end is available to respond. If the client
TCP retransmit timeout backs off exponentially, it can take
a while before the client sees any response from the server.

> keepalive design should never even consider such a condition.

Like TCP, it appears that RDMA requires active traffic for
a sender to get confirmation that the remote is no longer
responding and/or the connection is stale.

> part of the architecture.

Upper Layer ping is problematic for any transport that
limits the number of outstanding RPC requests.

As you know well, Linux write traffic is bursty. When
the VFS finally decides to write, it dumps a lot of
writes onto the filesystem at once. The Linux NFS client
turns this into a long stream of NFS WRITE requests. The
first N of these immediately get RDMA credits, and the
remaining ones go into the backlog queue waiting for
the first N to complete.

There is a window after all the credited requests are
sent successfully and before the server has responded
to any of them. During this time, if the server crashes,
the client will wait patiently for a response, and not
send any new requests.

Now suppose an NFS client Upper Layer has a mechanism to
generate pings when it notices there are outstanding
requests that have not seen a response recently. This NFS
client will kick off the RPC ping request, but the request
is now stuck in the backlog queue behind the NFS WRITEs.
No new network activity is emitted. A NULL is ineffective
in this common scenario.

Dave's point is that if the client were to ignore the
credit limit and send the NULL anyway, on an otherwise
healthy connection, it is likely it would overrun the
server's receive buffers and prove needlessly fatal to
the connection.

Thus currently the upper layer has no reliable way to
generate additional traffic (in particular, a NULL
request) on an RDMA transport. Reserving a credit would
be one way to ensure that a NULL can always be sent, or
a specification could insist that servers keep an extra
receive buffer posted for such occasions. Using an RDMA
Write is another way to avoid the credit limit.

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Thu, 4 Feb 2016 15:00:37 -0500

> a while before the client sees any response from the server.

Indeed, and that's what I meant by "timeout or prompt RST". Just to
throw another log on the fire, remember that RST is not reliably
delivered, so even if the peer resets, a timeout can ensue.

> limits the number of outstanding RPC requests.

Well sure, but that doesn't mean it's the responsibility of the
lower layer. I believe it was suggested earlier in this thread that
one credit should be reserved for such a ping, that is, the sender
should never consume all its credits for operations that might
pend on the server. It's certainly something to discuss.

SMB Direct takes this approach, by the way. It has a "response
requested" bit which requires the peer to reply, perhaps with
an empty packet if it has nothing to send. The bit is used
for critical situations such as impending timeouts and credit
shortages. The peer never consumes its last credit for anything
else.

> server's receive buffers and prove needlessly fatal to

And for the record, I agree. But, for clients which do
consume all their credits, yet still want to probe, it's
arguable that they got themselves into the situation.
This true of any credit situation, once you spend it all.

> be one way to ensure that a NULL can always be sent, or

By "the" upper layer, you mean NFS, right? I don't see how
such a statement is universally applicable.

> receive buffer posted for such occasions.

This could be stated in an update to RFC5667, I guess. It's
only a recommendation if made by the RPC/RDMA layer.

> Using an RDMA

But, this is where we disagree. RDMA Write is unacknowledged,
therefore it does not guarantee a prompt transport failure.
And, as we just discussed, RDMA only probes the transport, it
makes no request of the NFS or backend layers.

Tom.

---

From: David Noveck <[email-masked]>
Date: Fri, 5 Feb 2016 13:12:42 -0500

> But, for clients which do

No dispute from me on that point.  Using that last credit so that
the server has one more WRITE to work on is definitely a mistake.

>  I guess. It's

True but I hope that something could be written that might be
helpful in alerting implementers to this trap.

---

From: Chuck Lever <[email-masked]>
Date: Fri, 5 Feb 2016 15:08:54 -0500

Hi Tom-

Thanks as always for helping me clarify my thoughts.

> On Feb 4, 2016, at 3:00 PM, Tom Talpey <[email-masked]> wrote:
> 

I don't understand.

Credits are managed entirely in the transport layer. They
are not visible to the upper layer at all. What would an
upper layer binding have to say about a resource that is
not controlled by or exposed to it?

Since RPC-over-RDMA is responsible for reserving a credit,
it will have to be told when that credit can be used.
Either we say in advance "only for RPC NULL," or an API
is exposed that allows the ULP to choose when to use it
(like SMB Direct).

The former case is transparent to the upper layer, and
therefore has some appeal. But what happens when a ULP
tries to use the reserved credit and there is already an
outstanding use of it? What if there is only a single
credit on offer between two peers, and that credit is
already in use?

> therefore it does not guarantee a prompt transport failure.

That is a good argument not to use RDMA Write. What about
RDMA Read or some other acknowledged operation? And here
we may be stepping into the realm of adding the use of
another abstract RDMA operation to RPC-over-RDMA.

> makes no request of the NFS or backend layers.

That's correct, only connection health is probed. The upper
layer is still responsible for more sophisticated problem
detection and recovery.

Our upper layer implementation designs appear to rely on
transports to tell them when a connection is gone. The
specific mechanics of loss detection are unique to each
underlying network technology. Thus I claim connection
loss detection is a problem for the transport, not the
upper layer, to deal with.

Once a connection loss is detected, the client endpoint is
able to take more aggressive recovery steps. The sooner it
can do this, the better.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Sat, 6 Feb 2016 08:59:31 -0500

> are not visible to the upper layer at all.

WhIch seems to lead pretty directly, when combined with common
ways of doing writes, to the main problem we are discussing.

> What would an

Since RFC5667 or a possible successor is the only document
where use of RPC/RDMA specifically for NFS is discussed, I
think it has to say something about the issue.  I think we can
say something helpful, even if RFC2119 terms are not used.

For example:

Often NFS client implementations will schedule a large number of
write requests, making it likely that all available credits will be
consumed by these requests.  When the connection is in that
state, probing requests such as NFS NULL would not have access
to an available credit, making it difficult to determine promptly whether
the server is still functional.

While the specifics of the interface between the client and the
transport are outside the scope of this specification, implementers
would be well advised to take steps to avoid the situation discussed
above.  Some possibilities worth considering:

   - Providing special handling  for NFS NULL requests, only allowing those
   requests to consume the last requests.
   - Accepting a priority indication when requests are made allowing only
   high-priority requests to consume the last credit.

> it will have to be told when that credit can be used.

I'd hope that implementers can decide on their own, if given appropriate
non-normative guidance.

> Either we say in advance "only for RPC NULL,"

MIght be a problem for any non-NFS users of RPC/RDMA.

> or an API
> ( like SMB Direct).

I don't think we're in the business of defining APIs.

> therefore has some appeal.

Not to me.  I think it is kind of hacky but that' choice should be up
to the implementer.

> But what happens when a ULP
> outstanding use of it?

If I were the implementer, that request would wait for a credit and would
be queued ahead of all requests that did not have high priority.

> What if there is only a single

Every connection starts out that way but if the first response still only
has a single credit, I would probably leave the high-priority feature off
until the first response that showed multiple credits.

> And here

We would be, but it could be done in a compatible way and might be a valid
extension within the scope of Version
One, given the kind f extensibility model we have been discussing for
RFC5666bis,

> Thus I claim connection

> able to take more aggressive recovery steps. The sooner it

Agree.

---

From: Tom Talpey <[email-masked]>
Date: Sat, 6 Feb 2016 11:30:28 -0500

> another abstract RDMA operation to RPC-over-RDMA.

On this point, what you propose is still problematic. An RDMA Read
would still potentially go through a regular best-effort timeout
at the RDMA layer, which is to say it's no more prompt.

For another, it would require the server to advertise a memory
region to the client, which is something the RPC/RDMA layer has
no real business deciding (in my opinion).

Finally, there's the problem of how to tell the client what that
handle is, which will require a protocol extension of some kind,
which is out of scope for 5666bis. It can't be "zero" and also
can't be "well known", due to the validity checking requirements
of RDMA.

If fast-fail is a desirable property (and in many cases I agree
it is), we should address it as a proper protocol matter, and
decide if it's an NFS or RPC or RDMA or TCP layer thing. Personally,
I feel the higher in the stack, the better.

---

From: Tom Talpey <[email-masked]>
Date: Sat, 6 Feb 2016 11:46:21 -0500

> ways of doing writes, to the main problem we are discussing.

Agreed. There's nothing in the protocol that makes credits not visible
to an upper layer. It's analogous to the sender checking if there is
sufficient TCP window space, or if a UDP message would overflow the
output queue. Purely a local matter whether it's exposed.

>       * Providing special handling  for NFS NULL requests, only allowing
>         those requests to consume the last requests.
>       * Accepting a priority indication when requests are made allowing
>         only high-priority requests to consume the last credit.

I think this is a fine suggestion as implementation advice. But it's
a little odd to discuss NFS NULL. To my knowledge there is no specific
requirement for NFS NULL behavior in the various NFS specs, just that
it exists and should be replied to. I do not think 5667 should make
"upward" requirements.

---

From: David Noveck <[email-masked]>
Date: Sat, 6 Feb 2016 12:54:46 -0500

> But it's
> a little odd to discuss NFS NULL.

I'm not sure why.  People have been discussing using NFS NULL as a probe
request for a while and I don't see that there is anything odd about that.

> To my knowledge there is no specific
> it exists and should be replied to.

That's what people are looking for:  a request that needs to be replied to,
and for which there is a reasonable expectation that if the server is
functioning,
will be quickly responded to.  I could rewrite this to avoid mentioning NFS
NULL
specifically, but the goal here is to tell implementers that they can get
an NFS
NULL (or a very simple COMPOUND)  responded to.   I think the trick in
writing this without using the words "NFS NULL" is that the goal is letting
the people
who will actually write the code know that they should be able to send an
NFS NULL
this way and that is hard to do without actually using those words. I think
we will eventually
say "e.g., NFS NULL".

---

From: Tom Talpey <[email-masked]>
Date: Sat, 6 Feb 2016 14:02:38 -0500

> request for a while and I don't see that there is anything odd about that.

Odd to discuss it in the context of 5666bis and 5667.

> will be quickly responded to.  I could rewrite this to avoid mentioning
> NFS NULL
> specifically, but the goal here is to tell implementers that they can
> get an NFS
> NULL (or a very simple COMPOUND)  responded to.

I'd recommend that approach. For example, there might be reasons to
use a different request to probe. Off the top of my head, I could
imagine a GETATTR of the root filehandle, or an empty COMPOUND with
just a PUTFH or something. The idea being, that it would actually
make a request instead of being an empty procedure.

To reiterate - I'm perfectly ok with pursuing a keepalive probe for
the purpose of fast-fail detection. I just don't agree that it should
be discussed in context of the RDMA transport.

Tom.

    I think the trick in
> writing this without using the words "NFS NULL" is that the goal is
> letting the people
> who will actually write the code know that they should be able to send
> an NFS NULL
> this way and that is hard to do without actually using those words. I
> think we will eventually
> say "e.g., NFS NULL".
>
> On Sat, Feb 6, 2016 at 11:46 AM, Tom Talpey <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>              to an available credit, making it difficult to determine
>         promptly
>              whether
>     [email-masked] <mailto:[email-masked]>

---

From: Chuck Lever <[email-masked]>
Date: Sun, 7 Feb 2016 14:05:34 -0500

> On Feb 6, 2016, at 8:59 AM, David Noveck <[email-masked]> wrote:
> 
> 	• Providing special handling  for NFS NULL requests, only allowing those requests to consume the last requests.
> 	• Accepting a priority indication when requests are made allowing only high-priority requests to consume the last credit.

Fair enough, I don't have an objection to making this
implementation quality advice, as long as there are no
server-side requirements to make it work (and so far
there don't seem to be).

> I'd hope that implementers can decide on their own, if given appropriate non-normative guidance.
> 
> If I were the implementer, that request would wait for a credit and would be queued ahead of all requests that did not have high priority.

That makes this mechanism unreliable (in the sense that
the upper layer is again stuck waiting for an RPC timeout
in some cases).

> Every connection starts out that way but if the first response still only has a single credit, I would probably leave the high-priority feature off until the first response that showed multiple credits.

Ditto: that means there are some cases where reserving a
single credit for high priority messages works only as
well as what we have now.

> We would be, but it could be done in a compatible way and might be a valid extension within the scope of Version 
> One, given the kind f extensibility model we have been discussing for RFC5666bis,
> 
> > Thus I claim connection

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Sun, 7 Feb 2016 22:24:28 -0500

>  I don't have an objection to making this

Regardless, of the specifics of the implementation advice,
the important point is
that the text has to avoid giving the implementer the impression that the
architecture
somehow constrains him to adopt an FCFS approach to using credits, with all
the
negative consequences that you drew our attention to.  I think Tom's
formulation of this point could be helpful.

> > If I were the implementer, that request would wait for a credit and
would be queued ahead of all requests that did not have high priority.

> the upper layer is again stuck waiting for an RPC timeout

It seems to me to be  "unreliable" only in the sense that the upper layer
can always
choose to do something that doesn't make sense and the transport cannot
always
protect him from the consequences of his mistakes.

As Tom noted:
> Using that last credit so that

So now we have a situation in which the client can issue a NFS NULL (or
other
probing) request.  If  that returns within a reasonable time, everything is
now cool
in that we realize the server is functioning.

If he doesn't get a response, it is probably time to break the connection
after
waiting some number of seconds.  What does not make sense is to retry that
probing request.  If the system is up, you will get the response from the
first
request since this is a reliable transport.  There is no benefit in having
two
outstanding, as you wait for the time to expire for you to conclue to kill
the connection.

So trying to have two credits outstanding for probing requests when one
would serve
the purpose is a mistake.

Even if the upper layer decides for some reason to have multiple such
requests
outstanding, there is no sense in him waiting for the second of those to be
sent.
What he should be waiting for is a response to the first one sent, or the
expiration
of the time that indicates it is no longer worth waiting for the server to
respond.

If the implementation is waiting for a specific probing requests to be
sent, there is
nothing that the transport can or should do to accommodate him.

> well as what we have now.

Yes, but every such case is one in which the server, after connecting, is
never
able/willing to offer more than a single credit.  In that case, reserving a
credit for
probing requests would not allow the client to do anything other than
probing
requests.

On Sun, Feb 7, 2016 at 2:05 PM, Chuck Lever <[email-masked]> wrote:

> >       • Providing special handling  for NFS NULL requests, only allowing
> >       • Accepting a priority indication when requests are made allowing
> > If I were the implementer, that request would wait for a credit and
> would be queued ahead of all requests that did not have high priority.
>
> > Every connection starts out that way but if the first response still
> only has a single credit, I would probably leave the high-priority feature
> off until the first response that showed multiple credits.
>
> > We would be, but it could be done in a compatible way and might be a
> valid extension within the scope of Version