Subject: [nfsv4] questing about NLM LOCK, CANCEL UNLOCK

From: Olga Kornievskaia <[email-masked]>
Date: Thu, 7 Dec 2017 10:57:00 -0500

Hi folks,

I'm looking for guidance for what are responsibilities of the client
or server to solve the following situation.

Client application is acquiring a blocking lock but shortly after that
application is killed via ctrl-C. NFS client sent out NLM_LOCK but
hasn't gotten a reply yet as ctrl-c arrived. To clean up, the client
is sending CANCEL and UNLOCK to the server.

Server ends up processing CANCEL and UNLOCK first (for whatever reason
one legitimate one could be network re-ordered packets) and server has
no state so it's just sending GRANTED replies back (which seems to be
a legitimate thing to do). Then server processes LOCK and replies to
the client.

Now we are in the situation where lock was granted to a client that
doesn't know it's holding one and it prevents other clients from
grabbing this lock.

Is there a solution or this is broken protocol?

Should it be client's responsibility to notice that it received a LOCK
reply for which it wasn't waiting and always follow up with an UNLOCK?

Thank you.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 7 Dec 2017 10:59:20 -0500

Hi Olga,

NLM isn't an IETF protocol.

Cheers
  Trond

On 7 December 2017 at 10:57, Olga Kornievskaia <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Olga Kornievskaia <[email-masked]>
Date: Thu, 7 Dec 2017 11:10:16 -0500

Ok. I thought that because RFC1813 covers NLM operations that it is.

I will extend this question to the Linux NFS mailing list as the
client implementation I'm interested is Linux.

---

From: [email-masked] (J. Bruce Fields)
Date: Thu, 7 Dec 2017 11:58:38 -0500

> Ok. I thought that because RFC1813 covers NLM operations that it is.

Yeah, I don't see a harm to the occasional NLM question on the v4
working group list.  It's a bit of an orphaned protocol, so there's not
really any other implementation-independent forum.

> client implementation I'm interested is Linux.

But that's fine too.

I thought that LOCK/CANCEL race was one of the motivations for NFSv4,
so...

> >> Is there a solution or this is broken protocol?

... I'd always assumed the protocol was impossible to implement 100%
correctly, though maybe there's some clever solution.

> >> reply for which it wasn't waiting and always follow up with an UNLOCK?

That would be tricky and still not handle all cases, I think.

--b.

---

From: Olga Kornievskaia <[email-masked]>
Date: Thu, 7 Dec 2017 13:00:32 -0500

> correctly, though maybe there's some clever solution.

Is such race still possible with the linux server implementation which
is single threaded and in my testing isn't processing LOCK/CANCEL
out-of-order then?

> That would be tricky and still not handle all cases, I think.

Yes I'm not sure the client can do it. It will have no info about the
lock and not sure how to create an appropriate UNLOCK reply (unless we
keep around lock info of all cancelled locks).

>
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email-masked]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

---

From: "Frank Filz" <[email-masked]>
Date: Thu, 7 Dec 2017 10:56:10 -0800

> > working group list.  It's a bit of an orphaned protocol, so there's
> > not really any other implementation-independent forum.
> >
> Is such race still possible with the linux server implementation which is
single
> threaded and in my testing isn't processing LOCK/CANCEL out-of-order then?
> 
> >> >> Should it be client's responsibility to notice that it received a
> >> >> LOCK reply for which it wasn't waiting and always follow up with an
> UNLOCK?
> >
> Yes I'm not sure the client can do it. It will have no info about the lock
and not
> sure how to create an appropriate UNLOCK reply (unless we keep around
> lock info of all cancelled locks).

The NLM4_GRANTED call has a response where the client can respond
NLM4_DENIED to indicate it is not accepting the lock.

The Ganesha NFS server could process out of order, however, if the client
responds to the grant with NLM4_DENIED, Ganesha will properly release the
lock.

The imperfectness of NLM is why I have seen requests for tools to free
wedged locks... We haven't managed to write such a tool for Ganesha yet (so
either we don't actually have that many NLM clients, or folks are using
other mechanisms to deal with the issue - restarting the server and making
all the clients reclaim the locks they want is one way to do it...).

Frank

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

---

From: Olga Kornievskaia <[email-masked]>
Date: Thu, 7 Dec 2017 14:17:22 -0500

> responds to the grant with NLM4_DENIED, Ganesha will properly release the

This is the case for when the server is doing a callback to the client
and granting the lock and client has ability to reply with denied. The
situation I'm curious about is just after the initial LOCK request is
sent and server hasn't processed it and it got&processed a
CANCEL/UNLOCK.

> all the clients reclaim the locks they want is one way to do it...).

Yes server-side lock breaking or rebooting is a way out this situation
in practice.

---

From: "Frank Filz" <[email-masked]>
Date: Thu, 7 Dec 2017 11:59:11 -0800

> >> On Thu, Dec 7, 2017 at 11:58 AM, J. Bruce Fields
> >> <[email-masked]>
> >> wrote:
> >> > I thought that LOCK/CANCEL race was one of the motivations for
> >> > NFSv4, so...
> >> >
> >> > ... I'd always assumed the protocol was impossible to implement
> >> > 100% correctly, though maybe there's some clever solution.
> >>
> >> >> >> Should it be client's responsibility to notice that it received
> >> >> >> a LOCK reply for which it wasn't waiting and always follow up
> >> >> >> with an
> >> UNLOCK?
> >> >
> >> lock
> > The Ganesha NFS server could process out of order, however, if the
> > client responds to the grant with NLM4_DENIED, Ganesha will properly
> > release the lock.
> 
> This is the case for when the server is doing a callback to the client and
> granting the lock and client has ability to reply with denied. The situation I'm
> curious about is just after the initial LOCK request is sent and server hasn't
> processed it and it got&processed a CANCEL/UNLOCK.

Oh, woops, yea... if the lock doesn't block, then the situation is different. When the client gets the LOCK response, it really should check and see if it was actually waiting for that response, or actually, the client should probably delay sending UNLOCK if it has not got a response to an outstanding lock request yet. CANCEL is only for cancelling a blocked lock, so shouldn't be sent unless the server has already responded to the LOCK with NLM4_BLOCKED, in which case there is no ordering issue (Ganesha should handle ordering issue between NLM4_GRANT call  and NLM4_CANCEL).

If we want NLM4_CANCEL to be able to cancel an inflight LOCK request, then we would need to add semantics which would include the server either responding to the cancel to let the client know it hasn't processed the LOCK request yet, or it should somehow remember the cancel so it can ignore the LOCK request when it processes it (which is probably too much for the server to deal with).

> > wedged locks... We haven't managed to write such a tool for Ganesha
> > yet (so either we don't actually have that many NLM clients, or folks
> > are using other mechanisms to deal with the issue - restarting the
> > server and making all the clients reclaim the locks they want is one way to
> do it...).
> 
> Yes server-side lock breaking or rebooting is a way out this situation in
> practice.
> >
> > Frank
> >
> >
> > ---

---

From: Olga Kornievskaia <[email-masked]>
Date: Wed, 13 Dec 2017 13:39:21 -0500

On Thu, Dec 7, 2017 at 2:59 PM, Frank Filz <[email-masked]> wrote:
> Oh, woops, yea... if the lock doesn't block, then the situation is different. When the client gets the LOCK response, it really should check and see if it was actually waiting for that response, or actually, the client should probably delay sending UNLOCK if it has not got a response to an outstanding lock request yet.

Current linux client does not do that.

> CANCEL is only for cancelling a blocked lock, so shouldn't be sent unless the server has already responded to the LOCK with NLM4_BLOCKED, in which case there is no ordering issue (Ganesha should handle ordering issue between NLM4_GRANT call  and NLM4_CANCEL).

Current linux client will send a CANCEL even if it didn't receive the
reply from the server.

> If we want NLM4_CANCEL to be able to cancel an inflight LOCK request, then we would need to add semantics which would include the server either responding to the cancel to let the client know it ha…

Sounds like Ganesha server (and Netapp) will have orphan locks that
would require manual lock break or server reboot.