Subject: [nfsv4] Issue28: denial of service in NFSv4.x state management

From: Mike Eisler <[email-masked]>
Date: Fri, 7 Apr 2006 04:01:12 -0700 (PDT)

http://www.nfsv4-editor.org/cgi-bin/roundup/nfsv4/file6/dos-nfsv4.txt

has the gory details.

With NFSv4.1 sessions we are more secure than
with NFSv4.0, but I think we still have a 
denial of service attack. Sessions adds
a SEQUENCE operation. Let's say an attacker issues
a COMPOUND { SEQUENCE, ... } with a slot id that he expects
the legitimate user -- on the legitimate client that owns the
session -- to use next. The client then
uses the slotid with the same sequence number, and the
server returns the attacker's result from the replay
cache. Not good.  For NFSv4.1 we could
include the credential with the slot in the replay
cache. Thus when Alice falls for Mallet's trap -- she uses
attacker Mallet's slot and sequence -- she doesn't get
back Mallet's response. So we could dispose of Mallet's
response, and treat Alice's request as legitimate. But
then, what is to prevent Mallet from doing the same unto
Alice? Alice sends a request, Mallet sends another before
Alice gets a response, and causes Alice's response to be
lost (perhaps by spoofing the server and sending a TCP
RST back to the client). Now Alice's response is removed
from the replay cache. Alice re-sends her request, and now
Mallet's response is removed from the cache. But because
Alice is sending a non-idempotent operation, she gets back
the wrong result.

If we give each user their own v4.1 session, then all
the problems disappear. But the additional state could be
be prohibitive.

A less costly approach is to create a new operation called:

	AUTHORIZE_SESSION

		args:

			secinfo4 asa_sec;
			utf8str_mixed  asa_who;
			opaque asa_sec_context<>;
			bool asa_give_as_rights;

		res:

			nfsstat4 status;
			uint64_t asa_auth_cookie;

The principal who created the session can issue an
AUTHORIZE_SESSION operation to let a given user use
the session with the specified security parameters. If
RPCSEC_GSS is being used, then asa_sec_context is an
RPCSEC_GSS handle for the given user.  This way, random
users cannot create RPCSEC_GSS sessions from user-level,
forge the sessionid and mess up other users. Unless they
go through the kernel they are defeated.

asa_give_as_rights is there to pass session ownership
rights on. This is for the scenario where there are no machine
creds; just user creds (think automounter; better yet think
NFSv4 global namespace). The user that created the session
might have his creds expire. You want future users of the
session to continue to use the session. What if all users
have their creds expire? Then the session is unusable and
will expire.

	UNAUTHORIZE_SESSION

		args:
			uint64_t usr_auth_cookie;

		res:
			nfsstat4 status;

Removes authorization. Any context with authorize
session rights can remove authorization.

The above two operations are issued in a COMPOUND that
leads with SEQUENCE.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Talpey, Thomas" <[email-masked]>
Date: Fri, 07 Apr 2006 14:13:55 -0400

I don't see how Alice issues requests with Mallet's slotid and
sequenceid, without having compromised the client NFS
implementation, or hijacking Mallet's TCP connection? Isn't
one or the other of these necessary to carry out the attack?

A question on authorize/unauthorize. I assume you intend that
clients will send many such authorizations, one for each distinct
user credential that is sent on the session?

Tom.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 7 Apr 2006 11:50:44 -0700 (PDT)

> sequenceid, without having compromised the client NFS

Actually Mallet is the compromiser. Alice just uses the next slotid
and sequenceid that the algorithm assigns, but Mallet has done that
already from another TCP connect.

> implementation, or hijacking Mallet's TCP connection? Isn't

If an attacker can compromise the NFS client implementation, then my suggestion
won't work either, because he can create RPCSEC_GSS contexts at will
and add them to the session via the AUTHORIZE_SESSION operation.

> one or the other of these necessary to carry out the attack?

I didn't see anything in the sessions text in
http://nfsv4-editor.org/drafts/draft-ietf-nfsv4-minorversion1-02.html
that binds client to server TCP connections to sessions Assuming there
is such a binding, then admittedly this makes the attack somewhat harder.
Hijacking the TCP connection of course lets the attacker do even worse
denial of service and other attacks. So i think we wouldn't need
AUTHORIZE_SESSION. Although there is the issue of what if all TCP
connections have broken. How can we securely re-establish a TCP connections
over the sessions. I don't understand the recovery issues around sessions.

Perhaps the streamchannelattrs4 data type has this binding, but there's no
definition of it. Just created issues 44 and 45 
(the latter is for rdmachannelattrs4).

> user credential that is sent on the session?

Yes. Just as it today sends many RPCSEC_GSS INIT procedures (overloading
the NULL proc).

---

From: "Talpey, Thomas" <[email-masked]>
Date: Fri, 07 Apr 2006 15:17:26 -0400

>already from another TCP connect.

Check. I got the names switched.

>that binds client to server TCP connections to sessions Assuming there

There used to be such an association in an early version of the protocol.
But it causes many problems, most notably in reconnect. If a connection
is bound to a session, then a client must re-bind after a connection loss.
When the protocol worked that way, it meant a client could not simply
retry after reconnecting - it would have to successfully rebind first. This
was, to say the least, problematic.

>over the sessions. I don't understand the recovery issues around sessions.

The concept was that if the server cared to enforce it, it would require
some form of strong authentication from the client, and verify that it was
matched that used at session creation time. You are positing a wrinkle in
that scenario however.

>(the latter is for rdmachannelattrs4).

Those attributes are currently undefined in the .x, but they are described
in the text. There are no Stream attributes but we can certainly add some!

Tom.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 10 Apr 2006 12:28:59 -0700 (PDT)

> was, to say the least, problematic.

I can't visualize the problems, but I'll accept they exist,
and will attempt to propose changes that avoid an explicit 
connection bind operation.

Let's stipulate that the principal/secinfo4 combination 
which created the session always has right to introduce new 
TCP connections to session.  Once a TCP connection is
bound to the session, any principal/secinfo4 combination can use the
session over that TCP connection (this doesn't mean they have authorization
to access any data; that depends on the export and file permissions on
the data). 

With these stipulations the protocol looks exactly the same as it does now.

Now, let's try to address the problem of an NFSv4.1 client that
doesn't have machine creds, and used the creds of a random user that
causes the client to the mount the server. The session will be created with
creds of random user Alice. Later, Bob logs into the client and uses
the same NFS mount and session. Then Alice logs off. After several hours, if not
before, Alice's creds expire, and because she has logged off, the in-kernel
NFS client cannot get new creds for Alice. This isn't a problem though until
the TCP connection breaks (say due to a network partition). Now, via the
in-kernel NFS client, Bob tries to bind a TCP connection to the session.
He can't because Alice has gone.

So, re-enter AUTHORIZE_SESSION:

 	AUTHORIZE_SESSION

		args:

			secinfo4 asa_sec;
  		        utf8str_mixed  asa_who;

When Bob tries to the use the NFSv4.1 session, the client authorizes
Bob to bind TCP connections to the session. So Bob, via the
NFSv4.1 client, can recover from connection drop. 

If server stores its replay cache in stable storage,
it records with the replay cache that Bob is authorized 
to bind connections to the session. When the server reboots, 
Bob can re-establish the connection. 

So far so good.

But what it is evil Eve and not Bob who follows Alice?
Eve gets authorized to bind TCP connections. Then
the server reboots. Eve re-establishes a connection
from the original client. Eve from another client also establishes
a connection by a synthetic client intended to do
denial of service, and binds the connection to the session.
Now Eve logs off the original client. From the new client,
Eve denies Bob service. 

We could make it much harder for Eve to do mischief if we required that
when number of connections is >= 1, the next connection bound to
the session MUST be done with using RPCSEC_GSS context that has been 
used previously on one one of the other bound connections.

We could make it even harder on Eve if we added a new operation,
BIND_ANOTHER_CONNECTION that provided the source address of the new
connection in its arguments. The flow would be:

            client connects to server
Eve would have to be able to hi-jack
TCP connections to do mischief, and if she can do that, then
she doesn't need to mess around with NFSv4.1 sessions to execute
attacks.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 10 Apr 2006 12:36:38 -0700 (PDT)

I hit the send key to soon:

              client converts source address of new connection into
              UADDR form.

              client issues a BIND_ANOTHER_CONNECTION operation on a
              previously bound connection with the UADDR of the new connection
              as arguments:
                      COMPOUND { SEQUENCE, BIND_ANOTHER_CONNECTION }

              client issues a COMPOUND { SEQUENCE } operation on the new
              connection, using the same RPCSEC_GSS context handle that was
              used for the BIND_ANOTHER_CONNECTION operation.

---

From: "Talpey, Thomas" <[email-masked]>
Date: Mon, 10 Apr 2006 16:10:18 -0400

At 03:36 PM 4/10/2006, Mike Eisler wrote:
>connection in its arguments.

I agree with the earlier message, but I have a problem with the
above. The whole point of the session is to get away from using
transport identifiers to uniquely describe a connection. A second
problem is that this presumes the new connection survives for long
enough to receive the benefit of this authorization. Finally, who's
to promise that some other connection won't come in with these
identifiers? Ports can be reassigned rather quickly, etc.

Is it unreasonable for a server to require any client to have valid
machine creds in order to receive this level of security? As you
said in your previous message, this is enough to implement the
protection without any additional ops.

Tom.

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 10 Apr 2006 15:57:13 -0500

> protection without any additional ops.

I would expect that it is actually those environments where
machine creds are not available that one would want additional
security considerations.  We have been making changes in the
Solaris client (to things like the automounter) to work without
machine/host credentials to remove the administrative burden
for VPN like clients.  I point this out only to suggest that
there will be clients without machine/host credentials or
at least the ability to configure them in this way.

Spencer

---

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 11 Apr 2006 08:55:29 -0400

Okay, but I think where we're going in this case is to have
what amounts to a machine credential.

What Mike is advocating is the useful addition of a capability
granting operation, bestowed by the creator of a session. This
isn't something we have. We want this capability to be unforgeable
and revokable. I'm thinking we have a security feature here.

Anyway, the transport identifiers don't get us there. I thought
of another problem - the source address might not be an IP address
at all, and in the presence of NAT, etc might not be the same at
both ends of the wire.

Is it reasonable to consider a per-user nonce of some kind, that
accompanies the sessionid in the SEQUENCE op? Eve would then
have to steal Adam's nonce to carry out the attack, but the nonce
would be protected by the local kernel and/or wire security.

Tom.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 12 Apr 2006 01:06:29 -0700 (PDT)

> at all, and in the presence of NAT, etc might not be the same at

Agreed.

> would be protected by the local kernel and/or wire security.

That leaves the problem of how to prevent Eve from eavesdropping
and determining the nonce. But your question
leads me to this stab...

After a session is created, the client creates a secret 
session verifier (SSV). The SSV is sent to the server, 
protected via RPCSEC_GSS with the privacy service. This
prevents an attacker from eavesdropping. If the server
keeps the replay cache in stable storage, the SSV needs to
be stored as well. The connection CREATESESSION is used
with is always bound to the session with no additional
hand shake. Setting the SSV requires a new trivial operation,
or perhaps we could just use SETATTR. The SSV can be
changed but it is changed via an XOR of the current
SSV with the new SSV. The XOR is not committed on the server
until the XOR has been confirmed on all connections. Each SEATTR
(SSV) change includes a count of the number of connections
remaining to confirm the SSV change over. So if
the current SSV is 0x101, and the XOR SSV is 0x110, and
there are 3 connections, and 3 SETATTR (SSV:0x110) operations
are done, one per connection, then once all done, the new SSV is
0x101^0x110 = 0x011. Once the remaining to confirm count goes
to zerom the new SSV is committed, and any remaining connections
cannot be used; the server will therefore disconnect them.
The reason will be apparent later.

GETATTR (SSV) is not supported.

A new operation, BIND_CONN_TO_SESSION, is added to bind 
an additional connection to a session. It MUST be used on
the connection being bound. It MUST be the first operation after
SEQUENCE if SEQUENCE is used on an unbound connection. 
Any principal, flavor, RPCSEC_GSS context can invoke the operation.
The arguments are:

  opaque bcts_ssv_digest<>;
  struct bcts_digest_input {
     uint64_t bdi_random;
     struct SEQUENCE4args bdi_seq;
     nfstime4 bdi_time_stamp;
  }

The bcts_ssv_digest<> contains an HMAC output. An HMAC is an algorithm
that combines a shared secret with public information to
produce an unforgeable message authentication code, using a one way hash
like md5, sha-1, etc.  The shared secret is the SSV. The public
information is bcts_digest_input. bdi_random is a random number the client
generates. bdi_seq is a copy of the preceding operation, a SEQUENCE op.
bdi_time_stamp is the client's idea of the time of day. If
client doesn't want to use synchronized time, it get can set bdi_time_stamp
to zero. There are probably other things we can jam in into the _input.

The server verifies the digest by re-computing the HMAC (and if bdi_time_stamp is
non-zero, making sure the time is within a reasonable delta of the server's
idea of time of day). The server in turn sends:

  opaque bcts_ssv_digest<>;
  struct bcts_digest_output {
     uint64_t bdo_random;
     struct SEQUENCE4res bdo_seq;
     nfstime4 bdo_time_stamp;
  }

bdo_random is a random value from the server, bdo_seq is the result of
the SEQUENCE operation, bdo_time_stamp is the current time
stamp. The client verifies the corresponding digest which is the HMAC
of the SSV and  bcts_digest_output. 

Basically this is CHAP authentication.

The server is resistant to replay attack because the slot and
sequence numbers are the input to the digest. 
Further more, if the client uses RPCSEC_GSS
the server can record in the replay cache 
what security flavor and principal was
used for the BIND_CONN_TO_SESSION operation.

If the machine-cred-less client XORs the 
SSV each time a new principal
uses the session, and does this XOR 
via an encrypted SETATTR (SSV) which
uses the creds of that new principal, 
then this severely limits Eve's ability
to do a denial of service. She could
create a new connection from another client,
and then change the SSV before the original client
changes the SSV for Bob. The client would know that
something is amiss though, and so it would have the
option of creating a new session. Since Bob hasn't
had a chance to do any useful work, the only
person negatively affected is Eve: she loses her
replay cache and no one else does. 

Eve could just create and bind another connection to the
session from another client, but because Bob's use
of the session from the original client will have
a SETATTR (SSV) with a remaining to confirm connection
count of 0, Eve is out of luck.

If the client loses all connections, it needs to use
BIND_CONN_TO_SESSION to bind new connections. The server
won't have the SSV if the server has rebooted and the server
doesn't keep the replay cache in stable storage. In that event,
the preceding SEQUENCE op in the same compound
will have returned _BADSESSION, so the client's state machine
goes back to CREATESESSION. 

There's an issue where the client has lost some or all
connections for a session due to a network partition 
and the server isn't aware. The problem is 
that an SSV change has to happen on all
connections, but the client doesn't have anyway to do that if the
client and server get out of sync with respect to connection
count. Again, SETATTR (SSV) includes a count indicating 
the number of remaining connections to confirm. 
If the server thinks there are more, then once
the connection count from the SETATTR(SSV) operation is zero,
the server disconnects the remaining connections.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 12 Apr 2006 01:50:26 -0700 (PDT)

Under the SSV model I just posted there's an issue 
if SETATTR (SSV operand to XOR) is sent, no response is returned, and the connection
disconnects. The client, per the sessions model, needs to retry the
SETATTR. But it needs a new connection to do so.

First the client reconnects.

If the SETATTR(SSV operand to XOR) was called with a remaining 
connection count > 0, then the old
SSV should be acceptable for BIND_CONN_TO_SESSION. But even
if it was called with a connection count == 0, the old SSV
may still be acceptble. So either way, the old SSV can be used.

If the old SSV cannot be used for the BIND_CONN_TO_SESSION
digest, and the connection remaining count argument of the
SETATTR was > 0, then the client has to re-create the session.
Either Eve is messing around, or the server is broken.

If old SSV cannot be used and the connection remaining 
count was == 0, then the new SSV needs 
to be used for the BIND_CONN_TO_SESSION
digest. If that still fails too, then client has to re-create
the session.

Once there is a session, the SETATTR is 
retried with old slot and sequence.

---

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 12 Apr 2006 17:58:18 -0400

After reading your previous message, I have a headache. :-)

But I have a question on this one:

>Either Eve is messing around, or the server is broken.

So, does the client need to inject a new operation after reconnecting,
and before retransmitting its old requests? It appears that the answer
would be yes in this case, in order to create a new session.

This type of approach is what was identified as a significant problem
in the earlier versions of the sessions protocol. There are no existing
interfaces by which client RPC layers can signal upper layers that such
a recovery is needed, without actually failing the in-progress operation.

If there is a security layer exchange, such as the one performed when
client creds require a refresh, then I think the story is more tractable
from a client implementation standpoint.

If my headache is clouding my understanding, please advise...

Tom.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 12 Apr 2006 15:50:53 -0700 (PDT)

> After reading your previous message, I have a headache. :-)

I'll have to re-read it later today to see if it makes as much sense now
as it did at 3am this morning..

> >may still be acceptble. So either way, the old SSV can be used.

s/used/attempted/

> >If the old SSV cannot be used for the BIND_CONN_TO_SESSION

s/cannot be use/is rejected/

> would be yes in this case, in order to create a new session.

I was thinking CREATESESSION would be used.

> a recovery is needed, without actually failing the in-progress operation.

If client can longer use the session, it just
stops using it, and uses another. The server can destroy the old replay
cache at its leisure. The situation of where a replay cache can be
forgotten by the client can arise when a client crashes hard
in the middle of session and never comes back. So the server has to
cope with this anyway.

In my proposal, the only way the client and server can get out of
sync with respect to what the SSV is if Eve creates mischief or client/server
bug. For the former, my analysis is that only Eve is affected. For the
latter ... crap happens. Either way, these are rare situations.