Subject: [Curdle] Client-side SSH_MSG_EXT_INFO: Use it or lose it principle!

From: denis bider <[email-masked]>
Date: Sun, 26 Apr 2020 07:03:21 -0500

Hey all:

it has come to my attention that at least one SSH server implementation (a)
advertises support for SSH_MSG_EXT_INFO as defined in RFC 8308, and (b)
disconnects on actual receipt of an EXT_INFO message from the client.

This happens when we define a general mechanism, but then the most widely
used implementations only use certain aspects of it. Lots of
implementations now support EXT_INFO sent by the server because it's needed
for RSA-SHA2 signatures, but only a handful implementations send EXT_INFO
by the client.

Bitvise SSH Client sends EXT_INFO if the server supports it. It's free to
use or test with - but it runs on Windows and lots of people don't want to
test with that.

It's possible that your implementation has no need for any of the
extensions that would appear in EXT_INFO from the client. However, if you
wish for this extensibility to be available in the future, then if your
client supports EXT_INFO, I would strongly suggest that it sends it.

If you want to send a client-side EXT_INFO that takes minimal effort, I
suggest:

1. Ensure that your SSH client can correctly handle SSH_MSG_GLOBAL_REQUEST
received at any point. Your client should not disconnect when it receives
this message, or act like it received a channel open failure if it receives
a global request when it's waiting for a channel open confirmation. These
are the two most common error modes relating to global requests.

2. Once you have ensured the above, if the server supports EXT_INFO, send a
client-side EXT_INFO implementing the extension "global-requests-ok" as
defined in this draft:

https://tools.ietf.org/html/draft-ssh-global-requests-ok-00

This tells the server that your client properly implements global requests
so that the server can use them for things like active keep-alives.

Implementing this would take minimal effort and would help ensure that
client-side EXT_INFO remains an available extension mechanism 10 years down
the road for SSH.

denis

---

From: Ron Frederick <[email-masked]>
Date: Sun, 26 Apr 2020 10:33:56 -0700

Hi Denis,

On Apr 26, 2020, at 5:03 AM, denis bider <[email-masked]> wrote:
> it has come to my attention that at least one SSH server implementation (a) advertises support for SSH_MSG_EXT_INFO as defined in RFC 8308, and (b) disconnects on actual receipt of an EXT_INFO message from the client.
> 
> This happens when we define a general mechanism, but then the most widely used implementations only use certain aspects of it. Lots of implementations now support EXT_INFO sent by the server because it's needed for RSA-SHA2 signatures, but only a handful implementations send EXT_INFO by the client.
> 
> Bitvise SSH Client sends EXT_INFO if the server supports it. It's free to use or test with - but it runs on Windows and lots of people don't want to test with that.
> 
> It's possible that your implementation has no need for any of the extensions that would appear in EXT_INFO from the client. However, if you wish for this extensibility to be available in the future, then if your client supports EXT_INFO, I would strongly suggest that it sends it.
> 
> If you want to send a client-side EXT_INFO that takes minimal effort, I suggest:
> 
> 1. Ensure that your SSH client can correctly handle SSH_MSG_GLOBAL_REQUEST received at any point. Your client should not disconnect when it receives this message, or act like it received a channel open failure if it receives a global request when it's waiting for a channel open confirmation. These are the two most common error modes relating to global requests.
> 
> 2. Once you have ensured the above, if the server supports EXT_INFO, send a client-side EXT_INFO implementing the extension "global-requests-ok" as defined in this draft:
> 
> https://tools.ietf.org/html/draft-ssh-global-requests-ok-00 <https://tools.ietf.org/html/draft-ssh-global-requests-ok-00>
> 
> This tells the server that your client properly implements global requests so that the server can use them for things like active keep-alives.
> 
> Implementing this would take minimal effort and would help ensure that client-side EXT_INFO remains an available extension mechanism 10 years down the road for SSH.

I’ve made this change in the AsyncSSH “develop” branch as commit https://github.com/ronf/asyncssh/commit/b87291d <https://github.com/ronf/asyncssh/commit/b87291d>, and will roll this into the next release. Most of the support was already in place as I already had common code for generating and parsing EXT_INFO on both the client and server. The changes are as follows:

    - AsyncSSH now advertises that it will accept EXT_INFO from clients when acting as a server

    - AsyncSSH will send the global-requests-ok extension when acting as a client and talking to a server that supports EXT_INFO from clients

Since AsyncSSH doesn’t actually have any heuristics around disabling the use of global requests based on client software/version, it doesn’t actually do anything right now when it receives the ‘global-requests-ok’ extension. However, I confirmed it successfully parses received extensions, so it should be easy to process future extensions from the client when this is needed.

I also specifically tested this against the Bitvise server (thanks for making that available for free for non-commercial use!), and it seems to work well.
-- 
Ron Frederick
[email-masked]

---

From: denis bider <[email-masked]>
Date: Mon, 27 Apr 2020 00:33:46 -0500

Awesome!

Yes, the idea of "global-requests-ok" is not to make the client do anything
different. It's to reassure the server that the client won't violate the
spec if the server sends a global request.

The main motivating usage case is so the server can feel at liberty to send
the global request "[email-masked]" after user authentication:

https://cvsweb.openbsd.org/src/usr.bin/ssh/PROTOCOL?annotate=HEAD

This is essential functionality which allows clients and servers to change
host keys without requiring administrative intervention at each client
installation.

Without "global-requests-ok", the server needs a database of client version
strings to which it's safe to send the "hostkeys" global request.
Otherwise, it can't send it because a bunch of half-baked clients (I'm
counting 7 in our list) are borked and can't handle receiving a global
request. Or they can handle it usually, but not when they're expecting a
response to a channel open.

Meanwhile I've received more info about the implementation that disconnects
on receiving EXT_INFO from the client. The bug is unfortunately in a widely
used Java SSH library. The information I received suggests it's going to be
fixed, but I'm afraid there are already a fair number of deployed servers
that have the issue.

While I'm wishing things - I would also like the "delay-compression"
extension to be more widely supported because it's the only
delayed-compression mechanism that (1) does not have a race condition by
design ([email-masked]), and (2) does not require a second key exchange
after user authentication (PuTTY mitigation).

denis

On Sun, Apr 26, 2020 at 12:33 PM Ron Frederick <[email-masked]> wrote:

> it has come to my attention that at least one SSH server implementation
> (a) advertises support for SSH_MSG_EXT_INFO as defined in RFC 8308, and (b)
> 2. Once you have ensured the above, if the server supports EXT_INFO, send
> a client-side EXT_INFO implementing the extension "global-requests-ok" as
> defined in this draft:
>
> I’ve made this change in the AsyncSSH “develop” branch as commit
> https://github.com/ronf/asyncssh/commit/b87291d, and will roll this into
> the next release. Most of the support was already in place as I already had
> common code for generating and parsing EXT_INFO on both the client and
> server. The changes are as follows:
>
>     - AsyncSSH will send the global-requests-ok extension when acting as a
> client and talking to a server that supports EXT_INFO from clients
>
> Since AsyncSSH doesn’t actually have any heuristics around disabling the
> use of global requests based on client software/version, it doesn’t
> actually do anything right now when it receives the ‘global-requests-ok’
> extension. However, I confirmed it successfully parses received extensions,
> so it should be easy to process future extensions from the client when this
> is needed.
>
> I also specifically tested this against the Bitvise server (thanks for
> making that available for free for non-commercial use!), and it seems to
> work well.
> --
> Ron Frederick
> [email-masked]
>
>
>
>

---

From: Peter Gutmann <[email-masked]>
Date: Mon, 27 Apr 2020 05:37:07 +0000

>disconnects on actual receipt of an EXT_INFO message from the client.

Not wanting to do a public name-and-shame on this, but could you share the ID
string needed to fingerprint this server?  Looks like a lot of implementations
will need to be able to deal with this...

>used implementations only use certain aspects of it.

That's a more specific version of "an implementation is fully SSH standards-
compliant when it can connect to OpenSSH (client) or Putty can connect to it
(server)".  Those two are the universal benchmark for SSH implementations, for
better or for worse.  TLS dealt with this to some extent by adding a mechanism
bacronym'd as GREASE for sending random information in extensions to detect
implementations that broke on them, perhaps something similar could be done
for SSH.

Peer.

---

From: denis bider <[email-masked]>
Date: Mon, 27 Apr 2020 00:46:29 -0500

Unfortunately, the specific server implementations that use the Java
library may use an unrelated version string.

Unless I get new information, our next SSH Client version (8.42) will
disable sending of SSH_MSG_EXT_INFO if the version string contains:

CrushFTPSSHD
J2SSH_Maverick

However, I suspect the are more and/or the above might not be accurate. At
this point I'm pretty sure I've been receiving reports of this issue for a
while, but never figured it out because they tend to be ad hoc servers that
I couldn't connect to, and I never heard back from users with any
server-side diagnostics. This is the first time a user pointed me to a
server with which I could test.

denis

On Mon, Apr 27, 2020 at 12:37 AM Peter Gutmann <[email-masked]>
wrote:

> (a)

---

From: denis bider <[email-masked]>
Date: Mon, 27 Apr 2020 04:00:12 -0500

About this:

> TLS dealt with this to some extent by adding a mechanism
> bacronym'd as GREASE for sending random information in
> extensions to detect implementations that broke on them,
> perhaps something similar could be done for SSH.

The challenge with this is that some widely used implementation -
*cough*OpenSSH*cough* - would need to exercise all functions of the
protocol that are legal, but not necessarily in widespread use. If they
want to prevent rusting at the joints, OpenSSH would have to do so for most
features, including those they do not CURRENTLY find useful.

For example, if a common problem is that clients fail to correctly handle
global requests if they receive them while waiting for channel open
confirmation - this is a bug that both OpenSSH and PuTTY had, at some point
long ago - then the way to exercise that would be to include a trivial
global request, in say 10% of cases, before a channel open confirmation.

I would welcome that, personally, and hence that's why I originally
included Damien in this thread. :)

denis