Subject: Re: Updated RSA SHA-2 draft / New draft: SSH Extension Negotiation

From: denis bider <[email-masked]>
Date: Tue, 10 Nov 2015 04:55:44 +0000

Niels:

>> - libssh handles the KEXINIT reserved field incorrectly
>> (ignores actual value, so key exchange fails if it's not zero)

> Is this a bug or not? 

This is a bug. The implementation forgets to decode this field. It does not disconnect on purpose. The disconnect is just a side effect of key exchange failing.

> The reserved field is intended for extensions, but from
> a quick look I can't see that the spec (RFC 4253) defines
> any valid behaviour when a client or server receives a
> non-zero value there. I think a complient implementation
> ought to disconnect.

You cannot extend the protocol using this field, if existing implementations disconnect when they see a new value. That would not allow for extension. It would require replacement of the protocol with a new and incompatible version.

This goes to show that this kind of thing is not obvious to everyone, and must be explicitly spelled out. It is a flaw of RFC 4253 that it assumes people will understand how extension fields function, and does not spend a paragraph on this.

> So in the short term, this field seems pretty useless.

The common convention with extension flags is as follows. If you are unaware of any extensions:

(1) Always set the defined non-extended value. This is usually zero.
(2) DO NOT EVER VERIFY that you have received some particular value.

That's all there is to it. That's how extension fields function. An application that is aware of the extension will notice a non-zero value if it's received, and will know what to do with it.

The problem is that this extension strategy is fragile. For this to work, all implementations have to do the correct thing: which is to reliably send the non-extended value if they're unaware; and never verify the received value.

> > client-req-ok
> This is essentially an extension saying "I don't have
> known bug X".

The behavior isn't known to be a bug. It's an ambiguity in the spec, which produces this behavior when combined with implementers who think there's benefit in being difficult on purpose. (For illustration, see your reasoning with respect to KEXINIT extension field.)

This is an extension saying "I implement non-guaranteed behavior X".

> Sooner or later we'll see an implementation which advertises
> "no bug X" but actually has bug X,

There's no reason to expect that. Disconnecting on global request is not some kind of "whoops". It's a "feature" implemented by someone who likes to makes things difficult on purpose. It does not necessarily violate the spec because the spec is silent on whether clients should gracefully handle a global request.

> > no-handbrake
> Is the intention that the endpoints should buffer arbitrarily large
> amounts of data, or is the receiver of the data stream allowed to
> block and stop processing ssh messages while delivering the data?

The intention is that TCP-based flow control is used.

> I think I'd prefer to do it per-channel, and if enabled, let the
> parties do their best if other channels are starved for data
> or get blocked.

Enabling this by necessity changes flow control for the entire session.

> If we want to improve on the flow control in ssh-connection,
> it might also make sense to look at what other recent
> multiplexing protocols do, e.g, http/2.

I think our multiplexing is fine. This extension is to accommodate applications that don't want to multiplex.

denis

----- Original Message -----
From: Niels "Möller" 
Sent: Monday, November 9, 2015 13:05
To: denis bider 
Cc: [email-masked] ; Jeffrey Hutzelman ; Mark D. Baushke ; [email-masked] ; [email-masked] ; [email-masked] ; Peter Gutmann ; Max Horn 
Subject: Re: Updated RSA SHA-2 draft / New draft: SSH Extension Negotiation

denis bider <[email-masked]> writes:

> I have looked at alternatives, but:
> - libssh handles the KEXINIT reserved field incorrectly (ignores
> actual value, so key exchange fails if it's not zero)

Is this a bug or not? The reserved field is intended for extensions, but
from a quick look I can't see that the spec (RFC 4253) defines any valid
behaviour when a client or server receives a non-zero value there. I
think a complient implementation ought to disconnect.

It would make some sense to combine client's and server's value in some
way, e.g., bitwise and, or taking the maximum value, and use the result
to enable extensions. But we really have to define that mechanism first,
before we can define extensions using it... So in the short term, this
field seems pretty useless.

> client-req-ok: Allows clients to affirm that the server can send a
> global request (especially an unrecognized global request) without
> risking the client disconnecting. This should be supported by all
> clients. It is necessary so that servers can enable active keep-alive,
> which is not otherwise possible generally because some clients do
> disconnect when they receive a global request.

This is essentially an extension saying "I don't have known bug X". I
think we should avoid those things. If we want to interoperate with
buggy implementations, I'd actually prefer hacks based on the version
strings. Sooner or later we'll see an implementation which advertises
"no bug X" but actually has bug X, and then we're back to square one.

> no-handbrake: Peter ought to like this. :) A few years late, but this
> makes window size infinite for both directions in a channel, at the
> cost of restricting the session to one simultaneous channel. This
> should help file transfer applications for which the channel flow
> control in SSH is an impediment, rather than a feature.

Makes some sense. Is the intention that the endpoints should buffer
arbitrarily large amounts of data, or is the receiver of the data stream
allowed to block and stop processing ssh messages while delivering the
data?

I think I'd prefer to do it per-channel, and if enabled, let the parties
do their best if other channels are starved for data or get blocked. And
if it is intended to be a global state, a GLOBAL_REQUEST seems more
appropriate than an extension to the *transport* layer of the ssh
protocol.

If we want to improve on the flow control in ssh-connection, it might
also make sense to look at what other recent multiplexing protocols do,
e.g, http/2.

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
Internet email is subject to wholesale government surveillance.

---

From: denis bider <[email-masked]>
Date: Fri, 13 Nov 2015 14:18:59 +0000

Niels (regarding EXT_INFO):

> But please don't mix that up with the simpler issue
> of algorithm updates.

Yes - this is why I have separated the RFCs, so they can progress, or not progress, independently.

> Note that's what relevant for public key user auth
> isn't really the algorithms supported by the server,
> but the algorithms of the keys which are actually
> authorized for login.

This seems to me the kind of setting that's likely to be enabled or disabled by server-wide policy or algorithm availability, not by per-user configuration. For example, if the server doesn't support rsa-sha2-512, but does support rsa-sha2-256, that's probably going to be server-wide, not user-specific. If the server administrator disabled ssh-rsa due to SHA-1 concerns, that's also going to be server-wide.

> It makes some sense to filter available keys depending
> on what the server supports, but I don't think it will
> make much of a practical difference.

Consider that you have a server that supports rsa-sha2-256, but not ssh-rsa. You have a client who's inclined to try rsa-sha2-512 first, and rsa-sha2-256 next. Or the other way around.

Without EXT_INFO, you spend:
- one round-trip in SERVICE_REQUEST and SERVICE_ACCEPT
- one round-trip in initial request with rsa-sha2-512 signature, rejected
- one round-trip in next request with rsa-sha2-256 signature, accepted

Total: 3 round-trips. This might be 0.6 seconds over the Pacific (200 ms per round-trip), or 6 seconds over a satellite link (2s per round-trip).

With EXT_INFO, you spend:
- half a round-trip waiting for server's EXT_INFO, which comes with NEWKEYS
- one round-trip in request with rsa-sha2-256 signature, accepted

Total cost: 1.5 round-trips. This might be 0.3 seconds over the Pacific, or 3 seconds over a satellite link.
    
Now consider a future scenario where we add rsa-sha3-256 and rsa-sha3-512. :) Now you might be spending 5 round-trips guessing the right algorithm without EXT_INFO. Whereas with EXT_INFO, you are done in 1.5 round-trips. These 5 round-trips might be 1 second over the Pacific, or 10 seconds over a satellite link.

With EXT_INFO, we have a flat cost that does not increase. Without EXT_INFO, cost may not look so bad right now (say 3 round-trips if second guess is right); but future round-trip cost increases.

----- Original Message -----
From: Niels "Möller" 
Sent: Friday, November 13, 2015 02:16
To: denis bider 
Cc: [email-masked] ; Damien Miller ; Jeffrey Hutzelman ; Mark D. Baushke ; [email-masked] ; [email-masked] ; Peter Gutmann ; Max Horn 
Subject: Re: Updated RSA SHA-2 draft / New draft: SSH Extension Negotiation

denis bider <[email-masked]> writes:

> It seems to me that SSH needs a proper extension mechanism to avoid
> version-string-based hacks and other kinds of hacks.

Maybe. But please don't mix that up with the simpler issue of algorithm
updates.

> - Does not make signature algorithm information available in time for
> the client's first user auth request. This costs a round-trip if the
> client's first guess is incorrect.

Note that's what relevant for public key user auth isn't really the
algorithms supported by the server, but the algorithms of the keys which
are actually authorized for login. It makes some sense to filter
available keys depending on what the server supports, but I don't think
it will make much of a practical difference.

Regards,
/Niels

---

From: denis bider <[email-masked]>
Date: Fri, 13 Nov 2015 13:42:26 +0000

If you're not seeing full network utilization to the extent permitted by your TCP stack (note that the TCP stack may itself be throttling you below network potential), then the problem is very likely either mismanaged channel window size (probably in the sending direction), or insufficient SFTP request pipelining.

In general, the channel window must be large enough that the sender doesn't get throttled by the window size before the receiver can respond with a window adjust. The size required increases linearly with a multiple of connection latency and bandwidth.

Note that the required window size for full throughput can be fairly large with long fat pipes; WAY larger than would be appropriate for a slow, low-latency connection. If you're sending at 40 Mbps (which is not even that fast now) across the Atlantic or Pacific (200 ms round-trip) you need a window size on the order of 1 MB. If you want to send at gigabit speeds with same latency, you need appropriately more; e.g. 20 MB.

The SFTP performance problem can be solved, and we believe it to be solved in our software, using both appropriate window sizing and SFTP request pipelining. So the "no-handbrake" extension is not strictly NECESSARY, it's possible to reach network potential without it.

What the "no-handbrake" extension does is remove a level of complexity for people writing simple, special-purpose SSH implementations, who want to achieve decent performance without having to make a PhD out of it. If the channel window sizing implementation is poor, then if "no-handbrake" is not available, performance will suck. But if it's available, performance will be as good as the TCP stack permits (which is written by professionals).

If you have a multipurpose implementation that supports multiple channels, I think you should just do channel window sizing properly. I would not like general-purpose, multichannel implementations resorting to this. You're already accepting a higher level of complexity by supporting multiple channels; so why not do it right.

----- Original Message -----
From: Niels "Möller" 
Sent: Friday, November 13, 2015 00:21
To: denis bider 
Cc: Peter Gutmann ; [email-masked] ; Jeffrey Hutzelman ; Mark D. Baushke ; [email-masked] ; [email-masked] ; [email-masked] ; Max Horn 
Subject: Re: Updated RSA SHA-2 draft / New draft: SSH Extension Negotiation

denis bider <[email-masked]> writes:

> It seems to me the only known applications that might want to use
> "no-handbrake" are those that don't want to use multiplexing. I don't
> know of a different usage case.

One usecase: Make make s single general-purpose connection from my
workstation to my server, using it for one or more shell sessions, port
worfwarding, etc. I use "connection sharing", i.e., a local private unix
socket that can be used to create additional channels over the same
connection.

Then I decide to do a large file transfer, so I create a new channel
just for that purpose, and I really want it to finish as soon as
possible.

Now, this is a problem in my implementation which I haven't yet bothered
to solve, because it uses a fix and pretty small window size, and
typically I won't get close to full network utilization. So I don't know
if it's good enough to just set a much larger window size (but if it
doesn' I think that ought to solve all "no-handbrake" usecases).

> In this situation, you must stop reading from the SSH socket, which
> affects all channels.

Agreed. And we should note that the "no-handbrake" will not work well if
the bottleneck of the data flow actually is after ssh, so users need to
enable it with care. One option might be to timeout after a few seconds
and close the connection; then we kill the affected channel, without
hanging the connection indefinitely.

> The way I see it, the only clean usage scenario for this feature is
> where there's a single channel.

Also in this case, if you get overwelmed with data, you have to stop
reading the ssh socket, and hence making timely key reexchange
impossible.

> That's why I think it makes sense to negotiate it on the transport
> layer. The transport layer must be aware of this.

Maybe. but I'd prefer a solution to both the single-channel use-case and
the usecase above. I think we need to take a step back and sort out what
the use cases are. I've been thinking that the poor utilization I've
seen is simply a sign of poorly chosen window sizes in my
implementation. If it's a more general problem, we need to understand
what the usecases and failure modes are.

A different solution which I've been considering is a channel mechanism
to automatically increase the window size when the ssh flow control is
the bottleneck. It could be as easy as checking on reception of
SSH_MSG_DATA if the receive window goes down to 0, and if so increase
the size of the receive buffer and send a WINDOW_ADJUST which reflects
not delivery of data, but the increased buffer.

Regards,
/Niels