Subject: Why SFTP performance sucks, and how to fix it

From: [email-masked] (Peter Gutmann)
Date: Tue, 8 Jul 2003 16:01:02 +1200

Now that I've got your attention... :-).

The following is a section of the (not-yet-published) paper "Performance
Characteristics of Application-level Security Protocols", which looks at,
well, performance characteristics of application-level security protocols.
One of these is SSH (the rest are SSL, PGP, and S/MIME, in case anyone cares).
The paper hasn't been published yet (it's still a work in progress), since it
probably won't be published for awhile (I'm probably submitting it to Usenix
Security next year) and since the information in this section could be useful
to SSH developers, I'm posting it here.  If people find it of any use, I might
later post it to one or two general crypto lists to let other crypto
developers know, since it describes the SFTP performance problem and how to
fix it.

-- Snip --

6.3 The SSHv2 and SFTP Performance Handbrake

In 1977, Ward Christensen created the Xmodem data transfer protocol
[Christensen 1977].  Coming in an era of 300bps modems and unreliable links,
Xmodem divided data into 128-byte packets and required an Ack to be sent for
each packet before the next one could be transmitted.  As modems became faster
and links more reliable, the need to Ack each Xmodem packet became more and
more of a performance handbrake, since no matter how fast or reliable the
link, no more than 128 bytes of data could be sent without waiting 1 RTT for
the Ack.  The solution to the problem was to increase the packet size (Ymodem,
Xmodem-1K), and drop the requirement to Ack each packet (Ymodem-g, Zmodem)
[Forsberg 1988].  The latter was perfectly acceptable, since by then modems
included their own error correction and flow control mechanism.

Unfortunately this performance handbrake was reinvented in the SSHv2 protocol.
Like Ymodem-g and Zmodem running over modern modems, TCP/IP provides a
reliable, flow-controlled transport layer for the SSH protocol.  SSHv2 however
introduced an additional form of flow control that, like Xmodem, requires the
receiver to Ack each packet before more can be sent (the details aren't quite
as straightforward as this since the SSHv2 specification describes things in
terms of packets and data windows, but effectively it's the Xmodem per-packet
Ack).  Most implementations seem to use packet sizes of 16K or occasionally
32K, with some going as low as 4K.  What this means is that no matter how fast
the link, every (say) 16K the transmission stops for 1 RTT until the other
side has sent its Ack (referred to as a window adjust in SSHv2 terminology).
Consider for example the effect of this on a T1 international link with a
half-second RTT.  With the handbrake in operation, the link can run at only
17% of its total capacity.  This performance hit is so noticeable that it is
mentioned in the FAQs of some SSH implementations [PuttyFAQ].

In addition to the protocol-level handbrake, the SFTP protocol that runs on
top of SSH contains its own handbrake.  This protocol recommends that reads
and writes consist of no more than 32K of data, even though it's running over
the reliable SSH transport which is in turn running over the reliable TCP/IP
transport.  One common implementation limits SFTP packets to 4K bytes,
resulting in a mere 4% link utilisation in the previously-presented scenario.

The fix for this problem is obvious: Remove the handbrake.  This is no good
reason for the per-packet Ack, and certainly other protocols such as SSHv1 and
SSL/TLS function perfectly without it (the absence of the handbrake in SSHv1
is why SSH FAQs observe that the SSHv1 scp is so much faster than the SSHv2
SFTP, even though SFTP is overall a better design).  The effect of running
without the handbrake on were investigated using cryptlib with a fairly
rudimentary implementation of SFTP running over the built-in SSHv2.
cryptlib's SSH implementation has always set the window size to INT_MAX (some
implementations have problems with UINT_MAX as the window size), which
effectively disables the SSH-level handbrake.  The SFTP implementation
followed suit, requesting a read/write of the entire file at once rather than
breaking it up into little packets at the SFTP level (packetisation is already
handled at the SSH and TCP/IP layers).  Run over an international link (pretty
much a given when you're in New Zealand), this SFTP implementation was around
five times faster than the Putty implementation of SFTP talking to OpenSSH,
which sends data in 4K SFTP packets and (by extension) 4K SSH packets.  Even
over a low-latency link, the difference was impressive: cryptlib was an order
of magnitude faster than Putty on the loopback interface (latency being
relative in this case).

The SSH-level handbrake can therefore be provisionally removed by having
implementations set the window size to INT_MAX, and permanently removed by
deprecating the Ack/window-based flow control and perhaps optionally providing
Xon/Xoff-style flow control if absolutely necessary (as was mentioned earlier,
both SSHv1 and SSL/TLS function fine without requiring this).  The SFTP-level
handbrake can be removed by eliminating the maximum packet-size wording of the
SFTP specification, and recommending that implementations read and write all
data at once rather than engaging in additional redundant packetisation at the
SFTP level.

Most of this can be effected through a simple code change, however
implementors should be aware that many implementations will still stop and
wait for an Ack after a certain amount of data has been transmitted, even with
an effectively infinite-sized windows.  On the sender side things aren't quite
so bad, experimentation has shown that it's fairly safe to ignore the
receiver's window size and send data at the maximum rate possible, discarding
any window adjusts that arrive (the only slight complication is that it's
occasionally necessary to stop sending for a moment and clear the read channel
of the accumulation of Acks that have arrived while sending).  Since most
implementations include a facility for checking the peer's software version to
identify and work around implementation bugs, detecting pre-handbrake-fix
implementations and providing the appropriate slower interpretation of the
protocol should be relatively straightforward.  In addition, FAQs about the
poor performance of SFTP will need to be updated.

[Christensen 1977] "MODEM.ASM", Ward Christensen, August 1977 (the Xmodem
protocol was defined in terms of "What this program does" rather than being
formally documented, the author described it in a Compuserve post some years
later as "a quick hack I threw together").

[Forsberg 1988] "Xmodem/Ymodem Protocol Reference: A compendium of documents
describing the Xmodem and Ymodem File Transfer Protocols", Chuck Forsberg,
October 1988.

[PuttyFAQ] "PuTTY FAQ", Simon Tatham, 2003,
http://www.chiark.greenend.org.uk/~sgtatham/putty/faq.html, question A.6.8,
"PSFTP transfers files much slower than PSCP".

-- Snip --

While I'm pointing out things that should be fixed in the spec, my other big
gripe is the way the initial message is handled.  Currently the spec describes
a rather messy mechanism where both sides start by shouting at each other and
then engage in a complex dance to sort out what's what afterwards (the
"guessing" stuff).  This leads to really messy implementations when one of the
partners doesn't get the dance steps right.

There is no good reason for this complication in the protocol.  I don't buy
the RTT argument given in the SSH-transport draft, the guessing stuff saves
one whole RTT, but then the incredibly chatty authentication protocol ("Would
you like to authenticate then?" - "Yes I'd like to authenticate" - "How would
you like to authenticate?" - "Well, would the following suit you?" - "That
looks about right, let's do it" - "Right, I'm about to start" - etc etc etc)
more than makes up for any miniscule savings during the initial handshake.

The way to fix this is simple: Replace all the guessing stuff and the complex
rules that go with it with:

  Key exchange begins by each side sending lists of supported algorithms.  The
  server sends its list of supported algorithms first, the client chooses
  which ones it prefers that it also supports and sends back its choice in the
  reply.

That removes all of the handshake-dance complexity, and vastly simplifies
implementations.

Oh yes, in case anyone finds the SFTP info above useful and wants to reference
it for some reason, please cite it as '"Performance Characteristics of
Application-level Security Protocols", Peter Gutmann, to appear', since it's
not officially published yet.

Peter.

---

From: Martin Pool <[email-masked]>
Date: Tue, 8 Jul 2003 14:20:45 +1000

> resulting in a mere 4% link utilisation in the previously-presented

This reminds me of a problem in rsync: A fundamental protocol design
isssue is whether the recipient of a message should be able to
interrupt it during receipt to indicate an error.  The simplest and
cleanest solution is to require errors be sent after the request is
received, as SFTP does.

If the requests are very large, this causes a problem when, for
example, a destination file cannot be written to because of an
out-of-space error.  A simple server implementation will need to read
in the whole request before it is able to notify the client of the
error.  It is entirely realistic to have a file that takes many
minutes to transfer, and possibly incurs a significant traffic cost.
It is annoying to have all that time or money be wasted when the
server discards the request.

Alternatively the server can drop the connection, or send an
out-of-band error (e.g. over ssh stderr), but both of these introduce
other problems.  Similar issues can be seen to a lesser extent in
HTTP.

I think writing a file as a series of IO chunks is a reasonable and
clean solution but it requires that the length be chosen properly, and
ideally for pipelining to be used.

Of course I agree that 4k or even 32k is too small for most modern
networks.  But I did want to point out that going too far in the
opposite direction can be a problem too.

-- 
Martin

---

From: Markus Friedl <[email-masked]>
Date: Tue, 8 Jul 2003 09:12:16 +0200

On Tue, Jul 08, 2003 at 04:01:02PM +1200, Peter Gutmann wrote:
> TCP/IP provides a
> reliable, flow-controlled transport layer for the SSH protocol.

when multiplexing multiple ssh channels over one connection and one
of the consumers is very slow, then the tcp flow control will not
help. AFAIK, this is one of the reasons why there's an additional
per-channel flowcontrol.

---

From: Simon Tatham <[email-masked]>
Date: Tue, 08 Jul 2003 09:55:53 +0100

Peter Gutmann <[email-masked]> wrote:
> The SSH-level handbrake can therefore be provisionally removed by
> having implementations set the window size to INT_MAX, and
> permanently removed by deprecating the Ack/window-based flow control
> and perhaps optionally providing Xon/Xoff-style flow control if
> absolutely necessary (as was mentioned earlier, both SSHv1 and
> SSL/TLS function fine without requiring this).  The SFTP-level
> handbrake can be removed by eliminating the maximum packet-size
> wording of the SFTP specification, and recommending that
> implementations read and write all data at once rather than engaging
> in additional redundant packetisation at the SFTP level.

I'm unconvinced by this, since what you're throwing away here is
error reporting and controllability. The useful effect of breaking
down reads and writes into individual packets is that you get back
intermediate success and failure reports from each one.

If the server suddenly stops accepting data half way through writing
a 200Mb file - perhaps the disk has filled up - then ideally you
want that information communicated back to the client in a timely
manner, and not to already be committed to spewing another 100Mb of
data down a network connection whose other end can do nothing useful
with it. Similarly, if _your_ disk fills up half way through
receiving a 200Mb file, you want to be able to ask the server to
stop sending you data you can't use. This is a _problem_ with SCP,
which SFTP has fixed; don't throw out the baby with the bathwater.

I still think a better solution to the SFTP-level `handbrake' would
be to use the Kermit approach: simply queue k multiple requests at a
time, and every time the nth ack comes in, submit the (n+k)th data
packet. Adjust k appropriately (my vague plan is to stop sending
data packets as soon as my TCP layer reports that sent data is being
queued locally). This is what I plan to do to PSFTP in the
reasonably near future.

At the SSH level: I'm sure I remember trying to set packet sizes to
INT_MAX in an early version of PuTTY. This did cause me trouble,
although I don't recall the precise details. I wouldn't be too
surprised, though, if some implementation tried to allocate a buffer
the size of the window, to avoid fiddly memory allocations going on
all the time during channel operation. In which case your `ignore
window adjusts and send as fast as you can' strategy might well run
into trouble!

As Markus observes, the per-channel windows are vital for
flow-controlling multiplexed connections running at different rates,
so I doubt _unconditionally_ upping window sizes would be a useful
change to a general SSH implementation. I might give it a try in
PSFTP and PSCP only, however, and see if anyone's implementation
still barfs on the idea of a window that big.

Do you have any way to fix the SSH level for SFTP uploads? An SSH
client which knows it's being used for SFTP can change its own
maximum window size, so that the server can send arbitrary
quantities of data with no constraint beyond the TCP-level acks. But
for uploading, you'd have to persuade the SSH _server_ to change its
window size depending on the use the client planned to put the
connection to. (Having the server unconditionally set all window
sizes to INT_MAX is a silly idea, for multiplexing reasons.)

> While I'm pointing out things that should be fixed in the spec, my
> other big gripe is the way the initial message is handled.
[...]
> The way to fix this is simple: Replace all the guessing stuff and
> the complex rules that go with it with:
> 
>   Key exchange begins by each side sending lists of supported
>   algorithms.  The server sends its list of supported algorithms
>   first, the client chooses which ones it prefers that it also
>   supports and sends back its choice in the reply.

Well, no problem there. All you have to do to get this fixed is to
point it out three years ago, _before_ the draft RFCs were in last
call and a large number of fairly polished implementation existed! :-)
-- 
Simon Tatham         "Every person has a thinking part that wonders what
<[email-masked]>    the part that isn't thinking isn't thinking about."

---

From: Markus Friedl <[email-masked]>
Date: Tue, 8 Jul 2003 11:00:48 +0200

> resulting in a mere 4% link utilisation in the previously-presented scenario.

OpenSSH's SFTP client sends multiple read-requests before waiting
for a reply, so I don't think that the 32K limit is a big problem
(you can try to experiment with sftp(1)'s -B and -R option).

---

From: Markus Friedl <[email-masked]>
Date: Tue, 8 Jul 2003 11:02:42 +0200

> help. AFAIK, this is one of the reasons why there's an additional

I'd like to add that ssh/sshd act as a 'distributed' TCP-proxy,
so they break end-to-end TCP flow control for forwarded TCP
connections.

From this point of view, the channel flow control mechanism makes
very much sense.

---

From: "denis bider" <[email-masked]>
Date: Tue, 8 Jul 2003 11:54:19 +0200

Peter,

I totally agree that the SSH protocol in some places is more complicated
than it needs to be; however, the protocol does not impede performance
if properly implemented. According to our experience, it is quite
possible to develop a fast SSH implementation if only one invests some
thought and care into its design. The ssh.com implementation as well as
ours are proof that speeds of several MB per second can be obtained. 

Further, there are now dozens of implementations that would be broken by
your suggestions unnecessarily. While your proposals would make it
easier for a naive developer to put together a fast implementation, they
remove functionality that makes the protocol complete - making a so-so
protocol out of a sophisticated one. 

The protocol provides you with the means to control the flow of the
session, but it doesn't tell you how to do it efficiently. If you want
to resolve this, the solution is to write a document describing how to
use the control elements for best performance; not to remove them
altogether. The protocol is fine.

Regards,

denis

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Peter Gutmann
> Sent: Tuesday, July 08, 2003 06:01
> To: [email-masked]
> Subject: Why SFTP performance sucks, and how to fix it
> 
> 
> algorithms.  The

---

From: Markus Friedl <[email-masked]>
Date: Tue, 8 Jul 2003 14:58:48 +0200

> partners doesn't get the dance steps right.

I think that the 'guessing" stuff' is ugly, but it's not very hard
to skip the first kex packet in case the 'guess' is wrong.

---

From: Bill Sommerfeld <[email-masked]>
Date: Tue, 08 Jul 2003 10:24:24 -0400

I believe I may have been responsible for demonstrating to one of the
sshv1 implementors why the omission of per-channel flow control was a
mistake.

I don't recall exactly *which* test case I used to demonstrate it but
it either involved regular connection forwarding or X11 forwarding.

In short, if you pause an entity reading off a channel, you either
need infinite buffering in the ssh receiver point (to allow the
receiver to read and buffer arbitrary amounts of data for the paused
channel), or else you will end up flow-controlling the ssh connection,
possibly leading to a deadlock (because the only way to unpause the
channel is by sending data over another channel..)

example: stopping an X client using either UNIX job control or a
debugger.  the X client will keep receiving events from the X server
while stopped; eventually enough will build up to begin to require
flow control backpressure..

If the main ssh pipe is blocked, you can't send the "continue"..

Now, I think the trick for good performance is to ensure that the ssh
windows are large enough that when the applications are *not* flow
controlled you can keep the the TCP pipe filled more often than not.
I suspect this means "somewhat larger than the TCP window".  But some
experimentation is likely in order..

						- Bill

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 8 Jul 2003 08:46:10 -0700

Good write up!

Implementations could start with small SECSH channel window sizes and
grow them as needed so as to maximize throughout without creating
bottlenecks.  A slow-start for SSHv2, so to speak.

Consider an implementation such as OpenSSH where the SECSH protocol and
the SFTP protocol are implemented in separate processes communicating
over IPC.  With such an implementation one might, in some situations,
end up having a bottleneck in the IPC connection, rather than in the
client-server TCP connection, and under such circumstances one might
like the SECSH channel flow control mechanism to work as advertised.

But when there is no bottleneck flow control does become a "handbrake."

And I've already pointed out that SFTP has no Ack handbrake.

> partners doesn't get the dance steps right.

I don't think the optimistic key exchange concept is difficult to
implement.  I am, however, bothered that only one kex attempt can be
made per-connection (whereas many authentication attempts can be made
per-connection).

I also think that the kex negotiation is a bit clumsy in some ways - see
the recent threads started by Joel N. Webber II.

On the other hand, the key exchange negotiation part of the protocol and
how it factors into the session ID generation is a crucial aspect of the
protocol's security, and so keeping it simple/stupid is actually quite a
reasonable thing to do.

>   which ones it prefers that it also supports and sends back its choice in the

We've recently seen that exchanging lists of algorithms is not good
enough.

> That removes all of the handshake-dance complexity, and vastly simplifies

Of course, it's really too late to consider modifying the key exchange
protocol.  Perhaps for version 2.1 of the protocol...

Cheers,

Nico
--

---

From: RJ Atkinson <[email-masked]>
Date: Tue, 8 Jul 2003 12:02:27 -0400

An Informational RFC describing the issues here, including why
we are where we are with the spec, but also discussing the other
implications and possible future directions to consider, would
seem useful.

Starting with an Informational RFC (rather than discussing immediate
changes to specifications that are long overdue) would avoid de-railing
the current effort to get the specifications out as RFCs, document the
issues clearly for the benefit of both implementers and operators,
and help focus discussion in the WG on possible paths forward.

IMHO,

Ran

---

From: Markus Friedl <[email-masked]>
Date: Tue, 8 Jul 2003 18:05:25 +0200

> like the SECSH channel flow control mechanism to work as advertised.

feel free to send patches....

---

From: Jeffrey Altman <[email-masked]>
Date: Tue, 08 Jul 2003 12:19:02 -0400

If someone wants to experiment, feel free to take a look at the SKERMIT 
(Secure shell Kermit) subsystem which is available for use with OpenSSH.

   http://www.kermit-project.org/skermit.html

This is simply C-Kermit 8.0 running as an SSH subsystem just like SFTP. 
  You can control from the C-Kermit or Kermit 95 client whether or not 
you wish to use Streaming Kermit transfers (the default) or traditional 
Sliding-windows Kermit with slow start.  When slow start is used the 
packet size starts small and grows to the maximum implemented 9024 bytes 
(Kermit protocol allows packets up to 866,304 bytes but no one has ever 
  required it.)

It might give you a better idea of the performance characters.

Jeffrey Altman

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 8 Jul 2003 11:01:11 -0700

> feel free to send patches....

Before we talk about patches I'd like to get a few more details
straightened out.

OpenSSH has a per-channel max window size that controls the max value
for the actual window size.

I think a way slow-start (and slow-stop) could be implemented would be
to adjust the per-channel max window size according to the status of the
per-channel data sink when data is received:

 - Slow-start

   When data is received on a channel and the channel's sink is
   writable, increase the channel's max window size by some amount.

 - Slow-stop

   When data is received on a channel and the channel's sink is not
   writable, reduce the channel's max window size by some amount, but
   don't allow the channel's max window size to become lower than the
   actual window size.

The question is: what should this "some amount" be?  It could a
constant, or the size of the data received that triggered the max window
size adjustment.  Or maybe the max window size adjustments algorithm
could be more aggressive so that slow-starts/stops aren't that slow...

Cheers,

Nico
--

---

From: Markus Friedl <[email-masked]>
Date: Tue, 8 Jul 2003 20:33:42 +0200

On Tue, Jul 08, 2003 at 04:01:02PM +1200, Peter Gutmann wrote:
> What this means is that no matter how fast

Well, the problem here is the window size, not the packet size.

> top of SSH contains its own handbrake.

That's not true.  While early SFTP implementations did wait for the
per-packet ACK, most recent implemenations (both ssh.com and OpenSSH)
send multiple concurrent requests.  For example, when running with
2 concurrent requests (sftp -R 2) the throughput improves from
~600K/s to ~1.9M/s for simple tests.

> SSL/TLS function perfectly without it (the absence of the handbrake in SSHv1

SSL/TLS does not do port-forwarding, and SSHv1's lack of flow
control can cause problems when multiplexing different
channels over SSH.

> SFTP, even though SFTP is overall a better design).

With OpenSSH's default (16 concurrent requests) I almost get 
scp(1) like performance in sftp(1), so I doubt it's a SFTP protocol
issue.

> implementations set the window size to INT_MAX, and permanently removed by

This would mean that the implementatios have to buffer up to INT_MAX
bytes per channel.

> Xon/Xoff-style flow control if absolutely necessary

SSH is more than just a file transfer protocol.

> (as was mentioned earlier,
> both SSHv1 and SSL/TLS function fine without requiring this).

Again SSHv1 does not function fine :)

> The SFTP-level

Interleaving requests could be mentioned in the SFTP spec in an
implementation hint section.

> experimentation has shown that it's fairly safe to ignore the
> any window adjusts that arrive

I hope there are not many SSH implementations out there doing this :)

-markus

---

From: "Joel N. Weber II" <[email-masked]>
Date: Tue, 08 Jul 2003 18:13:29 -0400

> made per-connection (whereas many authentication attempts can be made

In looking at the transport draft for the word ``disconnect'', it
seems that the places where disconnection is explicitly a MUST related
to key exchange are places where there are no algorithms in common at
all, the idea apparently being that you list all the algorithms that
are supported, because you either support a particular key exchange
algorithm, or you don't, and you either are willing to use a
particular host key algorithm, or you aren't, etc.

But I don't see any reason why we would need to define a different
protocol version number if we wanted to offer the option of trying a
second round of key exchange when doing initial keying.  A Secure
Shell implementation that wants to do this could simply use
SSH_MSG_DEBUG rather than SSH_MSG_DISCONNECT for error reporting, and
then assume that doing another round of key exchange will work.  Any
implementation that follows the current draft will presumably decide
that it's got a fatal error when it sees a key exchange message in the
wrong place, and will fall back to the old behavior of simply
terminating the connection.

And when rekeying, you could simply keep using the old keys if
rekeying fails, and rekey early enough that you'll be able to retry a
few times before your keys actually need to expire.

> protocol's security, and so keeping it simple/stupid is actually quite a

Unless you can get into a state where the server thinks that key
exchange was successful and the client thinks it was unsuccessful, or
vice versa, I'm not sure why this actually becomes a problem.

The places key exchange can fail when both sides support the same
algorithm, assuming the connection doesn't break and you don't have a
man in the middle:

1) GSSAPI failures, which generally are going to happen early in the
   process, before you successfully transfer your large numbers (since
   the public diffie-hellman numbers get encrypted with GSSAPI).

2) The client deciding that it doesn't trust a particular public key.
   This is more annoying in some ways, since the public key is sent at
   the end of the process, and the server has already done all the
   computation it is going to do for key exchange by the time the
   client is able to figure out that there is a problem.  In
   particular, I think a server could send SSH_MSG_NEWKEYS right after
   it sends the message containing its public key and the signature,
   at which point, if the client decides it doesn't trust that key,
   you have a random untrusted encryption key encrypting one direction
   of the second attempt at initial key exchange.

   However, if the rule is that the session ID used for the session by
   the userauth protocol is taken from the first key exchange which
   resulted in both the client and the server sending SSH_MSG_NEWKEYS,
   I don't see where this is a problem.

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 8 Jul 2003 15:22:31 -0700

[OT]

>    the public diffie-hellman numbers get encrypted with GSSAPI).

I've experienced this one enough times (for a variety of reasons) that
I've wished that the client could then re-try w/o gss keyex.

Cheers,

Nico
--

---

From: Joseph Galbraith <[email-masked]>
Date: Tue, 08 Jul 2003 16:35:43 -0600

>sizes to INT_MAX is a silly idea, for multiplexing reasons.)
Well, the problem is that subsystems take a channel of type a and turn
it into a channel of type b.  If the server knew up front what you wanted,
it could pick appropriate values.  I've always thought subsystems were
a bad idea.

That said, the server could notice when you make a sftp-subsystem request
and send you a window adjust message for window you haven't consumed.
(I don't believe anything in the drafts prohibits this.)  This would 
essentially
increase your total available window.

Unfortunately, there is nothing that can be done about the maximum
packet size on the channel (which also plays into this whole mess.)
Well, we could introduct a channel request to notify the remote pee
that the per channel max packet size has been changed could be sent.
If that channel request succeeds, then larger packets could be sent.

- Joseph

---

From: Joseph Galbraith <[email-masked]>
Date: Wed, 09 Jul 2003 16:19:51 -0600

>side has sent its Ack (referred to as a window adjust in SSHv2 terminology).
The ack (a window adjust message) can be sent for any amount of data at
any time.  There is not one-to-one correspondane between adjust messages
and packets.

In addition, if the the window size is greater than the packet size, one 
could
easily send acks in a timely fashion such that the remote side should never
run out of window.  I.e., if the max packet size on a channel is 16K, 
but the
windows size is 128K, 8 packets must be sent before the window is exhausted
and the sender must halt and wait for an adjust message.  If the 
reciever sends
adjust messages after receiving 32 K of data, additional window should 
become
available to the sender before it exhausts its window, and no throttling 
should
ever occur.

>resulting in a mere 4% link utilisation in the previously-presented scenario.
Because reads and writes do not have to be synchronous, this is only a 
'hand-brake'
if implementations choose to make it so.  In other words, even if my 
writes must
consist of no more than 32K packets, I need not wait for the status of 
that packet
before sending the next.  I'm free to send 32K packets out as fast as I 
can (limitted
by the channel window, but as stated above, that shouldn't be an issue 
as long
as the server can actually process the data as fast as I send it.)  The 
results
of those operations will then come back in their own due time.

Similarly, I can send as many read requests, each for 32K, to the server
as I want.  The server should then process them, streaming data to me
in 32K packets.  If I give it another read request as soon as it completes
one, it should be continually processing 32K read requests until the file
is finished.

Since the packet length must be determined before the packet can be sent,
it might be burdensome for implementations to be required to read huge
chunks of data and send it.  What if I issue a read request for 4 gigabytes?

Sure, I could send the packet header, claiming 4 gig of data and begin 
reading
data out of the file in reasonable sized chunks and sending it down the 
wire.
But what if the file is truncated to 2 gig after I've read and sent one 
gig of
it?  What if the file is on a server and that server fails after 2 gig have
been read.  They are corner cases, but if the protocol is incapable of
handling such things, the protocol is flawed.

However, in order to facilitate writing higher performing SFTP clients, 
I have
considered that it might make sense to add a second read command to SFTP,
which allowed reads of any size, but which can result in multiple data 
packets,
culminating in a final status packet.

I.E.
    Client: SSH_FXP_MULTI_READ: req-id=243, offset=0, bytes=4,000,0000
    Server: SSH_FXP_DATA: req-id=243, 64K
    Server: SSH_FXP_DATA: req-id=243, 64K
    Server: SSH_FXP_DATA: req-id=243, 64K
    ...
    Server: SSH_FXP_DATA: req-id=243, 64K
    Server: SSH_FXP_STATUS: req-id=243, SUCCESS

or
    Server: SSH_FXP_DATA: req-id=243, 64K
    Server: SSH_FXP_DATA: req-id=243, 64K
    Server: SSH_FXP_STATUS: req-id=243, EOF

This would allow a client to request the entire file in a single read 
operation
without requiring the server to allocate huge buffers or being vulnerable to
corner cases (such as mid-read truncation.)

>SFTP, even though SFTP is overall a better design).
I believe others have noted that SSHv1's through-put is throttled by the
slowest channel.

Try the following with SSHv1:  write a simple socket server that never reads
the socket.  Write a simple socket client that sends data.  Connect the 
client
to the server through a port forward.  Eventually your entire connection 
will
stall.  You can no longer type at the shell prompt.  Other port forwards 
no longer pass
data.

Any protocol which can transfer multiple streams of  independant data has
to have a per stream flow control mechanism.  Otherwise, the entire protocol
has to be stalled for the slowest stream.

- Joseph

---

From: "Joel N. Weber II" <[email-masked]>
Date: Wed, 09 Jul 2003 18:41:43 -0400

So, thinking a bit more about what I said yesterday, I think if we
want to support multiple attempts at key exchange, the correct
sematics are:

If you fail trying to do GSSAPI key exchange, use the messages already
defined for GSSAPI key exchange error reporting, but don't disconnect.

If you send the last message in a key exchange sequence, wait to see
if SSH_MSG_NEWKEYS comes.  If it does, your peer accepted what you
sent in that last message, and you can send SSH_MSG_NEWKEYS too.
(This avoids having only one side use keys from a key exchange: you
get either both or neither, which simplifies the session identifier
question a bit.)

If you need to try again, just send SSH_MSG_KEXINIT again.

I'm willing to write up an internet-draft along these lines after IETF
if people think it's a good idea.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 9 Jul 2003 15:53:27 -0700

> get either both or neither, which simplifies the session identifier

If the client got an error from the peer then it knows that the
SSH_MSG_NEWKEYS won't come and so it can just try again immediately.

> if people think it's a good idea.

There's a problem though, and I'm not sure how it could be surmounted:

The session ID needs to be derived from all messages involved in the key
exchange, even for the failed exechanges so as to avoid downgrade
attacks.

But the session ID computation is kex-type specific so you'd have to
modify the formula for the session ID for each of the three(?) existing
kex methods.  This is most unfortunate.

Perhaps the session ID computation should be respecified to be as
follows:

H = hash(V_C || V_S || KEX_C1 || KEX_S1 || KEX_C2 || KEX_S2 || ...  ||
         KEX_Cn || KEX_Sn || <kex-method specific stuff>)

where KEX_Cx and KEX_Sx are the key exchange messages sent by the client
and server, respectively and where <kex-method specific stuff> is the
(K_S || e || f || K) part of the existing kexes.

Cheers,

Nico
--

---

From: "Joel N. Weber II" <[email-masked]>
Date: Wed, 09 Jul 2003 20:12:52 -0400

> SSH_MSG_NEWKEYS won't come and so it can just try again immediately.

True.

The case I was thinking of, though, is the case where the client
decides it doesn't trust the certificate presented by the server, and
that it wants to try another method.  For example, a common problem is
that your average web browser doesn't recognize the X.509 CA that
issues certificates for most HTTPS web servers in the mit.edu domain.
And with OpenPGP, everyone has their own set of trust values, and
there's no chance you could ever predict whether a client will trust a
host key without feeding the client the host key and asking whether it
trusts it.

Since the host key is sent in the last message of the public key key
exchange algorithms, the client doesn't realize that it doesn't trust
the key until key exchange would appear to have finished judging from
the messages being passed back and forth.

> exchange, even for the failed exechanges so as to avoid downgrade

You're right; it makes perfect sense, but I hadn't realized that.

(Tangentially, if an attacker can intercept the messages between your
application server and your Kerberos KDC, there is probably a
downgrade attack there, unless your GSSAPI implementation is really
clever.  We may want to say that a GSSAPI implementation MUST abort if
an error happens which indicates that communication with the KDC
failed.  Except that, like, KDC isn't a GSSAPI term.)

> kex methods.

Every kex method, I believe, has a V_C and a V_S input for data that
needs to be in the hash that came before the key exchange algorithm
started running.  I think all we have to do is define that for the
first attempt at initial keyex, and for any rekeying (yes, ugh about
the ``for any rekeying, but that's where we are now), they are what
they are now, and for attempts at initial keyex after the first, V_C
and V_S somehow end up being a hash of initial V_C and V_S plus the
previous attempts at keyex.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 9 Jul 2003 17:19:06 -0700

> failed.  Except that, like, KDC isn't a GSSAPI term.)

Well, no.  While what you say about Kerberos is true, the client would
fail to acquire GSS-API credentials in such cases and so would not even
propose gsskeyex.

(Incidentally, that is one problem that we'll try to address with the
Kerberos extensions work.)

> previous attempts at keyex.

The problem is that we're talking about modifying existing kex methods
that are defined in multiple different documents; although we're
modifying them to take into account a mode of operation not currently
available, we would be partially obsoleting these docs, and in terms of
IETF process I think that'll mean re-spinning the RFCs (don't even think
of getting this into the I-Ds now - or, rather, you can try, but I think
you'll be told by all that it's too late).

Cheers,

Nico
--

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Wed, 16 Jul 2003 09:58:53 +0200 (CEST)

On Wed, 9 Jul 2003, Joel N. Weber II wrote:

> decides it doesn't trust the certificate presented by the server,

But this case is simple to handle -- you disconnect and try again.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 16 Jul 2003 06:20:43 -0700

> But this case is simple to handle -- you disconnect and try again.

If keyex was re-triable and the session ID hash was taken over the
failed kex messages too (in addition to the other things that go into
it) then it would be possible to detect downgrade attacks.

Disconnecting after kex failure without the possibility of retrying
may leave the user (oh, probably not the average user) wondering if a
downgrade attack was not taking place.

Of course, all of this assumes that one kex is weaker, or, if not, less
desirable, than another.

Anyways, we can fix this some other time.

Cheers,

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 16 Jul 2003 12:12:32 -0700

> the messages being passed back and forth.

BTW, this would apply to GSS-API keyex with SPKM as well.

I think the problem can be fixed without revving the protocol version,
but it wouldn't be pretty (I imagine using a bogus alg name to indicate
support for retriable keyex).  If has to be pretty then the protocol
will have to be revved.  Either way the fix will have to wait.

Cheers,

Nico
--

---

From: "Joel N. Weber II" <[email-masked]>
Date: Wed, 16 Jul 2003 15:23:24 -0400

> BTW, this would apply to GSS-API keyex with SPKM as well.

What does SPKM do that GSI and x509v3-sign-rsa/x509v3-sign-dss don't?
It's bad enough that we already have lots of potential for two
gratuitously incompatible ways to use simple bare Verisign-signed
X.509 certificates.  (And I think there are implementations of both
approaches out there.)

If we want to discuss another GSSAPI mechanism that might possibly be
worth supporting in the future, SRP might be more interesting to
discuss.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 16 Jul 2003 13:31:19 -0700

> What does SPKM do that GSI and x509v3-sign-rsa/x509v3-sign-dss don't?

Nothing.  That's the point.  By the time that the SPKM initiator
(client) finds out what the acceptor's (server's) certificate is the
client and server have committed to doing SSHv2 GSS-API kex w/ SPKM and
CAN'T fallback on another kex method should the acceptor's cert not be
good enough for the client.

I was pointing out more scenarios in which the inability to re-try kex
could be obnoxious.

> worth supporting in the future, SRP might be more interesting to

Not in this WG.

Cheers,

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 16 Jul 2003 15:53:06 -0500

> > worth supporting in the future, SRP might be more interesting to

I should correct myself here: if you can get the charter of SECSH
ammended so it can design a new GSS-API mechanism using SRP, go for it.

The CAT WG is not open, so there's no single best forum to discuss new
GSS-API mechanisms.  If that is what you want then I see some options:

 - discuss the new mechanism in the old CAT WG mailing list and proceed
   with an individual I-D

 - get some WG's charter amended so it can be responsible for the
   proposed GSS-API mechanism

 - get a new WG created or have CAT revived

I think a number of issues are slowly piling up that ought to lead to
the revival of CAT, eventually.

Cheers,

Nico
--

---

From: "Joseph Salowey" <[email-masked]>
Date: Wed, 16 Jul 2003 14:31:12 -0700

I believe 

http://www.ietf.org/internet-drafts/draft-burdis-cat-srp-sasl-08.txt

Is an SRP SASL mechanism that also uses GSSAPI framing, perhaps this is
enough (I haven't looked at it yet).

Joe

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Nicolas Williams
> Sent: Wednesday, July 16, 2003 1:53 PM
> To: Joel N. Weber II
> Cc: [email-masked]
> Subject: GSS-API SRP mech (was Re: retrying keyex ...)
> 
> 
> > > If we want to discuss another GSSAPI mechanism that might 
> possibly 
> > > be worth supporting in the future, SRP might be more 
> interesting to 
> > > discuss.
> > 
> > Not in this WG.
> 
> I should correct myself here: if you can get the charter of 
> SECSH ammended so it can design a new GSS-API mechanism using 
> SRP, go for it.
> 
> The CAT WG is not open, so there's no single best forum to 
> discuss new GSS-API mechanisms.  If that is what you want 
> then I see some options:
> 
> I think a number of issues are slowly piling up that ought to 
> lead to the revival of CAT, eventually.
> 
> Cheers,
> 
> Nico
> -- 
>

---

From: "Joel N. Weber II" <[email-masked]>
Date: Wed, 16 Jul 2003 18:02:31 -0400

> enough (I haven't looked at it yet).

It appears that that by itself doesn't provide integrity protection,
and draft-ietf-secsh-gsskeyex says:

   The key exchange method described in section 1 of this document
   depends on the underlying GSSAPI mechanism to provide both mutual
   authentication and per-message integrity services.  If either of
   these features is not supported by a particular GSSAPI mechanism, or
   by a particular implementation of a GSSAPI mechanism, then the key
   exchange is not secure and MUST fail.

In theory, you can use that SASL mechanism with an intergrity
protection layer.  In practice, since it appears that the SASL SRP
mechanism basically does the things you want a Secure Shell key
exchange to accomplish, it may be better to define a new Secure Shell
key exchange algorithm to support SRP.

However, this is probably a mostly academic discussion at the moment,
due to IPR issues related to SRP.  Fortunately, patents do expire
eventually.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 16 Jul 2003 15:22:44 -0700

> enough (I haven't looked at it yet).

I've just read the relevant portions of that I-D and find it lacking.  I
have a number of objections, particularly to section 8 where the GSS-API
mechanism is defined.

This seems really way off-topic for [email-masked]; I do not expect
to continue this topic here.

Nico
--

---

From: Tom Wu <[email-masked]>
Date: Wed, 16 Jul 2003 15:51:45 -0700

> key exchange algorithm to support SRP.

There are implementations of SRP as an SSH key exchange mechanism 'in 
the wild', and at least one expired I-D documenting them.  They can 
presumably be revived as needed.

> due to IPR issues related to SRP.  Fortunately, patents do expire

But SRP has a royalty-free license: http://www.ietf.org/ietf/IPR/WU-SRP. 
  Although there is some concern over third-party IPR, that should not 
(and has not) prevented movement on standards documents in this space. 
Ultimately it is up to the marketplace to decide.

Tom
-- 
Tom Wu
Chief Security Architect
Arcot Systems
(408) 969-6124

---

From: Markus Friedl <[email-masked]>
Date: Thu, 17 Jul 2003 10:04:41 +0200

> Ultimately it is up to the marketplace to decide.

Things like this prevent SRP support in OpenSSH, for example.

---

From: Bill Sommerfeld <[email-masked]>
Date: Thu, 17 Jul 2003 04:45:59 -0400

> ammended so it can design a new GSS-API mechanism using SRP, go for it.

The current working group chair (me) would very much like the working
group to finish up, ship a bunch of documents and go inactive.

Expanding the charter doesn't sound like the way to get there.

					- Bill

---

From: Sam Hartman <[email-masked]>
Date: Thu, 17 Jul 2003 07:59:24 -0400

>>>>> "Tom" == Tom Wu <[email-masked]> writes:

    Tom> Joel N. Weber II wrote:
    >> protection layer.  In practice, since it appears that the SASL
    >> SRP mechanism basically does the things you want a Secure Shell
    >> key exchange to accomplish, it may be better to define a new
    >> Secure Shell key exchange algorithm to support SRP.

    Tom> There are implementations of SRP as an SSH key exchange
    Tom> mechanism 'in the wild', and at least one expired I-D
    Tom> documenting them.  They can presumably be revived as needed.

Please don't; use Keith's GSSAPI mechanism.  I believe that getting
his draft in shape should not take too much time at all.

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 17 Jul 2003 14:24:57 +0200 (CEST)

> the revival of CAT, eventually.

Well, the creation of a new WG, since cat has concluded, but yes.
However, a number of the people involved have been rather busy with
Kerberos lately, and so we've been putting off trying to spin up KITTEN
(BTW, if you can think of a name to go with that acronym, let me know).
At this point don't expect to see that work begin before late next year...

-- Jeff

---

From: Simon Tatham <[email-masked]>
Date: Thu, 17 Jul 2003 13:55:25 +0100

Sam Hartman  <[email-masked]> wrote:
[existing implementations of SRP authentication in SSH]
> his draft in shape should not take too much time at all.

Stop me if I'm being completely ignorant, but I thought GSSAPI
required the use of a null host key?

One of the great things about SRP as implemented in the hacked
OpenSSH I saw is that it uses the shared secret output from the SRP
exchange to validate the ordinary SSH host key - so that you can
connect to a machine whose host key you don't already know, enter
your SRP passphrase in confidence that it can't be stolen even if
the server has been spoofed, and if the SRP exchange completes
successfully then you can also be sure that _that host key_ belongs
to a machine which already knew what your passphrase should have
been. Then, later, you can use other forms of authentication to talk
to the same server and now be confident of its host key.

I think that any SRP implementation which worked by avoiding the
host key completely would be a step backwards from this. The whole
host key problem is a major hassle to many SSH users, and making it
simpler would be a serious gain.

Cheers,
Simon
-- 
Simon Tatham         "You may call that a cheap shot.
<[email-masked]>    I prefer to think of it as good value."

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 17 Jul 2003 15:46:23 +0200 (CEST)

> required the use of a null host key?

No.  The GSSAPI key exchange mechanisms support the use of any server host
key type, and do provide a mechanism to (optionally) transport and
authenticate the server host key in a secure fashion.

The GSSAPI key exchange document defines the 'null' host key type so that
hosts which have no host key can satisfy the following requirement,
included in section 5.1 of the ssh transport draft, under the description
of the 'server_host_key_algorithms' field:

         Algorithm selection depends on whether the chosen key exchange
         algorithm requires a signature or encryption capable host key.
         It MUST be possible to determine this from the public key
         algorithm name.  The first algorithm on the client's list that
         satisfies the requirements and is also supported by the server
         MUST be chosen.  If there is no such algorithm, both sides MUST
         disconnect.

Servers will list only host key algorithms which they can actually use; if
no host key is present for a particular algorithm, then that algorithm
will not be listed.  What this means is that if a host has no host keys at
all, then no algorithms will be listed, and the requirement in the
above-quoted paragraph cannot possibly be satisfied.  What the 'null' host
key algorithm does is provide a 'placeholder' to be included in the list
when no other host key algorithms are present.

-- Jeffrey T. Hutzelman (N3NHS) <[email-masked]>
   Sr. Research Systems Programmer
   School of Computer Science - Research Computing Facility
   Carnegie Mellon University - Pittsburgh, PA

---

From: Tom Wu <[email-masked]>
Date: Thu, 17 Jul 2003 12:31:20 -0700

> simpler would be a serious gain.

I agree completely - one of the big benefits of SRP is that it does both 
strong password authentication *and* key exchange, and a protocol can 
leverage the exchanged key for subsequent secure communications.  The 
important technical property is the tying of the SRP exchange to the 
protocol's native session security layer.  As of now, the two popular 
approaches are:

- Use the SRP session key directly as a session key (e.g. SRP/TLS)
- Use the SRP key exchange to validate another, previously 
unauthenticated, key exchange (e.g. SSH).

The latter approach seems useful for SSH since its own key exchange 
mechanism is based on the host key.  I am also ignorant of how GSSAPI 
would handle this.  The important question to ask is:  What happens if 
the SSH host public key is modified by an attacker in transit when doing 
an SRP authentication?  With the patched OpenSSH, since it includes a 
hash of the public key inside the SRP verification messages, it would 
cause authentication to fail, thwarting the MITM attack.

Tom
-- 
Tom Wu
Chief Security Architect
Arcot Systems
(408) 969-6124

---

From: Markus Friedl <[email-masked]>
Date: Fri, 18 Jul 2003 09:36:27 +0200

On Thu, Jul 17, 2003 at 12:31:20PM -0700, Tom Wu wrote:
> With the patched OpenSSH, since it includes a 

I don't see how this is an improvement over pubkey auth, since
it allows you to detect a MITM attack as well.

---

From: Simon Tatham <[email-masked]>
Date: Fri, 18 Jul 2003 09:11:12 +0100

> it allows you to detect a MITM attack as well.

It isn't an improvement over pubkey auth. It's an improvement over
_password_ auth - the authentication method you use when you're
logging in from (say) a new computer and don't have your private key
conveniently to hand.

Once you have bootstrapped your authentication of the host using an
SRP login, you can then create a public key on your new system and
set the server up to recognise it.

Cheers,
Simon
-- 
Simon Tatham         "That all men should be brothers is a
<[email-masked]>    dream of people who have no brothers."