Subject: Re: Sftp: performance enhancing changes

From: [email-masked] (Peter Gutmann)
Date: Tue, 9 Sep 2003 23:12:09 +1200

Bill Sommerfeld <[email-masked]> writes:
>>>o Ignoring channel window makes for a screaming
>>>  sftp transfer. (unacceptable, breaks multiplexing.)
>>
>>It's only unacceptable if you're actually doing multiplexing.  If you're just
>>using SFTP as a secure FTP replacement (which is what many people seem to be
>>using it for), it's a simple, quick fix for the SFTP performance problems.
>
>My understanding based on reading the earlier discussion is that this is a
>red herring -- the fix (which doesn't involve violating the specs) is to
>pipeline read/write requests and ensure the channel window is substantially
>larger than the TCP window).

Two comments on this..

1. It doesn't involve violating the spec.  You just advertise a maximum-size
   window (for the SSH level) and read a file as a single large chunk rather
   than lots of little bits and pieces (at the SFTP level).  Neither violate
   the spec.

2. Pipelining reads/writes and whatnot is a nice theoretical fix, but the fact
   that this problem has been around for what, five years now without anyone
   fixing it, and is enough of a problem that it comes up again and again in
   newsgroups, mailing lists, etc etc, indicates that just because a
   theoretical fix exists doesn't mean that implementations are able or
   willing to make use of it (heck, Tanenbaum's 1981 edition of Computer
   Networks covered this problem, and yet it's still present in stuff released
   this year, which indicates that it's not going to go away any time soon).
   
   Let me provide a concrete example of where this sort of thing leads... the
   motivation for me originally doing the handbrake writeup was a conversation
   with someone at a large US financial institution who were using SFTP to
   transfer some sort of large money-related files around (batch EDI
   transactions I assume).  They were doing this over fixed-bandwidth, very
   expensive leased lines (slow, high-latency, the perfect environment for the
   handbrake to take effect), and found that they were getting only a fraction
   of the expected throughput.  Their options were to either pay for more
   expensive leased lines, or (as they eventually found via trial and error)
   switch to SSL.  In the end they dumped SSH and switched to SSL instead,
   which, even though it does exactly the same thing as SSH, ran at the full
   link speed.  I offered to do them a handbrake-free STFP on top of my
   handbrake-free SSH code, but by then they had committed to SSL.
   
So although it's possible to say "We have a theoretical fix which, if
correctly applied by both sides and under ideal conditions, should work OK",
the fact that anything except carefully-optimised, tuned SSH implementations
provide poor throughput isn't really helping anyone except the SSL vendors,
whose implementations don't have this problem.  Since there are a large number
of SSH implementations out there (surprisingly so, it's turning up as an add-
on to all sorts of Windows apps that do file/data transfer, which is nice to
see), more and more stuff is being SSH-enabled by people who don't know about
the handbrake and don't know or care about spending a lot of time doing the
necessary tuning to get adequate performance.  The situation is just going to
get worse...

Peter.

---

From: Joseph Galbraith <[email-masked]>
Date: Tue, 09 Sep 2003 08:52:59 -0600

>    than lots of little bits and pieces (at the SFTP level).  Neither violate

Well, perhaps I misunderstood what you were proposing then.

Ignoring the channel window during data send is certainly
a protocol violation.

However, if the remote side has given the client a huge
window (something the client has no control over), the client
is certainly free to use it.  And more: it should use it.

The limit on the size of a single SFTP read exists for good
reason (as discussed previously.)  I don't think we can
remove it.

There are two different solutions, however, if you want to
achieve the same result:

1. Issue multiple reads, for some reasonable amount of data.
    Feel free to issue all the reads needed to fetch the entire
    file at once, if you want to.

    This requires no change in the protocol, and can be done
    with any previously documented version of the protocol.

2. Add support for multi-response reads to the protocol.  I.e.,
    allow the server to respond to your gaint read request with
    multiple smaller sftp data packets.  That way the server
    doesn't have to commit to sending you a 40 gig sftp data
    packet only to have the last 39gig dissappear before it can
    send it, and be forced to send you 39 gig of zeros before
    it can tell you that an error occured.

>    fixing it

Well, unless I'm mistaken, Markus said that OpenSSH had fixed
it and was now seeing throughput roughly equivilant to SCP.

The next version of VanDyke's software will also contain
optimizations of this nature.

- Joseph

---

From: Bill Sommerfeld <[email-masked]>
Date: Tue, 09 Sep 2003 17:19:45 -0400

>    than lots of little bits and pieces (at the SFTP level).  Neither violate

Earlier, you said "ignoring the window".  this proposal is
significantly different.

					- Bill

---

From: Damien Miller <[email-masked]>
Date: Wed, 10 Sep 2003 09:16:10 +1000

>    that this problem has been around for what, five years now without anyone

What implementation? OpenSSH has pipelined transfers for over a year and 
a half. The code is there for anyone to use...

-d

---

From: Damien Miller <[email-masked]>
Date: Wed, 10 Sep 2003 09:20:38 +1000

>    than lots of little bits and pieces (at the SFTP level).  Neither violate

What do you do if the file is larger than the defined sftp packet size?

-d

---

From: Markus Friedl <[email-masked]>
Date: Wed, 10 Sep 2003 23:55:36 +0200

On Tue, Sep 09, 2003 at 11:12:09PM +1200, Peter Gutmann wrote:
>    fixing it,

it's not a "theoretical fix".  both openssh and ssh.com fixed this
very early.  it was not fixed before, because in the first place
cared less about performance than about interoperability.