Subject: Re: Why SFTP performance sucks, and how to fix it

From: [email-masked] (Peter Gutmann)
Date: Tue, 8 Jul 2003 17:14:54 +1200

Martin Pool <[email-masked]> writes:

>Of course I agree that 4k or even 32k is too small for most modern networks.
>But I did want to point out that going too far in the opposite direction can
>be a problem too.

Fair enough.  I guess the XON/XOFF thing (to replace the Ack) would do this,
or perhaps a channel close depending on the seriousness of the error.  I think
it'd need some experimentation/going through usage cases to see where/how it's
useful:

- Typical usage: SSHv1 and SSL/TLS don't seem to need any flow control, which
  would imply that in most cases you don't have to worry about it.

- Out of disk space: Probably a channel close, since it's a fatal error and
  you want the sender to stop permanently ("Please wait while the sysadmin
  hot-plugs some more RAID storage" probably won't work :-).

- Temporary resource problem (can't think of a good example at the moment but
  I'm sure there's something): Send XOFF, perhaps with an optional timeout
  indication, if the sender doesn't get an XON in that time they can consider
  it a long-term/fatal error as above.

Another thing to keep in mind here is that if you signal a read/write of the
entire file at once, the receiver knows how much disk space it needs and can
lock the space before starting the receive - it allows far better data
transfer management then sending an arbitrary number of tiny little bits that
the receiver has to take care of on the fly.  I'd really prefer to know before
I start that a transfer is going to fail, rather than write 2GB and then get
the out-of-disk error.

If you follow this stragegy you never need to signal out-of-disk during a
transfer, only at the start.  Even better would be to require (or at least
recommend) that clients include the file size in the FXP_OPEN *before* any
data transfer is about to take place, so the receiver can respond to the open
request with not-enough-disk-space error.  This is perfectly feasible in most
cases where SFTP is used, since you're sending files of a known size (I
haven't played with this too much, but it doesn't appear that implementations
indicate the size at open much, which would require a code update).  You can
really optimise the transfer management by following a few simple rules in
which the sender transmits information required to ease data processing in
advance.

If it's of any use to people, I could write a small informational appendix or
whatever for the draft indicating how to use the protocol in a manner that
makes data transfer management easier.

Peter.

---

From: Martin Pool <[email-masked]>
Date: Tue, 8 Jul 2003 15:38:48 +1000

On  8 Jul 2003, Peter Gutmann <[email-masked]> wrote:
> or perhaps a channel close depending on the seriousness of the
> error.

Channel close is a terrible error handling method.  To start with, it
gives no indication of what in particular went wrong.

>  I think it'd need some experimentation/going through usage cases to
> see where/how it's useful:
> 

>   hot-plugs some more RAID storage" probably won't work :-).

Well, out of disk space was only an example of the kind of error that
can occur at any point.  There are others.  A more common one for
rsync is some kind of permission problem.  SECSH, unlike rsync, can
probably trap that in the OPEN operation before starting to send bulk
data, but the general problem remains.

User quotas are another quite realistic possibility where an
interactive user might very well want to stay connected and delete
some files. 

> lock the space before starting the receive

I don't know of a means for a Unix or Windows server to "lock" disk
space in that way.  I suppose the server might zero-fill the blocks
between getting the start of the request and reading the bulk of it,
but that seems highly contrived.

> I'd really prefer to know before I start that a transfer is going to
> fail, rather than write 2GB and then get the out-of-disk error.

The general problem I'm talking about here is that in a protocol like
SECSH, the size of the request blocks is the amount of data "at risk"
at any point: if something goes wrong, as it unavoidably may, then you
might have wasted your time and money transmitting it.

Here's another example, taken from experience with Samba: if you're
going to send chunks of 2GB at a time they presumably have to be
"streamed" from disk and not built in a temporary buffer.  (Indeed,
people may well want to transfer files larger than their VM - 6GB
files on a 32bit machine.)  There are unavoidable errors where you can
in fact send less data than you originally thought, if e.g. somebody
else truncates the file while you're reading it.  That means that the
header of the request ("write 6G") is impossible to fulfil, so you
need to either drop the connection or write nulls, both of which are
deeply undesirable.

I suspect that sending a block-at-a-time can give good performance if
you choose a good block size and pipeline transmissions, and if
problems with the underlying transport are addressed.  Perhaps IOs
should be roughly proportional to the amount of data in flight?  I
think the request/response model in SECSH is a great thing for
simplicity and robustness and it shouldn't be lightly cast aside.

-- 
Martin

---

From: "Howard Chu" <[email-masked]>
Date: Tue, 8 Jul 2003 01:47:14 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf
Of Martin Pool

> but that seems highly contrived.

The FTP protocol has an ALLO command for allocating space in advance. On some
FTP servers I've seen, they do exactly this - zero-fill that much space. It
is a pretty foolproof way of accomplishing the task, even if it seems
contrived.

> > fail, rather than write 2GB and then get the out-of-disk error.

Definitely.

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 8 Jul 2003 08:00:19 -0700

> gives no indication of what in particular went wrong.

Indeed.  SFTP is not like FTP where there's separate channels for
control data and data - closing the channel means killing the session
with no useful error indication.

> but that seems highly contrived.

And inefficient.

> simplicity and robustness and it shouldn't be lightly cast aside.

The SFTP protocol is, in some ways, more like a remote filesystem
protocol than like a file transfer protocol (my observation - I'm not
saying it should be).  Thus a client can interleave read/write requests
relating to multiple files over the same channel.

But note that the client is not required to wait for SFTP "Acks" before
proceeding with other requests.  SFTP clients are allowed[1] to issue
reads/writes for entire files before handling a single reply to any of
them.  Doing so may not be wise for very large file transfers, as Martin
points out.

The fact is that SFTP does not have an Ack "handbrake" - it has Acks,
but no Ack handbrake.  Implementations may, but not the specification.

[1] Or, rather, such client behaviour is not forbidden by the text of
    the I-D.  Pipelining block-at-a-time requests is not disallowed;
    perhaps it should be encouraged.

Cheers,

Nico
--

---

From: "'Martin Pool'" <[email-masked]>
Date: Wed, 9 Jul 2003 15:12:49 +1000

> is a pretty foolproof way of accomplishing the task, even if it seems

There are realistic cases where it might fail: compressed filesystems
might be able to store 2G of zeros, but not 2G of real data; versioned
filesystems may run out of space when rewriting an existing file; or
the user might be in a grace period for disk quotas that expires
during the transfer.  It would give some assurance that there is
enough space, but it's not foolproof.

But my main point is that it is *unavoidable* that disk operations can
fail during writing.  Out of space is only one possibility; hardware
errors are another.  

A protocol that relies on writing the whole file in one operation can
only handle this by either wasting time and bandwidth on an operation
that is known to have failed, or by dropping the connection without
(inline) explanation.  

Another advantage of course is that the client can reconnect and
resume a transfer.

> > > fail, rather than write 2GB and then get the out-of-disk error.

I suppose you could usefully add a preallocation operation as an
extension command, but it won't be a guarantee.

-- 
Martin

---

From: Joseph Galbraith <[email-masked]>
Date: Wed, 09 Jul 2003 11:26:06 -0600

>extension command, but it won't be a guarantee.
The current draft specifies that if file size in the attrs
field is present during file creation, it should be considered
a hint as to the file's eventual size. (Section 5.3)

I'm not sure if this is good enough or not though.

- Joseph