Subject: avoiding reinventing the wheel on file transfer.

From: Bill Sommerfeld <[email-masked]>
Date: Tue, 27 Mar 2001 19:27:10 -0500

So, I have this concern that we're going to spend a whole lot of time
on the file transfer draft, and in the process reinvent a couple
wheels which have already been built.

Here's a half-formed suggestion for the working group...

file transfer involves a bunch of really ugly issues (newline
conventions, binary/ascii/... modes, i18n of filenames, etc., etc.).
Many of these problems have already been tackled by FTP.

A radically different and simpler approach (from a standards-writing
viewpoint, at least) may well be to define a way to tunnel "normal"
ftp through a collection of ssh channels, so we can avoid refighting
all the battles which were fought over FTP... i.e., you start off with
one channel for the FTP control channel, and then open up new ssh
channels where "normal" FTP would open a new TCP connection....

						- Bill

---

From: David Terrell <[email-masked]>
Date: Tue, 27 Mar 2001 17:00:15 -0800

> channels where "normal" FTP would open a new TCP connection....

As a point of reference, I believe SecureFX from vandyke does
exactly this by using creative PORT commands and dynamic remote 
port forwarding against an existing FTP daemon on the sshd host.

-- 
David Terrell            | "When we said that you needed to cut the
[email-masked]             | wires for ultimate security, we didn't
Nebcorp Prime Minister   | mean that you should go wireless instead."
http://wwn.nebcorp.com/  |   - Casper Dik

---

From: [email-masked] (Albert Lunde)
Date: Tue, 27 Mar 2001 19:39:06 CST

> channels where "normal" FTP would open a new TCP connection....

Speaking as one who has had a very minor role in the wu-ftpd
testing process the last year or two, there are a lot of
mis-features of FTP and/or FTP implementations that _deserve_
to be reinvented. What you describe would fix the firewall-unfriendly
pattern of TCP connections, but not other historical warts on the 
protocol.

If you could "profile" the FTP protocol, and throw out, say, half
the legacy features, and bless some recently drafted additions, you
might have something more plausible.

--
    Albert Lunde          [email-masked] (new address)
                          [email-masked] (old address)

---

From: Markus Friedl <[email-masked]>
Date: Wed, 28 Mar 2001 09:37:49 +0200

> channels where "normal" FTP would open a new TCP connection....

there is no need to 'define' this approach, it has already been
implemented with success -- VanDyke's SecureFX shows this.

however, i think that a SFTP subsystem is still useful since it
implements a much simpler protocol than FTP.  if you want to reuse
existing FTP clients, than you can implement a ftp-sftp bridge like the
one from mindbright.se.

---

From: Simon Tatham <[email-masked]>
Date: Wed, 28 Mar 2001 09:13:29 +0100

> Many of these problems have already been tackled by FTP.

I'd applaud any efforts to use existing solutions to these ugly
issues, but I for one do like SFTP as a concept. It has useful
things that FTP doesn't and probably couldn't without serious
effort:

 - a well-defined, portable and securable chmod (SITE EXEC doesn't
   cut it, not least because it's hard to vet carefully at the
   server end and so most FTP servers simply don't bother to support
   it AFAICT).

 - a well-defined machine readable format for directory listings, so
   you can write automated clients that do complex manipulation of
   the remote fs, because they can look at a dir listing and
   reliably get things like file sizes back out. Also you can
   sensibly write a GUI client which displays sizes in a separate
   Windows-like column, without having to parse the size data out of
   an ASCII listing which you're _assuming_ has the format you
   expect.

 - no reliance on command line details at the server end. In FTP, if
   you want a long listing, or a short listing, or a listing in
   columns, or a listing sorted by this or that, you have to know
   what command line options the server's `ls' binary happens to
   take - and in general, the server doesn't even tell you this!
   Whereas in SFTP, you can process command line options (or
   whatever local equivalent you like) locally, which means you can
   _document_ them carefully and fully for your users. Then you
   retrieve raw dir listing data from the server, and you format and
   sort it exactly as you like.

 - SFTP has the extensibility philosophy of SSH2 in general; I can
   define an extension, publish documentation of it, and rely on
   nobody else's extension clashing with it or interoperating badly
   with it.

I know I have no official clout with the WG, but if I did, I would
strongly oppose any move to abandon SFTP in favour of a thinly
modified FTP. By all means look at FTP to see how it solves problems
SFTP still needs to solve, but _please_ don't throw the baby out
with the bathwater. SFTP is a massive improvement on FTP in many
areas, and really we want a protocol with the best features of
_both_.

Cheers,
Simon
-- 
Simon Tatham         "Imagine what the world would be like if
<[email-masked]>    there were no hypothetical situations..."

---

From: [email-masked] (Niels M�ller)
Date: 28 Mar 2001 11:59:46 +0200

> ... i.e., you start off with

It would probably be good somebody could write up the details of this,
so we could compare it to sftp.

I would also like to see a summary of how ftp have dealt with the
problems you mention.

For binary/ascii, my understanding of the ftp "solution" is that the
client has to know a priory whether or not a file is text or binary
and invoke the appropriate mode (some servers seem to sometimes
default to binary, but a client can't *rely* on the server choosing
the right mode). It's not obvious to me how that is significantly
easier for the client to know than also knowing which new-line
conventions are used for a text file and convert it locally.

As for i18n of filenames, I don't know what the ftp solution is, but I
do know that there are some hard problems. Before saying that ftp is
better than sftp in this respect, I'd like to see some evidence that
ftp has *solved* the problems, not just "tackled" them. (From the
discussion on this list, it seems that the hardest problem is how to
do both of (i) proper normalization, and (ii) support for systems
where file names really are octet strings with no associated character
set. My current preference is to let the base protocol treat file
names as plain octet strings, and use some extension to negotiate
character sets when that is desired).

If we're serious about starting over with file transfer stuff, we
should probably start with some requirements document. I mean, we want
to solve at least the problems solved by the current sftp and scp, and
probably some more, and we really need to know what problems we want
to solve before we can evaluate ftp-based (or, for that matter,
http-based) variants.

<rant>
  FTP was the kitchen-sink protocol of the 70:s and perhaps 80:s (look
  at RFC 737 and RFC 743...), while HTTP is today's kitchen-sink.
  Directory listings with FTP sucks (as has already been pointed out),
  and HTTP is even worse. Is that really what we want?
</rant>

Regards,
/Niels

---

From: Damien Miller <[email-masked]>
Date: Wed, 28 Mar 2001 20:35:31 +1000 (EST)

> channels where "normal" FTP would open a new TCP connection....

I can't say this appeals to me. Special-casing a clean protocol so we
can use it to wrap an ugly one for a few dubious features doesn't seem
like a win to me.

Others have observed that sftp is more NFS-like than FTP-like anyway,
perhaps it would server as a better source of inspiration wrt issues
such as i18n (I haven't been convinced that binary/ascii is even an
issue).

-d

-- 
| Damien Miller <[email-masked]> \ ``E-mail attachments are the poor man's
| http://www.mindrot.org          /   distributed filesystem'' - Dan Geer

---

From: "David B. Coningham" <[email-masked]>
Date: Thu, 29 Mar 2001 11:28:51 -0700

> port forwarding against an existing FTP daemon on the sshd host.

Thank you for pointing this out.  SecureFX does indeed perform FTP
transfers over a single SSH connection (establishing one channel for
the control connection and one channel each time a data connection is
required).

One minor point of correction to the above statement is that SecureFX
uses PASV to establish the data connection rather than PORT.  PORT
currently cannot be used very effectively because of a limitation with
the remote port forwarding mechanism: it does not provide a way to
determine the actual port number to which the listening socket is bound
(if 0 is requested).

For this reason it would be desirable to see the remote port forwarding
response changed to include the port number.

---
David Coningham
[email-masked]

---

From: Martin Forssen <[email-masked]>
Date: Fri, 30 Mar 2001 08:55:39 +0200 (MEST)

> the control connection and one channel each time a data connection is

Just to give another data-point I can mention that we at AppGate have
implemented an ftp-proxy which uses one single channel for both command
and data. This allows us to handle both active and passive ftp. However
I am not saying we should adapt this solution for the filexfer draft. I
am also of the opinion that we need to find something better than ftp.

	/MaF

---

From: Mats Andersson <[email-masked]>
Date: Fri, 30 Mar 2001 10:34:51 +0200 (MEST)

Hi,

On Thu, 29 Mar 2001, David B. Coningham wrote:
> determine the actual port number to which the listening socket is
> bound (if 0 is requested).

You can of course support PORT too (which we do) since you can just loop
and take a random high port to listen to and see if it works (almost
always works on first attempt also so mostly not very inefficient :-).

> For this reason it would be desirable to see the remote port
> forwarding response changed to include the port number.

However, as you say (and many have said earlier) it would be a nice (and
trivial) addition that listening on 0 remotely replies with an extra field
with the actual port number if succesful.

Cheers,

/Mats