Subject: Re: A future for the SSH File Transfer Protocol?

From: Darren Moffat <[email-masked]>
Date: Fri, 9 Nov 2001 15:47:15 -0800 (PST)

> - Instead of separate getuserbyname and getuserbyuid calls, we have
>   a single "getuser" call, whose argument is itself a USER data

I hate to say it but this whole SFTP + USERDB is starting to look
suspiciously like NFSv2 + NIS ;-)

I'd like to suggest that before the group goes any further with SFTP we
stop and do a requirements spec.  

What problem are we really trying to solve for the end user ?

In particular is the goal a file transfer protocol along the lines of
FTP (as the name would suggest) or is it a file sharing protocol more
in line with NFS or CIFS.  Some of the suggestions make me thing it is
drifting somewhere between the two at the moment (which isn't a bad
thing as long as we know what problem we are trying to solve).

Why doesn't running the FTP protocol over SSH transport help,
(I know about the issues with FTP as a protocol but I think it wouldn't
hurt for the wg to document them the a requirements spec).

Why doesn't NFSv4 solve their problems either (I think it would in
theory be possible to have the SSH protocol stack as a GSS mechanism
that NFSv4 could run over).

One final thing, if SFTP is truly independent of the SSH transport as
the last revision of the draft stated is SECSH-WG really the correct
group to be designing it ?  What is it about SFTP that makes it
suitable for the security area ? Would a group in the applications area
be more appropriate ?

--
Darren J Moffat

---

From: RJ Atkinson <[email-masked]>
Date: Fri, 09 Nov 2001 20:13:35 -0500

>What problem are we really trying to solve for the end user ?

Yes please.  Lets define the problem and the solution requirements
FIRST.  No particular objection to SFTP, just want to nail down
what we're undertaking before the land gets swampy...

Ran
[email-masked]

---

From: Simon Tatham <[email-masked]>
Date: Sat, 10 Nov 2001 09:21:08 +0000

Darren Moffat  <[email-masked]> wrote:
> in line with NFS or CIFS.

As I see SFTP, its purpose is to allow a useful level of file system
access, in a platform-independent manner, _making use of SSH
authentication_.

In NFS:

 - you have a single centralised NFS arrangement between the
   administrators of a pair of machines, and each user gets to make
   use of that same arrangement.

 - the admins negotiate a uid mapping, finding something clever to
   do about uids that aren't matched on the other machine.
   Alternatively, they arrange to have matching uids everywhere.

 - each side trusts the other not to abuse that uid mapping; if the
   client were to make a mistake in enforcement, one of its users
   might be able to write to a file they aren't supposed to be
   allowed to. So the server is extending its trust perimeter to
   include the client's kernel.

Whereas in SFTP:

 - each user who wants filesystem access authenticates separately,
   so there isn't a need for the admins to cooperate.

 - uid mapping can be done, if the user feels it would be helpful,
   but it is informational only, because the real guarantee of the
   access control being respected on the server is that the
   sftp-server process doesn't _have_ the privileges to access files
   it shouldn't.

 - So if a uid mapping is wrong, there is no security breach. The
   client user gets confusing information, but it's only the
   _information_ that is faulty. Access control is still intact.

 - Hence, the server does not need to trust the client to enforce
   the access controls: it enforces them itself.

All of this is the case _irrespective_ of whether it's being used as
a networked file system or an FTP-like file transfer client; the
interface presented to the client user is irrelevant. (Although I
suppose uid mapping is a bit irrelevant in the latter case :-)

Interestingly, of course, one major area of user interfaces for SFTP
clients is the Windows Explorer-like drag-and-drop interface - which
precisely _does_ blur the user-level distinction between a dedicated
file transfer client and a networked file system!

> Why doesn't running the FTP protocol over SSH transport help,

FTP lacks several _very_ nice features of SFTP:

 - the ability to retrieve only part of a file (REST is helpful in
   some situations but doesn't really cut it)

 - a properly specified, reliably available chmod and chgrp (and I
   suppose chown, for people mad enough to do it as root :-)

 - directory listings returned in a machine-readable form

 - of course, the well-known chaos with multiple network connections

In general, it would be very difficult to write a specialised client
that would use FTP to perform a specific and complex file
manipulation on a remote server. But SFTP makes this easy (not least
because it has access to SSH's hands-off authentication), and
therefore it's a qualitative improvement over FTP. (Shame about
ASCII transfer, of course, but it's an extensible protocol so that
can be fixed...)

In summary: I see SFTP as being an excellent middle way between NFS
and FTP. It's not sysadmin-heavy like NFS, and yet it's way more
flexible and more automatable than FTP. The basic structure of the
protocol is _exactly_ what I would want to see in a file transfer
_or_ filesystem access protocol, and it's highly extensible so its
current lack of polish can be seamlessly fixed.

Cheers,
Simon
-- 
Simon Tatham         "A cynic is a person who smells flowers and
<[email-masked]>    immediately looks around for a coffin."

---

From: Markus Friedl <[email-masked]>
Date: Sat, 10 Nov 2001 19:42:17 +0100

> What problem are we really trying to solve for the end user ?

a very simple protocol for copying files.

as simple as RCP, but without the many drawbacks.

> thing as long as we know what problem we are trying to solve).

no, SFTP is not for sharing files. sftp should not become
as bloated.

> hurt for the wg to document them the a requirements spec).

because ftp is big, bloated and requires a 2nd connection
for transfering files. doing FTP over TCP-forwarding is
not the same as SFTP, checking the channel data stream
for "PORT" and opening another TCP-forwarding channel adds
unnecessary complexity to an SSH implementation. why
not just use the 8bit clean channel SSH provides?

just ask the SSH vendors about how much SFTP simplified
the 'data transfer over SSH' problem? SFTP is simple and
can be very easily implented and deployed.

i don't understand the problem with SFTP.

-m

---

From: RJ Atkinson <[email-masked]>
Date: Sat, 10 Nov 2001 13:45:41 -0500

>i don't understand the problem with SFTP.

Might not be a problem, but folks want to understand 
the problem SFTP is trying to solve.

For example, I'm trying to puzzle out how SFTP
is different from SCP and why I would care about
that difference (if I knew what it was)...

Since this is a standards question, I'm deliberately
trimming the distribution to the IETF SECSH list.

Ran

---

From: Thor Simon <[email-masked]>
Date: Sun, 11 Nov 2001 02:10:59 -0500

> trimming the distribution to the IETF SECSH list.

From my point of view, there are three differences:

1) SCP is a lightweight protocol which is constrained in purpose and
   function to be a replacement for traditional Berkeley Unix 'rcp';
   SFTP is a much heavier protocol which tries to provide the functionality
   of FTP (and perhaps then some) atop the SSHv2 transport layer.

2) SFTP is standards-track; currently, SCP is very much not so.

3) SFTP is actually documented.  SCP -- because it's just basically rcp
   on top of SSH -- is not; there isn't an rcp RFC, not even an 
   informational.

Personally, my taste runs much more to a simple file transfer protocol than
to a very rich and complex one, but to date the SSH WG has felt otherwise.
I've committed to do an informational RFC on the rcp protocol but keep
failing to find the time for the last 10% of work to get it out the door;
at least then those who cared to implement SCP would have some documentation
other than the Berkeley code.

Thor

---

From: Markus Friedl <[email-masked]>
Date: Sun, 11 Nov 2001 11:46:58 +0100

> that difference (if I knew what it was)...

the RCP/SCP protocol has many drawbacks so SFTP acts as a simple
replacement for SCP. recent problems include:

error handling in RCP is broken: you send the filesize and the
filecontent to the remote host, if there is an readerror during the
transfer, you cannot stop the transfer for this file, because the two
scp processes might get out of sync.

RCP uses \n in the protocol, so you cannot transfer a filename
containing newlines.

if the remote host is the source of the transfer, the remote host tells
the local scp process the filesnames that should be created (e.g.  more
or different files than requested). with SFTP the local process sends
opendir/readdir/read messages and decides what files to copy.

and so on. i don't think it's worth using RCP/SCP as a starting point.

-m

---

From: Markus Friedl <[email-masked]>
Date: Sun, 11 Nov 2001 11:53:12 +0100

>    of FTP (and perhaps then some) atop the SSHv2 transport layer.

i don't agree. SFTP fixes the problems in SCP, while FTP
is much more complex than SFTP. people are just confused
because of the 3 common letters FTP.

> to a very rich and complex one, but to date the SSH WG has felt otherwise.

that's wrong. SFTP is very simple, all it does is
	open/read/write/close/opendir/readdir

SCP might be simpler, but has too many problems.

you should ask vendors who actually tried to implement both.

> other than the Berkeley code.

it's not worth the trouble.

-m

---

From: Simon Tatham <[email-masked]>
Date: Sun, 11 Nov 2001 10:59:54 +0000

>    on top of SSH -- is not; there isn't an rcp RFC, not even an 

Also

 (4) There is a _standardised_ means of invoking SFTP on a server,
     which doesn't depend on the client knowing or guessing details
     of the server setup such as its OS or its PATH or the location
     of its scp binary, or indeed its command line quoting rules
     (try SCPing a file with weird characters in the name!).

 (5) SFTP doesn't require the client to trust the server's wildcard
     expansion. (Suppose you send a request for "*.c". You would
     like the server not to be able to send back filenames that
     don't actually match this pattern, so it can't maliciously
     overwrite ".bashrc" in the same directory. Sadly in SCP the
     wildcard matching is done on the server, by the server's rules,
     and the client doesn't necessarily know what those rules are
     (even if the end user does and has correctly constructed a
     wildcard within them). For example, "[ab]*.c" would match
     extremely different things on a VMS and a Unix system, since in
     VMS [ab] is a directory specifier and in Unix it's a wildcard!
     
     SFTP solves this by not _having_ a means of wildcard matching -
     the client must request the whole directory, do the matching
     itself by its own rules, and then request specific files.
     Another solution would be to standardise a set of wildcard
     rules, so that the client could validate returned filenames
     against them.

SCP is a _really bad_ protocol.

> failing to find the time for the last 10% of work to get it out the door;

I hope it treats the above issues carefully! :-)

Cheers,
Simon
-- 
Simon Tatham         "I thought I'd put my foot so far into my mouth I
<[email-masked]>    wouldn't be able to sit down without standing up."

---

From: Anthony DeRobertis <[email-masked]>
Date: Sun, 11 Nov 2001 14:21:02 -0500

> as simple as RCP, but without the many drawbacks.

That's called scp, methinks.

---

From: Theo de Raadt <[email-masked]>
Date: Sun, 11 Nov 2001 12:35:29 -0700

> That's called scp, methinks.

Then you are massively uninformed.

scp protocol IS rcp protocol.  They are 100% the same, and have 100%
the same flaws.

Now hush.

---

From: Markus Friedl <[email-masked]>
Date: Sun, 11 Nov 2001 22:08:46 +0100

> That's called scp, methinks.

sorry, you are wrong. SCP==RCP. scp just runs rcp over ssh.

-m

---

From: Anthony DeRobertis <[email-masked]>
Date: Sun, 11 Nov 2001 16:53:43 -0500

> scp protocol IS rcp protocol.  They are 100% the same, and have 100%

IMO, the biggest flaws with rcp are the lack of strong 
authentication and the lack of encryption.

Running rcp over ssh provides both of those. I am aware of the 
other issues, like file name issues (I've run into them myself); 
and things like the remote side handling expansions can be 
checked on the local side. I don't consider that nearly as 
important as the first two. There are also some problems with 
passing metadata (I'm on a Mac at the moment), but those can 
also be taken care of.

Now, if sftp is to be more like FTP than rcp, I'd understand 
that. But I don't think 'rcp without flaws' is a very good spec 
of what you're doing.

---

From: Damien Miller <[email-masked]>
Date: Mon, 12 Nov 2001 12:07:25 +1100 (EST)

> authentication and the lack of encryption.

These have already been mentioned:

- Inability to use \n in file names
- Poor error handling
- Serverside globbing
- To be secure a client needs to second-guess the paths sent from the server.

I'll add these:

- Inability to do basic filesystem operations (e.g. rename)
- Inability to access parts of files, or continue aborted transfers

> important as the first two. 

You really want to have a protocol which forces clients to include code
to protect from a hostile server? That is asking for insecure 
implementations.

>There are also some problems with 

If we are going to change a 10+ y/o protocol, why not just write a new one
without the (many) inherent flaws. Even better, why not just tweak the one
that has _already been written and implemented_.

> that. But I don't think 'rcp without flaws' is a very good spec 

Why are we discussing requirements on somethings that is already
specified and widely deployed?

-d

-- 
| By convention there is color,       \\ Damien Miller <[email-masked]>
| By convention sweetness, By convention bitterness, \\ www.mindrot.org
| But in reality there are atoms and space - Democritus (c. 400 BCE)

---

From: Anthony DeRobertis <[email-masked]>
Date: Sun, 11 Nov 2001 21:09:17 -0500

> - Inability to access parts of files, or continue aborted transfers

Those are issues, especially the second. But that's not rcp anymore.
> to protect from a hostile server?

I think, by definition, cryptographic implementations must. And 
we've certainly seen a bunch of mistakes (ssh1, for example). I 
think that's a lot harder than making sure returned file names 
match the glob.

But, sure, if we didn't have to do that, that'd be great. I just 
wanted to point out that is not a fatal flaw.

> that has _already been written and implemented_.

No problem with either of those.

> specified and widely deployed?

Because we're talking about extending it? I'd think would be a 
good idea to specify what is being accomplished by the protocol 
before extending it.

---

From: David Butts <[email-masked]>
Date: Mon, 12 Nov 2001 06:42:12 -0500

Hello, all-

At the risk of making a fool of myself, I'd like to wade in here.

> because of the 3 common letters FTP.

The one advantage I know of that SCP has over SFTP is the ability
to restrict the source/target of a transfer with -f/-t.  That is,
IMHO, invaluable for any kind of batch copy process.  Without it,
it becomes much more difficult to prevent someone who has stolen
a private key from overwriting an authorized_keys file.  If there
is an equivalent means of restricting SFTP, then I'll certainly
use it, but without it, I can't give up SCP.

> SCP might be simpler, but has too many problems.

Again, at the risk of belaboring the point, I actually consider
SCP's lack of flexibility to be a feature, since I use it in
order to restrict access.  The two complement each other nicely,
at least from my perspective as a user, and I would miss SCP if
it went away.

David

---

From: Simon Tatham <[email-masked]>
Date: Mon, 12 Nov 2001 13:15:58 +0000

> use it, but without it, I can't give up SCP.

This sounds like an excellent idea and I can easily think of four or
five uses for such a feature myself, but surely it's nothing to do
with the _protocol_? From the protocol point of view, you can
attempt to do exactly the same set of things, and you either get
told Access Denied or Yeah, All Right.
-- 
Simon Tatham         "I'm going to pull his head off. Ear by ear."
<[email-masked]>                          - a games teacher

---

From: Darren J Moffat <[email-masked]>
Date: Mon, 12 Nov 2001 09:34:31 -0800

> specified and widely deployed?

Because there is discussion on the ietf-ssh list about extending and
chaning some of the functionality.

In particular how the SFTP protocol deals with the identity of the user.

Currently a uint32_t is used to send a "UID" which is fine for UNIX
since that is what most systems use as the uid, however it is not fine
for NT.  In parallel with this it has been proposed that the long
output goes away in favour of the client having to build it themselves
from the information it gets from the server.

To that end there has been proposals to change this area of
the protocol and augment it with a method for the client to translate
the opaque uid into a name suitable for display.  The rational being
that the mapping maybe different on the client than on the server.

My concern was that this is starting to get into the area and issues of
NFS and NIS: NFS does transfer NIS does name id mappings.

I fully agree that SCP is not suitable and neither is FTP over SSH but
before we go changing a lot of things I think it would be a good idea
to document where the end goal is so that we know when we are done and
we know if we are changing things because people really need that
functionality.

On the whol SFTP is a decent protocol but there are a few little warts
that need to be tidied up I just wanted to make sure that in tidying
these up SFTP didn't turn into a file sharing protocol and starting
becoming a poor mans NFS or CIFS.

I have no problem with documenting the protocol as it stands just now
just conern about where it seemed to be heading.

---

From: Barney Wolff <[email-masked]>
Date: Mon, 12 Nov 2001 13:24:02 -0500

> that the mapping maybe different on the client than on the server.
 
It strikes me that the whole notion of internal uid is the wrong way
to go and just makes life complex for both client and server.  I'd
propose instead that all conversation be in terms of external
user/group names.  If people really want to optimize directory listings,
something I'm not at all convinced is worthwhile, I'd propose doing
it the way DNS does it, with the first appearance of a name being
the full string and subsequent appearances being back-references.  What
is the point of making the client ever deal with the server's internal
uid's?
-- 
Barney Wolff

"Nonetheless, ease and peace had left this people still curiously tough.
They were, if it came to it, difficult to daunt or to kill; and they were,
perhaps, so unwearyingly fond of good things not least because they could,
when put to it, do without them, and could survive rough handling by grief,
foe, or weather in a way that astonished those who did not know them well
and looked no further than their bellies and their well-fed faces." J.R.R.T.

---

From: Markus Friedl <[email-masked]>
Date: Tue, 13 Nov 2001 09:52:35 +0100

> authentication and the lack of encryption.

this is a problem in your rcp client but not
in the rcp protocol.

-m

---

From: Markus Friedl <[email-masked]>
Date: Tue, 13 Nov 2001 09:52:45 +0100

On Sun, Nov 11, 2001 at 09:09:17PM -0500, Anthony DeRobertis wrote:
> I think, by definition, cryptographic implementations must.

but cryptography is not relevant for sftp/scp.

-m

---

From: Simon Tatham <[email-masked]>
Date: Tue, 13 Nov 2001 09:21:07 +0000

Barney Wolff  <[email-masked]> wrote:
> user/group names.

Yep, I'd sign up for that. Only trouble is, it would need an
incompatible protocol change. But if the politics of that can be
plausibly managed, it sounds like a good simplification.

(It wouldn't _completely_ remove the purpose of my extension,
because I'd still like to be able to retrieve the location of a
user's home directory. But it would let me condense the entire
extension down into one single SSH_FXP_HOMEDIR("fred") command,
which would be much nicer than the huge great proposal I've
currently got...)

> What is the point of making the client ever deal with the server's
> internal uid's?

As I understand NT, its security identifiers are quite complex, come
in several parts, and might not map exactly to a simple user name.
But then, if it came to that then the server could just produce a
textual representation of a SID and be done with it, so I think even
that isn't a good argument against your suggestion.

Cheers,
Simon
-- 
Simon Tatham         "The distinction between the enlightened and the
<[email-masked]>    terminally confused is only apparent to the latter."