Subject: RE: UTF8 in SFTP (was: solving the SFTP text mode issue)

From: Richard Whalen <[email-masked]>
Date: Tue, 14 May 2002 11:02:23 -0400

I believe that a text transfer mode is sufficient to meet our customers'
needs.  Users view SCP & SFTP as replacements for FTP and hence expect to be
able to transfer text files as well as binary files.  Though FTP defines a
record transfer mode it is seldom used, so I don't think that the
development of the SSH File Transfer Protocol should be encumbered with the
work that would need to be done if a record transfer mechanism were to be
added at this time.

The text transfer mechanism in the SSH File Transfer Protocol should define
a single method of encoding the line end to remove any ambiguity.  Systems
that encode line breaks differently from the specified method would be
responsible for scanning the data and performing the necessary substitution.

One additional thing to note: When a file is transferred in Text mode, the
size information reported for a file must be considered to be an estimate as
computing the exact size may consume too many resources or use too much time
to process the command in a timely manner.  The only way to determine that
all of the text from the file has been retrieved is through the receipt of
end of file status when there is a request for data.

----------------------
Richard Whalen
Process Software

> -----Original Message-----
> From: Joseph Galbraith [mailto:[email-masked]]
> Sent: Monday, May 13, 2002 10:53 AM
> To: Wei Dai; [email-masked]
> Cc: [email-masked]
> Subject: Re: UTF8 in SFTP (was: solving the SFTP text mode issue)
> 
> 
> > Here's another simple suggestion for your consideration. 
> Define a new
> > pflags flag for SSH_FXP_OPEN:
> > 
> >    #define SSH_FXF_TEXT            0x00000040
> > 
> > It would have the following meaning:
> > 
> >    SSH_FXF_TEXT
> >       File SHOULD be encoded as UTF-8 using either CR, LF, 
> or CR/LF to 
> >       indicate line end, or converted to this encoding 
> before transfer. 
> >       File access MUST be sequential.
> 
> First, I don't think SHOULD is strong enough here.
> For this to be useful, it would need to be MUST.
> Probably this text should be included:
> 
>         If the server can not comply, it must respond
>         with status SSH_FX_OP_UNSUPPORTED.
> 
> I don't think I can determine the current encoding
> of an arbitrary text file in order to convert
> it's contents to UTF-8.  (Some files will be
> encoded in unicode, and of course there is no
> problem with those.)
> 
> I believe the unix folks have the same problem
> with file names -- which is why I've been having
> such a hard time getting them to swallow UFT-8
> for file names.
> 
> I actually think that for both cases, it may be
> necessary to make UTF-8 optional, determined
> by the server.
> 
> I.e., the client MUST be able to read filenames and content
> encoded in UTF-8.  The server MAY send filenames and/or
> content in UTF-8.  If content/filenames are not encoded in UTF-8, 
> there encoding is unspecified, and user intervention may
> be required to determine how to display / save the file.
> 
> If the SSH_FXF_TEXT flag is set during open, and the server
> will send file content in UTF-8, it should respond with
> status code SSH_FX_OK_UTF8 (status code 9) instead of
> SSH_FX_OK.
> 
> If the server will encode filenames in UTF-8, it should
> include the following extension data in it's VERSION
> packet (if and only if the clients INIT packet specified
> a version >= 3.)
> 
>   "filename-utf8"        # extension name
>   ""                     # no extension data
> 
> - Joseph
> 
>

---

From: Wei Dai <[email-masked]>
Date: Tue, 14 May 2002 12:54:15 -0700

> responsible for scanning the data and performing the necessary substitution.

Ok, I now realize there is no point in allowing either CR, LF, or CR LF,
so let's just specify LF.

> end of file status when there is a request for data.

That seems reasonable.

> > status code SSH_FX_OK_UTF8 (status code 9) instead of

Instead of the above, how about a seperate flag SSH_FXF_UTF8, and status
code SSH_FX_UTF8_NOT_SUPPORTED. It seems likely that a client sometimes
will not be able to determine how to convert a file it wants to upload
into UTF-8 either. SSH_FXF_TEXT would then mean only using LF for line
end.

> >   "filename-utf8"        # extension name

Why shouldn't this extension apply to both sides symmetrically? We can say 
that if both sides support this extension, all filenames will be in UTF-8, 
otherwise their native encodings are used.

---

From: "Howard Chu" <[email-masked]>
Date: Tue, 14 May 2002 13:23:48 -0700

> From: [email-masked] [mailto:[email-masked]]On
> Behalf Of Wei Dai

> so let's just specify LF.

One complication to watch out for - some systems actually treat CR and LF
according to their original definition. You will find text files that use
CRs by themselves to achieve overstrike, and LFs by themselves to insert
whitespace, etc., in addition to CR/LF "line terminator" sequences. There's
a good reason why all the existing RFCs that deal with text transmission
specify CR/LF as the line break sequence; if you use just a single CR or LF
as suggested here, it is impossible to canonicalize a file without losing
formatting (of individual CR or LFs).

I'm still at a loss as to why SSH FTP wasn't simply FTP wrapped in SSH. FTP
has already been designed, what is with all this re-inventing the wheel
business. How are you going to accomodate record-oriented file transfers,
records with line numbers, and other structured file transfers? As much as I
like Unix and its
simple bytestream file model, there's still a place for record-oriented
files
in this world...

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support

---

From: Bill Sommerfeld <[email-masked]>
Date: Tue, 14 May 2002 16:25:44 -0400

> I'm still at a loss as to why SSH FTP wasn't simply FTP wrapped in
> SSH. 

for the record, the working group chair (me) is *still* at a loss to
understand why there seems to be so much of a desire to reinvent the
wheel here.

I'd personally welcome a simple document explaining how to run the ftp
protocol over an ssh connection.

					- Bill

---

From: Wei Dai <[email-masked]>
Date: Tue, 14 May 2002 13:50:22 -0700

> understand why there seems to be so much of a desire to reinvent the

I can think of a few reasons:

1. SFTP is very easy to understand and implement, once you've implemented
SSH. 
2. FTP's seperate control and data channels are an annoyance in the
context of SSH. 
3. Historically FTP implementations have been vulnerable
to various attacks. SFTP implementations have not been.
4. SFTP server typically runs under the user's account instead of root.
5. SFTP can be used as a basic network file system, again lightweight and 
secure compared to existing solutions.

Unless people foresee more problems beyond the current one (i.e. text 
transfer mode) with SFTP, why not finish it up?

---

From: "Dan O'Reilly" <[email-masked]>
Date: Tue, 14 May 2002 15:17:45 -0600

>transfer mode) with SFTP, why not finish it up?

I agree.  The issue here is, as I see it, one that regardless of the original
intent of what SFTP was supposed to be, the fact of the matter is, customers
in the real world are using it in a uniform way: as a secure replacement for
ftp.  And that's the bottom line.  Once it's in production, it's DARNED hard
to make somebody change.  No, it doesn't have to do everything ftp does; but
on the other hand, given what real-world users are using it for, it's
inevitable that it gets finished with the stuff we've been discussing.

I think Wei put it very well.

------
+-------------------------------+---------------------------------------+
| Dan O'Reilly                  |                                       |
| Principal Engineer            |  "Why should I care about posterity?  |
| Process Software              |   What's posterity ever done for me?" |
| http://www.process.com        |                    -- Groucho Marx    |
+-------------------------------+---------------------------------------+

---

From: "Howard Chu" <[email-masked]>
Date: Sun, 19 May 2002 14:26:13 -0700

> 1. SFTP is very easy to understand and implement, once you've implemented

The implication here is that FTP is not. This is a pretty odd statement; FTP
has been in use for so long, the volume of expertise with it surely exceeds
the volume of expertise in SSH/SFTP by several orders of magnitude. It's a
long-ago solved problem. There's nothing new to understand. Even if SFTP is
easy to understand, it's just creating work where none was needed. It's also
obvious, by the length of these discussions, that it is not well thought out
enough to address many desired usage scenarios.

> 2. FTP's seperate control and data channels are an annoyance in the

SSH itself is a multi-channel protocol. I don't see the problem in mapping
FTP PORT commands to SSH channels.

> to various attacks. SFTP implementations have not been.

This is like saying "historically libcurses has been insecure compared to
libdes." And just because an implementation is broken is no reason to
discard
the protocol definition.

> 4. SFTP server typically runs under the user's account instead of root.

This is an implementation detail, not a protocol feature.

> secure compared to existing solutions.

FTP has been used in this manner for many years. Again, it is not unique to
the protocol definition.
> transfer mode) with SFTP, why not finish it up?

If all you want is to come to consensus on how to finalize one last detail
of the current implementation, fine, do that as an informational RFC. I
don't believe this spec merits Standard status.

  -- Howard Chu

---

From: Simon Tatham <[email-masked]>
Date: Mon, 20 May 2002 09:53:04 +0100

Howard Chu <[email-masked]> wrote:
> SSH itself is a multi-channel protocol. I don't see the problem in
> mapping FTP PORT commands to SSH channels.

Mapping them how? By changing their syntax so that they show SSH
channel numbers rather than IP addresses and ports, or by having the
client somehow translate them _into_ IP addresses and ports, or
what?

A technique that is already practised is to forward an FTP control
channel over SSH, and to edit the PORT commands as they go down the
wire so that the FTP client ends up conducting a multi-connection
FTP dialogue with the SSH client, and the SSH server conducts a
different one with the FTP server. This works, but it's hardly what
I'd call elegant. Particularly since, after you've gone to the
effort of setting up SSH identity keys and arranging one-touch
authentication, the last thing you want is to have to type your
password into an FTP client every time you want to transfer a file!

An alternative would be to have a specialist FTP client integrated
with an SSH server; so that it would send a PASV command and then
ask the SSH server to open a fresh channel to the IP and port number
specified by the FTP server. This would work (modulo the above
authentication inconvenience) but suddenly no existing FTP client is
competent to do this - and what was the point of keeping FTP
unmodified if not to be able to use the large array of existing
clients?

I really don't see why people are still trying to tell us FTP is
perfectly sufficient. SFTP allows plausible automated clients, is
able to run over an already-authenticated SSH connection, and is
readily extensible by a well-defined means. Why, merely because FTP
has solved one or two problems that SFTP as yet hasn't, are people
exhorting us to go back to FTP which fails to solve all those
_other_ problems? I just don't see it.
-- 
Simon Tatham         "Imagine what the world would be like if
<[email-masked]>    there were no hypothetical situations..."