Subject: solving the SFTP text mode issue

From: "denis bider" <[email-masked]>
Date: Fri, 10 May 2002 21:19:11 +0200

Judging from the number of times that text file support has been requested
for SFTP, we might as well do something about it. I suggest that, in the
next version of the filexfer draft, we document a recommended way for client
software to do text file conversion. (On its own, of course, without any
ugly hacks into the protocol.)

What if we provide an SFTP command that the client could execute to learn
the line separator on the server machine?

We could then document a text file conversion algorithm such as the
following:
1. The client queries the server about the server's line separator.
2. If the server's line separator is the same as the client's, the file is
transferred in binary mode.
3. Otherwise, the client determines whether the file is textual using, for
instance, the Kermit 8 algorithms mentioned by Jeffrey earlier on this list.
If the file is textual, the client converts it.

---

From: "denis bider" <[email-masked]>
Date: Fri, 10 May 2002 21:43:06 +0200

> What if we provide an SFTP command that the client could
> execute to learn the line separator on the server machine?

I know that this doesn't take care of record-based text file formats, but it
does take care of Unix, Windows, and Mac, which is much better than nothing,
and is probably what most people want, anyway.

As for record-based text file formats... The basic assumption of SFTP is
that a file is a single binary block of data that may be manipulated however
one wants. I simply don't see how this can be reconciled with record-based
text files. SFTP simply wasn't designed with that in mind, and I for one
wouldn't like to see the protocol retrofitted with alien concepts.

I think that the extension that I suggested would be easy to define, easy to
understand, easy to implement, and would satisfy the majority of users. In
contrast, a universal solution that would also support record-based file
formats would be much more complex and would force this complexity down the
throats of the majority of people who have no need for it.

---

From: Wei Dai <[email-masked]>
Date: Fri, 10 May 2002 13:01:56 -0700

Here's another simple suggestion for your consideration. Define a new
pflags flag for SSH_FXP_OPEN:

   	#define SSH_FXF_TEXT            0x00000040

It would have the following meaning:

   SSH_FXF_TEXT
      File SHOULD be encoded as UTF-8 using either CR, LF, or CR/LF to 
      indicate line end, or converted to this encoding before transfer. 
      File access MUST be sequential.

---

From: "Joseph Galbraith" <[email-masked]>
Date: Fri, 10 May 2002 13:59:56 -0600

> If the file is textual, the client converts it.

This is what our server (and clients) do as an sftp connection.

The server sends as part of it's init packet

"[email-masked]"
"\r\n"

And the clients look for this to know what to do
with files that are text.

- Joseph

---

From: Damien Miller <[email-masked]>
Date: Sun, 12 May 2002 16:08:06 +1000 (EST)

>       File access MUST be sequential.

The server should return an error if this is not supported, rather than
silently failing.

-d

---

From: "Joseph Galbraith" <[email-masked]>
Date: Mon, 13 May 2002 08:53:07 -0600

>       File access MUST be sequential.

First, I don't think SHOULD is strong enough here.
For this to be useful, it would need to be MUST.
Probably this text should be included:

        If the server can not comply, it must respond
        with status SSH_FX_OP_UNSUPPORTED.

I don't think I can determine the current encoding
of an arbitrary text file in order to convert
it's contents to UTF-8.  (Some files will be
encoded in unicode, and of course there is no
problem with those.)

I believe the unix folks have the same problem
with file names -- which is why I've been having
such a hard time getting them to swallow UFT-8
for file names.

I actually think that for both cases, it may be
necessary to make UTF-8 optional, determined
by the server.

I.e., the client MUST be able to read filenames and content
encoded in UTF-8.  The server MAY send filenames and/or
content in UTF-8.  If content/filenames are not encoded in UTF-8, 
there encoding is unspecified, and user intervention may
be required to determine how to display / save the file.

If the SSH_FXF_TEXT flag is set during open, and the server
will send file content in UTF-8, it should respond with
status code SSH_FX_OK_UTF8 (status code 9) instead of
SSH_FX_OK.

If the server will encode filenames in UTF-8, it should
include the following extension data in it's VERSION
packet (if and only if the clients INIT packet specified
a version >= 3.)

  "filename-utf8"        # extension name
  ""                     # no extension data

- Joseph