Subject: SFTP File open modes

From: Richard Whalen <[email-masked]>
Date: Wed, 27 Mar 2002 14:36:06 -0500

The current specification of SFTP states that files are always opened in
"binary" mode - no translations between different character sets and newline
encodings.

It is well known that different systems use different methods of encoding
text files.  If SFTP desires to be able to provide a secure way of
interchanging files between systems, then it must specify a method in which
text files can be exchanged in a uniform manner.  To address this problem, I
propose the following:

add the flag SSH_FXF_ASCII to the SSH_FXP_OPEN message.
When this flag is set, the file will be transferred in system independent
ASCII text format, if appropriate.
The system independent format is defined as characters separated by a
newline character to specify end of line.  The system that is storing the
file may make any changes necessary to store the text in a manner that is
compatible with standard methods for that system.  When a file is
transferred in ASCII mode, the total number of bytes transferred may be
different from the size provided by the SSH_FXP_xSTAT function.  Ideally,
the number of bytes transmitted should never be greater.

A request to open a file in ASCII format that can not be opened in ASCII
format (and should probably be transferred in binary format) will return a
failure status of FX_OP_UNSUPPORTED.

It may be desirable to include information in the response to a xSTAT
function and READDIR stating whether or not a file is candidate to be opened
in ASCII format.  This could be indicated in the file type information, in a
manner similar to the way that a directory file is indicated.

----------------------
Richard Whalen
Process Software
508-879-6994

---

From: Simon Tatham <[email-masked]>
Date: Wed, 27 Mar 2002 22:02:32 +0000

> the number of bytes transmitted should never be greater.

This seems fine to me as far as it goes, but how do you split the
file into multiple FXP_READs when there's no persistent file pointer
and each FXP_READ must specify a precise file offset where the
previous one left off?
-- 
Simon Tatham         These are my opinions. There are many
<[email-masked]>   like them but these ones are mine.

---

From: nico <[email-masked]>
Date: Fri, 29 Mar 2002 22:12:31 -0500

How many well-known and often utilized ways of encoding ASCII text files
are there (answer: two)? Is an ASCII mode as you suggest likely to be
useful? Or do we want a mode for UTF-8 and UTF-16, EBCEDIC, and so on?

Perhaps it would be best to add an operation by which the client can ask
the server for a given file's content type and encoding. The server's
response should indicate whether it knows, whether it knows for certain
and, if it knows at all, what the content type is.

All encoding conversions though should be left to the client though.

IMHO. Cheers,

Nico

> "binary" mode - no translations between different character sets and newline

[...]

---

From: Albert Lunde <[email-masked]>
Date: Sat, 30 Mar 2002 10:32:15 -0600

>All encoding conversions though should be left to the client though.

* First, one should distingush between a literal _ASCII_ mode, and a
_text_ mode.

The most specific proposal posted suggested using Latin-1 as a
least-common-denominator wire encoding, with a bunch of negotiated
alternatives. In this day and age, I think it would make more sense
to use something like UTF-8 (further spectified to reduce ambiguties).

I count at least three widely-used ways of encoding end-of-lines with
characters (CR,LF, CRLF) plus fixed-length records, plus counted
varable-length records: I don't know what you are thinking of in
saying "two".

* Text mode is a big win in the case of an OS which has files in some
well-known but ideosyncratic format, because the clients don't have
to say, know all the varients of EBCDIC or VMS record formats, just a
single wire format.

* But I think you've got a point that a server may not _know_ the
encoding of files, or may not want to support the overhead of a text
mode.

I don't know any foolproof way to detect which files in my Mac are in
Mac-extended-ASCII, which are in Latin1, and which are in Shift-JIS
(because I've been running the Japaneses language kit on on of my
Macs for years). The Mac type/creator scheme is more or less
orthogonal to i18n issues.

So this isn't something we should expect to be able to implement on
every server, or for every file.
-- 
     Albert Lunde         [email-masked] (new address)
                          [email-masked] (old address)

---

From: nico <[email-masked]>
Date: Sun, 31 Mar 2002 11:45:51 -0500

> varable-length records: I don't know what you are thinking of in

I confess: I was thinking DOS vs. Unix (CRLF vs. LF).

> to say, know all the varients of EBCDIC or VMS record formats, just a

As someone else just pointed out, nowadays text documents come in
complex binary formats such as PDF and what not.

I think having dos2unix and unix2dos support in the client would be nice
for us *nix types. But I can do without. And certainly I don't think
this belongs on the server side.

> orthogonal to i18n issues.

Heh. Good luck.

> every server, or for every file.

Or any file types at all, not in the server, or rather, not in the
protocol. Having the server identify file types would facilitate
client-side file type conversions - having the server do the type
conversions (as proposed) complicates the protocol. Any conversions
should be optional (i.e., MAY).

Cheers,

Nico