Subject: Re: draft-ietf-secsh-publickeyfile-00.txt

From: Jeffrey Altman <[email-masked]>
Date: Thu, 25 Jan 2001 10:40:01 EST

30 years of experience transfering files between heterogeneous systems
are being ignored in this protocol.  Transfering everything as binary
data is fine when using homogeneous systems.  But it is not acceptable
when transfering text files among heterogeneous systems.  Just look at
what happens when you attempt to transfer a makefile produced on
Windows to a Unix host.  Or worse, to a Macintosh.

>   - make it very hard to restart large transfers which may have been
>     aborted

It is very hard to restart text transfers regardless of who is
performing the conversion because the size of the file changes when
the end of line indicator is manipulated.
 
>   - break the file offset addressing model used in the protocol (note that
>     it is essentially the same model used in POSIX file I/O)
> 
>   - increase cpu consumption on the server (probably not significant)

not significant.  Kermit and FTP have been performing these
conversions for years.

> Most MSDOS programmers I know of automatically add the "b" (binary) flag
> to every fopen call they write...

Microsoft Development tools do not.  Neither do most editors.

> Anyway, I think the client is the right place to do the conversions.

It is appropriate for both the client and the server to perform
conversions as necessary.  When transfering text files you must have a
well defined wire format.  CR-LF is the usual standard.  The client
converts (if necessary) to CR-LF and the server converts if necessary
to the local representation.

You can provide optimizations by negotiating the system type and
always use binary mode if the system types match.  

 Jeffrey Altman * Sr.Software Designer      C-Kermit 7.1 Alpha available
 The Kermit Project @ Columbia University   includes Secure Telnet and FTP
 http://www.kermit-project.org/             using Kerberos, SRP, and 
 [email-masked]          OpenSSL.  SSH soon to follow.

---

From: Tatu Ylonen <[email-masked]>
Date: Thu, 25 Jan 2001 22:02:00 +0200 (EET)

> Windows to a Unix host.  Or worse, to a Macintosh.

Most people hardly ever transfer a Makefile between Unix, Windows, or
Macintosh.  They transfer a tgz, a zip, or an hqx.

Transferring plain ASCII files is becoming extremely rare.  Even the
simplest memos now most often go with the mime type
application/msword.  Postscript has been replaced by pdf.  Images are
gifs, jpgs, tiffs, or some other binary format.  HTML pages
frequently have national characters from iso8859-x or some Japanese
character set.  Spreadsheets are usually made with Excel and presentations
with Powerpoint.  All these formats are binary.  I don't see a big move
back to non-binary formats in the future.

I may be an exception, but I have never, ever, in my 20 years of intensely
using computers (including some years of VAX, UNIVAC, Burroughs mainframe
use, and several years of CP/M, MSDOS and Windows use, and having worked
with full-text databases and newspaper editorial systems several years,
including many ten-hour kermit transfers over serial lines) intentionally
transferred a file using ASCII mode.  I have many times accidentially
transferred one in ASCII, and have sometimes spent considerable amount of
time debugging before figuring out what was wrong. I have also had several
customer complaints when the files they got had been accidentially
corrupted by ASCII transfer.

I will make the claim that automatically defaulting to ASCII mode is
HARMFUL, and is becoming MORE HARMFUL as time goes by.  If the user
explicitly wants to convert, he/she can then specify conversion type to
the client (e.g., convert for VAX/VMS).

(I have received some (1-3) requests for ASCII mode in SSH/SCP/SFTP during
the last five years; however, I have not been reading the support mailing
lists myself for several years now.  Some people are used to using ASCII
mode in legacy applications, which may warrant supporting it when
explicitly requested.)

> to the local representation.

I vote against adding the complexity of ASCII conversions into the
protocol.  My reason for this opinion is that I think it can be
implemented in the client with the same complexity without modifying
the protocol (also thinking of what the user has to do to enable it), and
keeping it as a client implementation issue avoids bloating
the complexity of the protocol and of server implementations.

I will never release an implementation that would have ASCII mode turned
on by default, because I think doing so would be seriously harmful for
ease of use.

If it has to be included in the protocol, I would recommend adding a "Text
mode" attribute as an extension attribute that can be passed in the file
open/create packet as part of the file attributes.  This would request the
server to manipulate the file in "text mode" according to its local
conventions, converting whatever newline to server local newline in
writes, and for reads the client side would always handle the conversion
to the client's local conventions.  Offsets would be defined to refer to
the wire format that has been used for that file handle.

However, I repeat that I am personally opposed to the adding the feature.

    Tatu

---

From: Mats Andersson <[email-masked]>
Date: Fri, 26 Jan 2001 09:48:39 +0100 (MET)

Hi,

> Macintosh.  They transfer a tgz, a zip, or an hqx.

Please, we are discussing a communications protocol here, not an
application!! The protocol is clear and simple, it doesn't need to include
information on what kind of files we are transfering! On top of this
protocol one can easily implement all sorts of filtering/transformation.
The protocol even supports these kinds of extensions through explicitly
defining how extensions can be done.

from draft-ietf-secsh-filexfer-00.txt:
...
8.  Vendor-Specific Extensions

The SSH_FXP_EXTENDED request provides a generic extension mechanism for
adding vendor-specific commands.
...

Also, when transfering ASCII that should be translated in one way or the
other, doesn't it often feel more intuitive that one have to do the
translation by "external" means after the file have been transfered?

So please let's not complicate any of these specs more than necessary.

Cheers,

/Mats

---

From: Damien Miller <[email-masked]>
Date: Sat, 27 Jan 2001 10:42:33 +1100 (EST)

On Thu, 25 Jan 2001, Jeffrey Altman wrote:

> 30 years of experience transfering files between heterogeneous
> systems are being ignored in this protocol.  Transfering everything
> as binary data is fine when using homogeneous systems.  But it is
> not acceptable when transfering text files among heterogeneous
> systems.  Just look at what happens when you attempt to transfer
> a makefile produced on Windows to a Unix host.  Or worse, to a
> Macintosh.

IMO ASCII conversion is more pain than gain - Its lack can be worked 
around with simple tools (most applications support multiple CRLF 
conventions), but I have never had anything but grief and corrupted
files from its presence.

-d

-- 
| ``We've all heard that a million monkeys banging on | Damien Miller -
| a million typewriters will eventually reproduce the | <[email-masked]>
| works of Shakespeare. Now, thanks to the Internet, / 
| we know this is not true.'' - Robert Wilensky UCB / http://www.mindrot.org

---

From: Michael Richardson <[email-masked]>
Date: Sun, 28 Jan 2001 13:20:37 -0500

>>>>> "Tatu" == Tatu Ylonen <[email-masked]> writes:

    Tatu> Most people hardly ever transfer a Makefile between Unix, Windows,
    Tatu> or Macintosh.  They transfer a tgz, a zip, or an hqx.

    Tatu> Transferring plain ASCII files is becoming extremely rare.  Even
    Tatu> the simplest memos now most often go with the mime type
    Tatu> application/msword.  Postscript has been replaced by pdf.  Images
    Tatu> are gifs, jpgs, tiffs, or some other binary format.  HTML pages

  Recall that this is being argued under the subject of the public key file format.
I will make the following comments:
  1) I've most frequently transfered it via cut&paste.
     cat ~/.ssh/identity.pub
     cat >>~/.ssh/authorized_keys

  2) I've sometimes had to uuencode it to get around xterm's with limited
    line length support.

  3) finally, i've used scp. I'd like to see "sftp" be capable of passing
  a MIME type for each file. 

    Tatu> frequently have national characters from iso8859-x or some Japanese
    Tatu> character set.  Spreadsheets are usually made with Excel and
    Tatu> presentations with Powerpoint.  All these formats are binary.  I
    Tatu> don't see a big move back to non-binary formats in the future.

  If one assumes the trend that you list, which is best characterized as
"Microsoft Office" dominance in file formats, then the format won't be ASCII, 
but rather will explicitely be "sequence of OLE objects", which is a
structured format.

    Tatu> I will make the claim that automatically defaulting to ASCII mode
    Tatu> is HARMFUL, and is becoming MORE HARMFUL as time goes by.  If the
    Tatu> user explicitly wants to convert, he/she can then specify
    Tatu> conversion type to the client (e.g., convert for VAX/VMS).

  I think that the topic of "defaults" is really a client issue.
  The protocol should explicitely specify it one way or another.

    Tatu> I vote against adding the complexity of ASCII conversions into the
    Tatu> protocol.  My reason for this opinion is that I think it can be
    Tatu> implemented in the client with the same complexity without
    Tatu> modifying the protocol (also thinking of what the user has to do to
    Tatu> enable it), and keeping it as a client implementation issue avoids
    Tatu> bloating the complexity of the protocol and of server
    Tatu> implementations.

  I strongly suggest putting MIME types in. If I open a file in "text/*"
then that may mean something. It does mean that I know what the conventions
are! they are specifically those defined in SMTP.

] Train travel features AC outlets with no take-off restrictions|gigabit is no[
]   Michael Richardson, Solidum Systems   Oh where, oh where has|problem  with[
]     [email-masked]   www.solidum.com   the little fishy gone?|PAX.port 1100[
] panic("Just another NetBSD/notebook using, kernel hacking, security guy");  [

---

From: Tatu Ylonen <[email-masked]>
Date: Mon, 29 Jan 2001 00:56:05 +0200 (EET)

> are! they are specifically those defined in SMTP.

I strongly oppose.  The server usually has no more information available
than the client for determining the mime type.

    Tatu

---

From: Michael Richardson <[email-masked]>
Date: Sun, 28 Jan 2001 19:25:11 -0500

>> then that may mean something. It does mean that I know what the
    >> conventions are! they are specifically those defined in SMTP.

    Tatu> I strongly oppose.  The server usually has no more information
    Tatu> available than the client for determining the mime type.

  On Unix systems, I agree. If the server doesn't know, it can use 
"application/octet-stream" and be done with it.

  On other systems (particularly Mac), I disagree. The hint is very useful
if it allows the receiving system to decide how to do the fopen().

---

From: Damien Miller <[email-masked]>
Date: Mon, 29 Jan 2001 14:15:42 +1100 (EST)

> if it allows the receiving system to decide how to do the fopen().

This seems like overkill for identifying text from binary and could be 
accomodated using the existing SSH_FILEXFER_ATTR_EXTENDED mechanism.

-d

---

From: [email-masked] (Niels M�ller)
Date: 29 Jan 2001 14:45:02 +0100

> are! they are specifically those defined in SMTP.

If you want MIME content types (and the system where the file lives happens to
know the content type), put the content type in the
SSH_FILEXFER_ATTR_EXTENDED. Not in the core protocol.

Actually, after some more thinking, I think that approach is
preferable als in the discussion of utf-8 filenames. In the core
protocol, treat the names of files as arbitrary octet strings, with no
structure or interpretation beyond using the /-character as a
directory separator. Treat the contents of files as octet streams.

If fancy conversions of file names and file contents, negotiation of
conventions on both sides, etc, are desired, add those features as
extensions, don't add them to the core protocol.

/Niels