Subject: Draft draft of coming sftp revision...

From: "Joseph Galbraith" <[email-masked]>
Date: Fri, 20 Sep 2002 22:19:39 -0600

Hello everyone,

It seems like it has been a while since we
had a ruckus on the mailing list -- so I
thought I'd post a draft-draft of the
coming version of the sftp draft.

My hope is that we can go at least one round
of changes quick and then publish in a
couple of weeks.

The list of things I've noted that have changed 
is below.  (This list is also part of the
document.)  There may be things I've missed.

I think most of this has been talked about
on the mailing list and at least some sort
of loose consensus has been achieved.

I unilaterally added the new error codes
and the  reserved range of packet types
for use by extensions  -- so I'm soliciting
feedback on these.

Also, I included the 'newline' extension for
use in conjunction with the SSH_FXF_TEXT
file open flag semi-uni-laterally.  So again,
if you can't stand it, scream now.  I like it
because it means most servers won't actually
have to do anything to implement SSH_FXF_TEXT.

Also, does anyone remember what version we
are on for the draft?  I guessed 04 (inside,
I didn't change the file name for the posting)
but I can't remember for sure.

Thanks,

- Joseph

PS. If anyone wants, I can send XML source as well.

   Many of the changes between version 4 and version 3 are to the
   attribute structure to make in more flexible for non-unix platforms.

   o  Make all filenames UTF-8.

   o  Added 'newline' extension.

   o  Made file attribute owner and group strings so they can actually
      be used on disparate systems.

   o  Added ctime field, and added separate flags for atime, ctime, and
      mtime so they can be set separately.

   o  Split the file type out of the permissions field and into it's own
      field (which is always present.)

   o  Added acl attribute.

   o  Added SSH_FXF_TEXT file open flag.

   o  Added flags field to the get stat commands so that the client can
      specifically request information the server might not normally
      included for performance reasons.

   o  Removed the long filename from the names structure-- it can now be
      built from information available in the attrs structure.

   o  Added reserved range of packet numbers for extensions.

   o  Added several additional error codes.

Joseph

---

From: Simon Tatham <[email-masked]>
Date: Tue, 24 Sep 2002 13:54:09 +0100

Joseph Galbraith <[email-masked]> wrote:
> It seems like it has been a while since we had a ruckus on the
> mailing list -- so I thought I'd post a draft-draft of the coming
> version of the sftp draft.

First things first: I _like_ the human-readable error message field
in SSH_FXP_STATUS. This is something that institutions I have
contact with have been complaining bitterly about for ages - one lot
refused to provide SFTP to their users _at all_ simply because it
was unable to report errors with the same richness as SCP. Even if
every other proposed change gets shouted down, KEEP THIS ONE!

>    o  Added SSH_FXF_TEXT file open flag.

These seem like a half-decent 90% solution (covers Windows, Mac and
Unix with no difficulty), but I'm concerned about the other 10% :-)

For a start, if you consider text file issues to be only about
newline conventions, you thereby ignore other problems such as
character sets. If a server uses EBCDIC and a client doesn't, is it
really the expected behaviour of a text-mode file transfer that it
should yield an EBCDIC file at the client end or an ASCII file on
the server? I was vaguely under the impression that (say) FTP's idea
of a text-mode transfer was that character sets would get converted.
Granted that EBCDIC is a pretty extreme example, but if one system
uses ISO-8859-1 while another uses UTF-8, or (more subtly) if two
systems use UTF-8 with different default normalisation forms, this
might still be an issue.

If the wire format for text files use \r\n as a newline, what should
an implementation do if it sees a lone \r or a lone \n? It's always
annoyed me that so many protocol specifications use \r\n as the
newline and don't mention anything about this situation. Can we have
a definitive specification of this?

Seeking in text files worries me a little. If each server can choose
to define the wire text format to be its own newline convention,
then this means 90% of servers (as above) can effectively treat text
files as binary and seeking isn't a problem. But what about things
like VMS? Don't VMS text files use a fixed-length record per line?
This surely isn't covered in the 90% solution that is SSH_FXF_TEXT.

I'm still not convinced it wouldn't be better to use a solution that
runs parallel to the standard C library's fpos_t: you define an
opaque type representing a position in a text file, with one
well-known value which represents beginning-of-file. Then you
introduce a text-mode variant of SSH_FXP_WRITE, perhaps called
SSH_FXP_WRITE_TEXT, which takes one of these as input in place of
FXP_WRITE's `offset' field, and _returns the updated one_ alongside
the data. This approach has the advantage that it can be implemented
entirely in terms of standard C library functions, so it can be
supported easily on _any_ OS which already supports ANSI C, no
matter how exotic its text file format. If you want to, of course,
you could still define an extension that allowed mainstream servers
to announce that their fpos_t was an ordinary file offset, so that
clients supporting this extension could seek arbitrarily in text
files.

That's just my 2p, though. Practically speaking, SSH_FXF_TEXT would
eliminate 99% of the complaints I get from SFTP users. Even if I
might wish it had been done a different way, what you've got here
would be better than nothing.

Cheers,
Simon
-- 
Simon Tatham         What do we want?        ROT13!
<[email-masked]>   When do we want it?     ABJ!

---

From: "Joseph Galbraith" <[email-masked]>
Date: Tue, 24 Sep 2002 09:22:41 -0600

> every other proposed change gets shouted down, KEEP THIS ONE!

That one actually existed in version 3 of
the protocol, before I started editing 
the draft :-)  But I was a big proponent
of it too.

> systems use UTF-8 with different default normalisation forms, this

I am not a FTP guru, but I just talked to our FTP guru,
and he tells me that stock FTP, at least, doesn't do
any charset translation.  There might be an extension fo
it.

I, personally, don't think we want to solve this one.
I also think that if it's important to you, you can
do it without ietf action -- a couple of possibilities
off the top of my head:

    o include a "[email-masked]" extension attribute when
      the file is stat'ed and then let the client deal
      with translation.
    o allow the client to send something like
      SSH_FXP_EXTENDED
      "[email-masked]"
      string handle      (must be opened using FXF_TEXT)
      string charset

      and then have the server translate.

Personally, I think the more work we can keep out of the
server the better.

> a definitive specification of this?

Okay; that's fair.  My first inclination is to
say that anything not matching the servers newline
sequence exactly (either the default \r\n, or newline
as specified by the server) SHOULD NOT be translated.
In particular, if the newline convention is \r\n, and
the client see's a lone \r or lone \n, it should simply
write the data through without modification.

What do you think?

> This surely isn't covered in the 90% solution that is SSH_FXF_TEXT.

This one is tricky-- I think the thing we actually
reached semi-consensus about on the list was that
TEXT files would not be seekable.  However, that
disallows resume of partial transfers.

So, I was trying to allow the following scenerios,
which I suspect fall into the 90% catagory, to work:
    1. Resume aborted transfer
        1a. Allow an aborted FTP / SCP / RCP
            transfer to be resumed
    2. Transfer just the tail of a growing text
       file (a log for example.)

But, usually the only thing we have client
side is the file data.  The trouble with the
fpos_t solution is that in 1a, and 2, and
at least for us, sometimes 1 as well, the
fpos_t would not be available because it
was not written to disk (not file data) and
the application has restarted.

I don't think the current proposal is undoable
for servers where translation is necessary--
it is difficult though.  But, it is a problem
that has been solved by FTP servers for a decade
for the REST (restart) command.

The solution as it currently stands actually works
very much like FTP -- with the exception that
90% of the implementations don't have to do
any work :-)

The server must be able to make following calculation
in order to support seek.

Given:
    o the servers stated newline sequence
    o An offset from the beginning of the data
      as it would appear on the wire

The server must be able to calculate
    o the actual offset in the file.

A client must also be able to do a similar mapping:

Given:
    o the servers stated newline sequence
    o A current offset in it's local file

The client must be able to calculate
    o the offset from the beginning of the data
      stream.

As I said, FTP has been tackling this problem for
a long time.

The other thing I would note is that if a server
want's to support resume, it only needs to support
the offset field on the first write or read.  And
it doesn't need to support that if it doesn't
want to support resume.  I wrote the text to
require clients to expect failure during
non-sequential operation.

Thanks for your feedback, by the way.  I expected
far more feedback than I've gotten -- so it's
nice to know it actually made it to the list :-)

- Joseph

---

From: Simon Tatham <[email-masked]>
Date: Tue, 24 Sep 2002 16:42:16 +0100

Joseph Galbraith <[email-masked]> wrote:
> I am not a FTP guru, but I just talked to our FTP guru, and he tells
> me that stock FTP, at least, doesn't do any charset translation.
> There might be an extension for it.

Oh, OK. Fair enough.

> [charset translation can be done with an extension]

That's fair enough too.

> Okay; that's fair.  My first inclination is to say that anything not
> matching the servers newline sequence exactly (either the default
> \r\n, or newline as specified by the server) SHOULD NOT be
> translated. In particular, if the newline convention is \r\n, and
> the client sees a lone \r or lone \n, it should simply write the
> data through without modification.
> 
> What do you think?

Yes, that seems as good as anything to me. Realistically the only
instances of this I can easily bring to mind are

 (a) text files being stored on the server in the client's format
     already, in which case this algorithm will preserve whichever
     of \r and \n was the client's line-ending character, so it will
     end up looking plausible.

 (b) DOS text files containing CRs for purposes of visual weirdness,
     in which case this algorithm will transfer them correctly to
     Unix. It'll fail horribly when transferring them to a Mac, but
     they were doomed anyway in that situation.

So I think this is a plausible solution, yes.

[resuming transfers]
> But, usually the only thing we have client side is the file data.

Yes, I was thinking that too. Even if you can seek in server-side
text files, you'd have to scan right through the client-side
partially transferred file to work out where to seek _to_.

> The trouble with the fpos_t solution is that in 1a, and 2, and at
> least for us, sometimes 1 as well, the fpos_t would not be available
> because it was not written to disk (not file data) and the
> application has restarted.

Irritating, but probably true in most cases with current software.

However, software might change. I can easily imagine a text-aware
SFTP client continuously updating a hidden temporary file (much like
CVS's `.#' files) which would contain a server-side file offset
(fpos_t or an ordinary file offset, according to taste) and a local
file size. Then a transfer could be resumed by seeking to the given
remote position on the server, and to the given local position on
the client, and continuing to copy data. The temp file would be
removed when the file was fully transferred.

> The other thing I would note is that if a server wants to support
> resume, it only needs to support the offset field on the first write
> or read.  And it doesn't need to support that if it doesn't want to
> support resume.  I wrote the text to require clients to expect
> failure during non-sequential operation.

I'd recommend supporting the offset field at least to some extent,
for purposes of multiple parallel sequential reads. Or would you
expect a client to open the same file several times to do this?

Cheers,
Simon
-- 
Simon Tatham         "loop, infinite _see_ infinite loop"
<[email-masked]>     - Index, Borland Pascal Language Guide

---

From: "Joseph Galbraith" <[email-masked]>
Date: Wed, 25 Sep 2002 13:54:38 -0600

> removed when the file was fully transferred.

But, consider the situation when the file wa
fully transferred, but has grown since the last
time I transfered it.  I just want to transfer
the new data.  A log file is a good example of this.
This looks just like a transfer that didn't complete.

Or, consider the solution where another protocol or a
different client preformed the first part of the transfer.
(I admit this is an edge case.)

Neither of these cases will work if additional metadata is
required to resume the transfer.

> expect a client to open the same file several times to do this?

Okay-- I think I see what you are saying.  I'm not sure
how to change things to be clear about how this should
work.

I don't think the client should open the same file several
times for this-- in fact, I think the correct behavior for
the client is that if the server returns SSH_FX_OP_UNSUPPORTED
for a parallel sequential read, the client should fall
back to non-parallel reads.

But I think it is now obvious we need specify more
clearly about how the offset field should be used
when FXF_TEXT is used, and what the server SHOULD
do and what it MAY do.

I'll see if I can't rework the section and then probably
repost the draft again.

Okay-- I thought about this for a couple of minutes;
here are my first and second ideas... let me what
you think.

----- 1 -----
"When FXF_TEXT|FXF_APPEND are specified, the offset field
 is ignored and all writes will be appended to the file.

"If FXF_TEXT|FXF_APPEND|FXF_READ are specified, the
 read offset field has special behavior.  An offset of
 0 is considered a request for the next sequential
 data in the file.  Any other read offset is considered
 a seek and read operation.

"It is not possible to seek to position zero in the file.
 Alternatively, clients can close and reopen the file to
 seek position 0.

"Servers SHOULD support seek-and-read in order to support
 resume.  However, a client MUST be prepared to receive
 SSH_FX_OP_UNSUPPORTED when attempting a seek-and-read
 operation.  The client can then try a fall-back
 strategy, if it has one."

"Servers SHOULD correctly process multiple parallel
 reads correctly in this mode.  However, if the client
 recieves SSH_FX_OP_UNSUPPORTED when attempting multiple
 parallel reads, it SHOULD fall back to serial reads."

----- or #2 -----

"Servers SHOULD correctly process multiple parallel
 reads correctly in this mode.  However, if the client
 recieves SSH_FX_OP_UNSUPPORTED when attempting multiple
 parallel reads, it SHOULD fall back to serial reads."

"Clients SHOULD use the SSH_FXF_APPEND flag to append
 data to a text file rather then using write with a
 calculated offset.

"When a file is opened with the FXF_TEXT flag, the
 offset field in both the read and write function
 are ignored.

"To support seeks on text file the following
 SSH_FXP_EXTENDED packet is defined"

"string 'text-seek'
 uint64 line-number

"line-number is the index of the line number to
 seek to, where byte 0 in the file is line number 0,
 and the byte directly following the first newline
 sequence in the file is line number 1 and so on.

"Servers SHOULD support at least on 'text-seek' in
 order to support resume.  However, a client MUST
 be prepared to receive SSH_FX_OP_UNSUPPORTED when
 attempting a 'text-seek' operation.  The client
 can then try a fall-back strategy, if it has one."

So what do people think?  I think I'm leaning towards
#2-- it doesn't involve overloading the meaning
of the FXF_APPEND flag, which I think is a cleaner
design.  It also doesn't have the restriction about
seeking position 0.  Additionally, I think it makes
the calculation of where to seek less error prone.

Thanks again for your feedback.

- Joseph