Subject: Fw: solving the SFTP text mode issue

From: "Joseph Galbraith" <[email-masked]>
Date: Fri, 10 May 2002 14:23:50 -0600

Whoops, forgot my reply all button.

- Joseph

----- Original Message -----
From: "Joseph Galbraith" <[email-masked]>
To: <[email-masked]>
Sent: Friday, May 10, 2002 14:23
Subject: Re: solving the SFTP text mode issue

> > > Text file conversion is best handled by the routines that
> > > access the file on the system that the file is stored on.
> >
> > That is true, but I see no nice way to implement this in SFTP, because
it
> > clashes with SFTP's assumption that all files are binary. It is
> incompatible
> > with SFTP's random-access concept.
> >
> > A perfect solution to the text-file issue probably requires a mechanism
> > completely independent of SFTP's read/write operations. But an
independent
> > mechanism for text files seems like an overkill for implementations that
> > will only interact with mainstream platforms anyway.
> >
> > I would prefer to have standardized a simple solution that solves the
> > problem for the mainstream platforms, and let the exotic people
implement
> > their own complicated extensions that I don't need to support.
> >
> > Unless someone can come up with a really straightforward solution that
> > covers exotic formats as well. Perhaps this can be done, but it will
have
> to
> > be done by an ingenious someone who is well-acquainted with those
formats.
>
> Hmm... I guess we could add
>
> SSH_FXP_READ_RECORD
> uint32 start record number
> uint32 number of records to read
>
> and the response would be
> SSH_FXP_RECORD
> uint32 start record number
> string record[n]
>
> and
> SSH_FXP_WRITE_RECORD
> uint32 start record number
> uint32 number of records to write
> string record[n]
>
> and a flag to open a file in record mode.
> SSH_FXF_RECORD_ORIENTED.
>
> Openning a text file in record mode would
> resulting in recieveing each line as a
> seperate record with no termination.
>
> A server could refuse to open an existing
> file in record mode if it couldn't figure
> out how to.
>
> I'm not sure I'm really advocating this
> approach ... I haven't thought about it
> for more than ten minutes... but it does
> seem like it could solve the more
> general problem?
>
> - Joseph
>

---

From: "denis bider" <[email-masked]>
Date: Fri, 10 May 2002 23:20:34 +0200

> seperate record with no termination.

Doesn't sound half bad... Seeking to the middle of an LF-separated text file
would require the server to manually count the lines, but I guess this isn't
really a problem if we consider that the other possibilities don't offer
random access in the first place.

Also, we might offer a special record number, e.g. -1, for use with
SSH_FXP_WRITE_RECORD to indicate that the record is to be appended. Or
perhaps we could simply provide another packet type called
SSH_FXP_APPEND_RECORD.

How does this solution resonate with you all VMS people?

---

From: "Dan O'Reilly" <[email-masked]>
Date: Fri, 10 May 2002 15:28:37 -0600

>How does this solution resonate with you all VMS people?

I think it's close.  There are 3 main fundamental formats we must deal
with:

- Stream LF (i.e., "UNIX" mode)
- delimited, variable-length records (usually, CR/LF pairs)
- variable-length records with a fixed-length binary record-size field
   at the beginning of each record

There are other permutations of text files in VMS, but they're basically
"variations on a theme", in that they're a derivation of one of the above
formats, and so what works for the above should work for them as well.

So, this solution would seem to apply to the 2nd format (the first format
is obviously handled today), but I'm not sure of the 3rd format.  We would
need to examine this proposal and see what the implications are.

------
+-------------------------------+---------------------------------------+
| Dan O'Reilly                  |                                       |
| Principal Engineer            |  "Why should I care about posterity?  |
| Process Software              |   What's posterity ever done for me?" |
| http://www.process.com        |                    -- Groucho Marx    |
+-------------------------------+---------------------------------------+

---

From: "denis bider" <[email-masked]>
Date: Fri, 10 May 2002 23:52:35 +0200

> variable-length records with a fixed-length binary record-size
> field at the beginning of each record
>
> So, this solution would seem to apply to the 2nd format (the
> first format is obviously handled today), but I'm not sure of
> the 3rd format.

What could be the problem? Random access?

If so, then perhaps Wei's suggestion is best - simply open the file in "text
mode", translate the lines to be LF-terminated if necessary, and enforce
sequential access. Perhaps provide another mode for appending to the file,
and that's it.

How does that sound in the VMS context?

As said, you VMS people must contribute actively, otherwise we'll just
implement something that works on Windows, et voila.

---

From: "Dan O'Reilly" <[email-masked]>
Date: Fri, 10 May 2002 16:00:22 -0600

>implement something that works on Windows, et voila.

and as I already said, we have to study it to make sure it's a good
solution.  We'll provide an answer when we have one, rather than just
jumping off without thinking it through.

------

---

From: "Joseph Galbraith" <[email-masked]>
Date: Fri, 10 May 2002 16:21:56 -0600

> need to examine this proposal and see what the implications are.

My model of the world in creating this was:

1. Binary files

   Protocol works as specified.

2. Record oriented files -- where record files have the following
   sub-cases:

   a. Fix length records
   b. Variable length records

      Variable length records might be either delimited
      or counted.

      In the delimited case, the delimiter doesn't really
      matter.  It could be \n, \r, \r\n, or , -- it doesn't
      matter.  (Both your stream LF and delimited variable-length
      records fit into this catagory.)

The solution proposed works extremely well in the
fixed length record cases -- in fact, I would say
it is ideal.  It provides, fast, efficient, random
record based access.

In the variable length record case, it isn't so
great in terms of efficiency.  The server would
probably have to do some optimization -- for
example, preread the file and record where the
record breaks are, or, more likely, simply record
where the last record ended and it's number,
so sequential reads work well.

Counted variable length maps pretty well,
since each string in the array containing
the records has its own count.  I.e., on
the wire it looks like:

Byte    SSH_FXP_RECORD
uint32  32                # starting at record 32
uint32  3                 # 3 records included

uint32  45                # Record 32 is 45 bytes
Byte[45]                  # The 45 bytes for record 32

uint32  143               # Record 33 is 143
BYTE[143]                 # The 143 bytes of data for recrd 33

uint32  1951              # Record 34 is 1951 bytes
BYTE[1951]                # The 1951 bytes of data for record 34

Delimited variable length records look to the client
like counted records, so the client would probably
need some other mechanism to recreate the original
file attributes.  I.e., something during stat
that says this file is variable length, \r\n delimited.

In addition, the delimited case has the additional
challange that the server needs to determine what
the delimiter is -- on VMS, it can probably retrieve
that information from the file attributes.

For the general case, we may want to add the
additional flag:

  SSH_FXF_TEXT_MODE

which implies RECORD_ORIENTED, variable length
records, with the servers native newline character
as the delimiter.

- Joseph

PS. Sorry Dan, I already sent this to you once--
I'm having a bad day with the 'reply all' thing.

---

From: "denis bider" <[email-masked]>
Date: Sat, 11 May 2002 01:51:09 +0200

> it is ideal.  It provides, fast, efficient, random

The question is: is there a need for random access to record-based files?

After some consideration, I think that the READ_RECORD/WRITE_RECORD proposal
is probably too enthusiastic. Implementing random access to files where
records are lines of text will be difficult on most platforms. Most
applications don't currently do random access in the first place, so it's
hard to imagine it being done much with records.

But if we dismiss random access, than the only remaining advantage of the
readrecord/writerecord proposal is that it allows LF to be embedded in
lines.

I think that Wei's proposal is the most balanced of all I've considered
today:

- It does seem to support all kinds of text files reasonably well, including
record-based text files. The latter is true as long as individual records
don't contain LF characters, which I assume is reasonable to expect from a
text file.

- It is simple to implement on Windows and Unix. It resembles the approach
taken by C/C++ standard libraries to solve the same problem. Translation
between CRLF and LF is simple. No convoluted algorithms to support random
access need to be implemented.

---

From: "Joseph Galbraith" <[email-masked]>
Date: Mon, 13 May 2002 08:42:46 -0600

> access need to be implemented.

In terms of solving the text transfer problem,
Wei's proposal is sufficient I believe, although
it does not provide for resuming a large text
transfer if it is interupted.  (Something our
customers want to be able to do.)

I'll talk more about Wei's solution in a response
directly to it.

I must say that the solution I proposed was
not aimed at solving the text transfer problem,
per say, but at solving the record file access
problem that VMS folks have (and probably
mainframe folks too.)

It just happens that the text access problem
is a subset of the record access problem.

Now, if the random access thing bothers your
too much, we might say something like:

If the SSH_FILEXFER_ATTR_RECORD_ACCESS
flag is not set, the file can not be
opened for RECORD access.  Any attempt
to do so will result in a
SSH_FX_INVALID_ACCESS_MODE error.

If the SSH_FILEXFER_ATTR_RANDOM_ACCESS
flag is not set, the offset and / or
record number field is ignore in
READ, READ_RECORD, WRITE, and WRITE_RECORD
operations and the file is read / written
in sequential order.

Now, VMS (or any other operating system that
natively supports random access record oriented
files) can natively support this, and NT or
unix, for example,

Of course, on the other hand, I'm not
really attached to this solution -- only
proposing that it solves more of the proposed
problem then other solutions proposed.

Personally, if we want to solve just the
text access solution, we have the server
advertise what line delimiter it will
use, and the MacOS and VMS folks who
deal with different kinds on a per file
basis can do conversions on the server side.

I.e., the VMS server would convert a
variable length counted file into \r\n
delimited, and always advertise \r\n
as its line delimiter.

As I said before, we already implement
this solution in our servers, and if
other servers implemented it, it would
solve 99% of our customers problems,
which is a lot more than we currently have.

And it doesn't really prevent implementors
working under VMS from using the solution --
they just have to work harder.  They have
to do the server side translation work
the rest of us are avoiding by not specifying
the canonical line termination in the draft,
but rather allowing the server to specify it's
own cananical line termination.

- Joseph