Subject: RE: solving the SFTP text mode issue

From: Richard Whalen <[email-masked]>
Date: Fri, 10 May 2002 15:36:21 -0400

Good suggestions, though the 'client' should not be required to understand
text storage modes.

Text file conversion is best handled by the routines that access the file on
the system that the file is stored on.  Doing it in the file access routines
makes it such that an implementation only needs to understand its storage
method(s) and the common storage method. Requiring the 'client' to do
conversion would mean that the client would have to have information about
storage methods for all systems that it desires to interoperate with
included in its (the clients) implementation.

Some methods do not have explicit line separator(s), but have an implicit
line break and the end of each record stored.  There may be additional
binary information that varies between the individual records (lines).

-----Original Message-----
From: denis bider [mailto:[email-masked]]
Sent: Friday, May 10, 2002 3:19 PM
To: [email-masked]
Subject: solving the SFTP text mode issue

Judging from the number of times that text file support has been requested
for SFTP, we might as well do something about it. I suggest that, in the
next version of the filexfer draft, we document a recommended way for client
software to do text file conversion. (On its own, of course, without any
ugly hacks into the protocol.)

What if we provide an SFTP command that the client could execute to learn
the line separator on the server machine?

We could then document a text file conversion algorithm such as the
following:
1. The client queries the server about the server's line separator.
2. If the server's line separator is the same as the client's, the file is
transferred in binary mode.
3. Otherwise, the client determines whether the file is textual using, for
instance, the Kermit 8 algorithms mentioned by Jeffrey earlier on this list.
If the file is textual, the client converts it.

---

From: "denis bider" <[email-masked]>
Date: Fri, 10 May 2002 22:00:44 +0200

> Text file conversion is best handled by the routines that
> access the file on the system that the file is stored on.

That is true, but I see no nice way to implement this in SFTP, because it
clashes with SFTP's assumption that all files are binary. It is incompatible
with SFTP's random-access concept.

A perfect solution to the text-file issue probably requires a mechanism
completely independent of SFTP's read/write operations. But an independent
mechanism for text files seems like an overkill for implementations that
will only interact with mainstream platforms anyway.

I would prefer to have standardized a simple solution that solves the
problem for the mainstream platforms, and let the exotic people implement
their own complicated extensions that I don't need to support.

Unless someone can come up with a really straightforward solution that
covers exotic formats as well. Perhaps this can be done, but it will have to
be done by an ingenious someone who is well-acquainted with those formats.

---

From: "Dan O'Reilly" <[email-masked]>
Date: Fri, 10 May 2002 14:13:33 -0600

>with SFTP's random-access concept.

Perhaps this means that the fundamental concept was flawed...?

>will only interact with mainstream platforms anyway.

Ah, the chauvinism shows through.  The issues being raised here are by
real, live customers who need to make their UNIX systems interact with
VMS platforms, not just between VMS platforms.  Unless, of course, UNIX
is gone from "mainstream" now.  And by the way, I suggest you revisit your
definition of "mainstream" after reviewing all the critical applications
that exist and will continue to exist on VMS platforms in critical
industries that REQUIRE security, such as banking, manufacturing and
communications.

But platform bias aside, the implementation of an adequate mechanism seems
to be only moderately difficult.  Why not take a chance and make a standard
a REAL standard, not just one that applies to selective platforms?

>their own complicated extensions that I don't need to support.

"I don't need to support"?  Hmmm...why is constant support required, once
the standard is established and the basic coding complete?  Surely aside
from minimal maintenance, this would not be the support nightmare you seem
to make it out to be.  And text format files hardly qualify as "exotic".

------
+-------------------------------+---------------------------------------+
| Dan O'Reilly                  |                                       |
| Principal Engineer            |  "Why should I care about posterity?  |
| Process Software              |   What's posterity ever done for me?" |
| http://www.process.com        |                    -- Groucho Marx    |
+-------------------------------+---------------------------------------+

---

From: "denis bider" <[email-masked]>
Date: Fri, 10 May 2002 23:03:55 +0200

> > That is true, but I see no nice way to implement this in SFTP,
> > because it clashes with SFTP's assumption that all files are
> > binary. It is incompatible with SFTP's random-access concept.
>

I don't think SFTP's fundamental concept is flawed. On the contrary, I find
it very appealing. It makes possible things that aren't at all possible with
protocols that can only transfer a whole file at a time.

The disadvantage of SFTP's design is that it assumes files to be binary, and
this makes the protocol less suitable for files that are not.

> Ah, the chauvinism shows through. [...] "I don't need to
> support"? Hmmm...why is constant support required, once
> the standard is established and the basic coding complete?

I don't see why you need to be so bitter. It's not like I'm forcing anything
down your throat. I'm just expressing my point of view, and that point of
view isn't solid, but is updated by each new argument that you contribute.

> But platform bias aside, the implementation of an adequate
> mechanism seems to be only moderately difficult.  Why not
> take a chance and make a standard a REAL standard, not just
> one that applies to selective platforms?

Sure, why not? But people like you must take initiative and/or provide
feedback, otherwise the problem won't be solved.

---

From: "Dan O'Reilly" <[email-masked]>
Date: Fri, 10 May 2002 15:20:17 -0600

>protocols that can only transfer a whole file at a time.

It's very obvious that the fundamental concept is based on the idea that any
file can be transferred intact and not require translation via a fixed-length
block format.  99.999% of the time, it's because the people doing the basic
conceptual design don't know anything save for UNIX and occasionally, Windows.
That's not a liability on their part (nobody knows everything about
everything), but it does translate, at least in this case, to a liability on
behalf of the concept which emerged.

So now we have (at least, some of us believe, and not just people within my
company) an expressed and rational reason why the concept is flawed, and
why in needs to be reexamined and updated.  The reaction: it's "exotic".
It's not "mainstream".  After all, the basic concept CAN'T be flawed!

>this makes the protocol less suitable for files that are not.

Exactly - and this is the reason the design needs to be enhanced.

>feedback, otherwise the problem won't be solved.

It sounds, by the tone of your recent messages on this subject, that there's
no problem to be solved, as everything outside the original concept for SFTP
is an "exotic" or "outside of mainstream" system, and hence, not worth
supporting.  And along with we of the VMS world, I suspect that those in,
for instance, the mainframe world would disagree with your definition and
labelling of "exotic" and "mainstream".

And hence, my responses to your messages.  We are more than willing to work
to see this issue gets addressed; what we expect is that the chauvinism be
put on the back burner and move on to the problem at hand.  If there are
valid TECHNICAL reasons for something not to be done, so be it.  But to
label things outside of Windows or UNIX as "exotic" and "out of the
mainstream" is counter-productive to getting anything changed.  In other
words, let's work the problem rather than label things you don't see the
case for in a pejorative sense.

The goal here is not to make this a VMS-centric protocol; rather, it's
to ensure that VMS can participate fully in all transactions without
regard to the file format.  And that's simply not the case today, given the
current state of the protocol.

------