Subject: Ftp-WG: NLST vs MLSD

From: Frank da Cruz <[email-masked]>
Date: Mon, 14 Oct 2002 09:29:55 -0500

Alun Jones <[email-masked]>, Sun, 13 Oct 2002 16:53:08 -0500:
: At 01:26 PM 10/13/2002, Frank da Cruz wrote:
: >That the MLSD spec does not permit filenames or mixtures of paths and
: >filenames in its argument; thus the user is not allowed to send this
: >argument to the server in an MLSD command.  Note that I don't advocate
: >changing MLSD in this respect.  But from the user's point of view, a
: >command that once worked will no longer work and this will cause confusion,
: 
: Why?  You've given no indication as to why MGET - the "command that once 
: worked" should use anything other than NLST.  Hence, MGET will work exactly 
: the same way it does now.
: 
I guess I need to spell it out:

 1. MGET works by getting a list of files from the server and then
    RETR'ing each file.

 2. New clients will want to use MLSD because of its enhanced list format.
    
 3. For backwards compatibility, clients will want to continue to have
    an MGET command with the same syntax as before.

What is the new client supposed to do with an argument like "pics/m*.jpg"?

 1. It can't include it as an MLSD argument because MLSD accepts only
    directory names.

 2. But how is the client supposed to know whether it's a directory name
    when it's in the syntax of the server's file system?

      pics/m*.jpg      (UNIX)
      pics:m+.jpg      (AOS/VS)
      pics\m*.jpg      (DOS)
      pics>m*.jpg      (VOS)
      m* jpg 197       (CMS)
      [.pics]m*.jpg    (VMS) 
      pics:m*.jpg      (also VMS)
      <.pics>m*.jpg    (TOPS-20) 
      [123,456]m*.jpg  (TOPS-10)

: >An operation that was possible before is suddenly impossible.
: 
: Only if you insist that MGET must use MLSD.  Why not stick with NLST, as 
: you are today?  MLST / MLSD replaces only one (inappropriate) use of LIST, 
: not NLST.
: 
If MLSD is not for use with MGET then what is it for?  It is not a LIST
replacement.  LIST is for people, MLSD is for machines.

I don't insist that MGET use only MLSD -- please go back and read:

  http://www.columbia.edu/kermit/newftp.html

Sometimes NLST is better, sometimes MLSD is better.  Thus a client can try
to choose the best FTP command in each situation.  But it can't always be
right when the choice depends on the server's file system syntax, which the
client can not be expected to know unless we reduce the world to Unix and
Windows.  Thus the user must be exposed to the details and tradeoffs of NLST
versus MLSD.

: >An operation that was possible before is suddenly impossible.
:
Anyway you took this statement out of context.  It applies to the situation
in which the client expects the server to expand wildcards (as virtually
every client does) but the server interprets RFC959 strictly and does not
accpet wildcards.

: >If a text file is sent in binary mode to a radically different platform,
: >the result is garbage (and vice-versa, of course).
: 
: Incorrect.  If a text file is sent in binary mode, the result may be a file 
: where the line-ends are not in the local format, but the file is far from 
: being garbage.  Tools exist even on DOS - very simple tools - to convert 
: "Mac" and "Unix" style line ends to "DOS" style, and vice-versa.
: 
If you restrict yourself to Unix and DOS that might be true.  Once you
allow for other platforms you find:

 . Variable-length record files with embedded record-length fields in
   a variety of formats (binary, ASCII numeric, packed decimal) and 
   having different lengths (2 bytes, 4 bytes, ...)

 . Fixed-length records.

 . EBCDIC instead of ASCII (and remember there isn't just one EBCDIC,
   there are hundreds of them).

 . Some proprietary character set representing a non-English language.

If you choose to ignore all but Unix and DOS file systems, you still can't
dismiss line-end conflicts -- these affect real users every day, who don't
have a clue what to do about them.

It is the business of open standards -- and a founding principal of the
ARPANET --  to put common intermediate representations on the wire.  Once
you start going down the road you're suggesting, you might as well abandon
the Internet to a single vendor.

For the purposes of cross-platform file interchange, the world has settled
on distinguishing between text and binary files.  It's not ideal but taking
this concept further (e.g. converting database formats between one product
and another) are beyond the scope of open standards.  Thus text files are
candidates for conversion to common format for transmission -- and that
means both record format AND character set -- and binary files are
transmitted without conversion.  We have decades of experience to tell
us which mode of transmission is appropriate for which file.

: >We're going in circles.  The topic here was recursive downloads.
: 
: No.  The topic was wildcards in MLSD.  There are wildcards that will 
: produce matches that list the contents of one or more directories; there 
: are wildcards that match some files and some directories.  If MLSD is to 
: match wildcards, it must be done reliably, and in a manner that can be 
: replicated by client and server.
: 
Again, I'm not suggesting that MLSD accept wildcards; rather I'm proposing
that MLSD-like commands be added that do so.

Wildcards exist.  As you pointed out, RFC959 deals with them, but in a way
that has been rejected by virtually every developer.  I agree it would be
nice to have standard wildcard syntax and semantics -- but it would also be
a large undertaking.  As matters stand, server-side wildcard interpretation
offers an "escape valve" to users who know what they are doing, that lets
them accomplish effects that could not be achieved otherwise.  In a word,
server-side wildcards can be "useful".

: >: Why a fact for MODE?
: >:
: >Why not?  If FTP has a MODE command with a selection of values, and if,
: >in a certain setting, the value makes a difference, then it would be
: >useful for the server to suggest the appropriate MODE for a given file
: >(if the server knows it) so the user doesn't receive garbage.
: 
: Because every mode can be used to transfer any file.  Hence, MODE is 
: entirely the choice of the client.
: 
Right, but the server can suggest the most appropriate one, since it might
know something the client doesn't.  Stream mode is appropriate for stream
files, block mode for structured files.

: >Years ago I, too, thought inspection was gross.  Then I tried it.  In
: >Unix and Windows, the effect of looking at (say) the first 48K of a file
: >is barely measurable (unless it's on a floppy disk or something).
: 
: Are you the same Frank that was proposing the server manage a million files 
: in a directory?
: 
I'm the same Frank that has discovered that certain ideas have value in
certain situations.  Situations can exist where the benefits of the server
offering useful information about a file outweigh the performance penalties.
It is the users of FTP who should make the choice about the tradeoffs based
on their needs at the time; it's not our business to make those decisions for
them based on assumptions like "performance has precedence over utility."

- Frank

---

From: "D. J. Bernstein" <[email-masked]>
Date: Tue, 15 Oct 2002 00:13:16 -0500

>     when it's in the syntax of the server's file system?

Normal people use _one_ client on _one_ operating system to talk to a
variety of servers. They find client-defined wildcard syntax stable and
comprehensible, while server-defined wildcard syntax is a royal mess.

People who bounce among clients have much bigger interface issues to
deal with than the wildcard syntax.

> What is the new client supposed to do with an argument like "pics/m*.jpg"?

CWD pics and then NLST. This is what ftpfs does. The code is trivial,
because ftpfs works with the wildcard handling in the client OS.

> replacement.  LIST is for people, MLSD is for machines.

You are, once again, confusing the client-server communication with the
user-client communication.

> As matters stand, server-side wildcard interpretation
> them accomplish effects that could not be achieved otherwise.

You are, once again, falsely claiming that something is impossible, when
the truth is merely that your broken client doesn't do it. Solution: Fix
your client.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: Robert Elz <[email-masked]>
Date: Tue, 15 Oct 2002 05:25:39 -0500

Date:        Mon, 14 Oct 2002 09:29:55 -0500
    From:        Frank da Cruz <[email-masked]>
    Message-ID:  <CMM.0.90.4.1034605177.fdc@watsol>

  |  1. MGET works by getting a list of files from the server and then
  |     RETR'ing each file.

MGET is a local invention by clients (anticipated by the FTP spec for sure).
How it works is up to the client.   What you have described is the essence,
without the detail.   That is, it should get a list of files, pick the
ones it wants, and RETR each of those.   In fact, that's what the typical
unix client does (with the human doing the picking).

There's also often that broken wildcard nonsense, that works by chance
(what's more, it can be useful, I often do "mget *" which fetches all
the files in all direct subdirectories of the current directory, but
no more than that ... but I know I'm abusing the hell out of the spec
and relying on details of a dumb server implementation to make that work).

But in any case, that is just additional filtering, the client still
needs to be able to filter the list provided by the server.

  |  2. New clients will want to use MLSD because of its enhanced list format.

They might, but that isn't what MLSD was intended for,

  |  2. But how is the client supposed to know whether it's a directory name
  |     when it's in the syntax of the server's file system?

It defines what is what in the syntax it accepts.   That's all FTP has
ever really allowed.   If it doesn't define it itself, then it has to
expose the "change directory and refer to file names only" model that
the protocol exposes.

With a TVFS server on the other hand, the client will be able to make
more assumptions, but using a (I hope) well defined specification for
exactly what means what.

  | If MLSD is not for use with MGET then what is it for?  It is not a LIST
  | replacement.  LIST is for people, MLSD is for machines.

It is a LIST replacement.   That was certainly its intent.   It is a LIST
replacement for when LIST output is needed by machines.   That is, to
provide a method other than "send LIST, hope the return format is ls -l
style, or one of a few others I believe I understand, and parse it"
that many programs engage in (mirror programs, GUI interfaces, and such).

It was never really intended as a replacement for NLST.   Though, I do
see how it can be used that way, the way your web page sets out (to
save the overhead of extra commands - particularly SIZE which should
never be used for cosmetic purposes).  But if you're going to use it that
way, you really have to use it within its definition.

I can't actually see any very compelling reason why a command line style,
traditional FTP client, would ever need to use MLSD (I might have it do
MLST to get size, mod time, ... before a RETR in one command/response).
But GUI clients, that like to show users the entire directory, and let
them click on files, are likely to use MLSD lots (rather than parsing LIST
output which is what they currently do).

  | But it can't always be
  | right when the choice depends on the server's file system syntax,

Best is to hide the server's file system syntax from users.  De facto
that's happened already, everyone now exposes FTP path names as URLs
rather than other schemes (then they work in browsers, and other places,
not only in (more or less) dedicated FTP clients).

  | which the client can not be expected to know unless we reduce the
  | world to Unix and Windows.

It is precisely because we can't do that, and the client can't know
either, that we have protocols.   Those are the agreed conventions
that everyone agrees to play by, so the client and the server can
get things done, without either having to know anything much about
the details of the other.

For that to work, you really have to restrict yourself to playing
within the rules.   You can't just go and take advantage of some
"extension" that some server happens to implement, or you're back
with the problem of having to know which servers do, and which don't.

Just as your implementation can't possibly cope with that (nor could
any other) - nor can the human users, and it is unreasonable to ask
them to have to figure out what all those wild syntaxes mean.

  | Thus the user must be exposed to the details and tradeoffs of NLST
  | versus MLSD.

No, for your purposes, you can just ignore MLSD, I think, there's no
requirement that a command line client use it.   If you want to get to
the ultimate in optimisation, then having all those switches for experts
to use to save a minute or two here and there is fine, but it really isn't
necessary.

  | It is the business of open standards -- and a founding principal of the
  | ARPANET --  to put common intermediate representations on the wire.

Yes, I agree.   That's why we put a single, common, intermediate
representation of the directory listing on the wire.   And why we
don't just send wildcards at the server and hope it interprets them
as we intended.

  | Again, I'm not suggesting that MLSD accept wildcards; rather I'm proposing
  | that MLSD-like commands be added that do so.

Go ahead and propose that - but you're going to have to propose all of the
details as well.   And that includes the definition of the wildcard
syntax (I haven't paid all that much attention, but it has always seemed
to me that the NNTP people are spending more time on defining the wildcard
stuff in their protocol (extenstions) than on the whole rest of the protocol)

It also means defining the result format, which if it is to be compatible
with MLSD, would require that you can handle multiple lines referring to
the same filename, and distinguish them from a different filenames (and
you can't use the unique fact for that, as that is for the underlying file,
nor can you really use, I suspect, textual equivalence of the path name).

  | I agree it would be
  | nice to have standard wildcard syntax and semantics -- but it would also be
  | a large undertaking.

Yes, large.   But not just nice, essential.   Just re-read what you
wrote above...

  | It is the business of open standards -- and a founding principal of the
  | ARPANET --  to put common intermediate representations on the wire.

which is exactly what would be required here.   Wildcards "work" now,
as much as they do, because everyone knows that they're breaching the
standards, and hence whatever they do is OK.   Mostly implementations
just copy what they think some other implementation does, and/or what
they believe to be the common subset that users actually use.  That's
fine as long as no-one can come to you and say "you're supposed to have
implemented this, and you haven't".   If we had a command that passed
wildcards around, that is exactly what would happen.

You can also rest upon my IETF experience, when I assure you that there's
zero chance of getting something published as an RFC which says anything
like "you send this undefined string, and an undefined list of files gets
sent back to you".

kre