Subject: Ftp-WG: NLST vs MLSD

From: Frank da Cruz <[email-masked]>
Date: Tue, 15 Oct 2002 09:04:20 -0500

: Robert Elz <[email-masked]>; Tue, 15 Oct 2002 05:25:39 -0500:
:   | Frank da Cruz <[email-masked]>; Mon, 14 Oct 2002 09:29:55 -0500:
:   |  1. MGET works by getting a list of files from the server and then
:   |     RETR'ing each file.
: 
: MGET is a local invention by clients (anticipated by the FTP spec for sure).
:
This is what bothers me.  Of course what you say is true, but I don't think
it's a good idea to ignore implementation, existing practice, and widespread
human behavior when modifying a key Internet protocol, if the modifications
can affect all that.

:   |  2. But how is the client supposed to know whether it's a directory 
:   |     name when it's in the syntax of the server's file system?
: 
: It defines what is what in the syntax it accepts.
:
How can it?  Either the FTP client knows the file system syntax of the
server host or it doesn't.  In common practice, most clients simply assume
the server to be Unix, which is no basis for an open protocol.  If it
doesn't know the host's syntax, it can't decompose strings like
"[.blah]*.jpg" into CWD and NLST commands (or whatever).

: With a TVFS server on the other hand, the client will be able to make
: more assumptions, but using a (I hope) well defined specification for
: exactly what means what.
: 
This is what we have done in Kermit since 1997 and it works nicely.  TVFS
is a good and appropriate solution to cross-platform file reference and
access.  You can map any flat or tree-structured file system to it.

Obviously other problems can come up too, like password-protected disks or
directories, as in CMS or TOPS-20.  Or conflicts between Space as a text
character (as in Windows) and as a separator (as in CMS).  Or the possibility
that some file system uses "/" in identifiers.  Mapping rules can be
devised, but then of course users must be exposed to them.

In other words, users are always going to have to know something about
the platform on the other end of the connection, even if wildcards were
abolished from the wire.

: I can't actually see any very compelling reason why a command line style,
: traditional FTP client, would ever need to use MLSD...
: ...for your purposes, you can just ignore MLSD, I think, there's no
: requirement that a command line client use it.
:
For recursive downloads.  There's no other way to do it.

:   | It is the business of open standards -- and a founding principal of 
:   | the ARPANET --  to put common intermediate representations on the
:   | wire.
: 
: which is exactly what would be required here.   Wildcards "work" now,
: as much as they do, because everyone knows that they're breaching the
: standards...
:
I used FTP for decades without knowing that.  I don't believe I'm atypical
in that respect.  Some consideration has to be given to existing practice.

: You can also rest upon my IETF experience, when I assure you that there's
: zero chance of getting something published as an RFC which says anything
: like "you send this undefined string, and an undefined list of files gets
: sent back to you".
: 
That's certainly a compelling argument.  Yet it is not applied consistently.
Even without wildcards, we have been sending undefined strings back and
forth forever, and this will continue with MLSD, unless TVFS notation
becomes mandatory.  But if that happened MLSD but not for other commands
(such as CWD), what would be the point?

Again, undefined strings are the "escape valve" that lets users accomplish
what they want when the protocol or software stands in their way.  They let
users to "go over the head" of the client and speak directly to the server.
It's like the big boss having an open door policy.  It's not necessarily a
bad thing.  But of course it should not be the only thing.

Standard wildcard syntax would be nice -- as opposed to regexp syntax,
which is similar but different.  Something akin to C-shell wildcards might
well cover most cases, but with some consideration to the way "." is a hard
separator in most file systems, but not in Unix.  And to the way Unix
treats filenames that start with ".".  And the device-directory
distinction.  And... and...  Anyway, it might turn out that server-side
wildcard interpretation is needed in practice, and if that demands a
standard wildcard notation, then one will emerge.

- Frank

---

From: "Brad Clements" <[email-masked]>
Date: Tue, 15 Oct 2002 09:43:01 -0500

> standard wildcard notation, then one will emerge.

I think this is a reasonable statement.

Would you be willing to write up an "RFC" for "a standard server-side wildcard 
specification"?

Brad Clements,                [email-masked]   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
AOL-IM: BKClements

---

From: Alun Jones <[email-masked]>
Date: Tue, 15 Oct 2002 13:49:10 -0500

At 09:04 AM 10/15/2002, Frank da Cruz wrote:
>This is what bothers me.  Of course what you say is true, but I don't 
>think it's a good idea to ignore implementation, existing practice, and 
>widespread human behavior when modifying a key Internet protocol, if the 
>modifications can affect all that.

To an extent.  MLST / MLSD was specifically supposed to ignore 
implementation, and existing practice, because what is there at present is 
wildly unreliable.  How many versions of "ls -l" are there?  That was what 
MLST and MLSD were supposed to replace - LIST, not NLST - and they were 
supposed to replace the LIST commands in some means other than declaring 
one or other "ls -l" to be the standard.

Apparently, there is already implementation of wildcards in MLSD on two 
servers, and at least one client implementation - the question becomes 
what, if anything, should be done about that?  Should we leave an 
undocumented (and counter-document) assumption in the works, or should we 
perhaps try and re-emphasize that an FTP client "MUST NOT" pass wildcards 
to MLSD?  Sadly, I don't have access at present to a server that implements 
wildcards in MLSD so that I can discover and/or demonstrate the failings 
inherent in unplanned wildcard support in MLSD.  As I mentioned earlier, 
certain assumptions inherent in the MLSD format break as soon as you allow 
wildcards, so MLSD either needs to change to support wildcards, or the use 
of wildcards in MLSD needs to be forbidden.

>"[.blah]*.jpg" into CWD and NLST commands (or whatever).

Without TVFS, if a client user types "MGET foo", the client must "NLST 
foo", and then RETR everything that it sees in response.  The same goes no 
matter what special characters other than "\r\n" are in 'foo'.  A compliant 
MLSD should respond that there is no such directory as "[.blah]*.jpg", 
rather than try and list files matching the description.

>the platform on the other end of the connection, even if wildcards were

There's almost certainly room for a specification that says "this command 
takes a specification that refers to one or more files or directories, and 
lists the matching items in the following format".  MLSD isn't it - so 
perhaps you'd like to write MLSW?

>what they want when the protocol or software stands in their way.  They 
>let users to "go over the head" of the client and speak directly to the 
>server. It's like the big boss having an open door policy.  It's not 
>necessarily a bad thing.  But of course it should not be the only thing.

And, where it is allowed, the results should be as useful as possible.  If 
I ask a wildcard-capable MLSD implementation for "*", am I going to get the 
names of all the files and directories in the current directory, or am I 
going to get the names of all files in the current directory as well as the 
names of all files in every subdirectory?  How, as a client, are you going 
to distinguish the files from one another?  In MLSD as it is currently 
defined, you have no hope.  So, either that support of wild-cards is going 
to be unreliable, or it's going to be to a standard that hasn't been 
documented.  If it's a standard that hasn't been documented, then the 
implementors need to explain it, and how to rely on it.  If it's 
unreliable, then it should be chopped out, and wildcards should be refused 
by the server.

Alun.
~~~~

--
Texas Imperial Software   | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place   | http://www.wftpd.com or email [email-masked]
Cedar Park TX 78613-1419  | VISA/MC accepted.  NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.

---

From: Robert Elz <[email-masked]>
Date: Wed, 16 Oct 2002 05:50:00 -0500

Date:        Tue, 15 Oct 2002 09:04:20 -0500
    From:        Frank da Cruz <[email-masked]>
    Message-ID:  <CMM.0.90.4.1034690042.fdc@watsol>

  | This is what bothers me.  Of course what you say is true, but I don't think
  | it's a good idea to ignore implementation, existing practice, and widespread
  | human behavior when modifying a key Internet protocol, if the modifications
  | can affect all that.

Which modifications are we talking about here?   Aside from the doc
of the ancient commands that were never documented (which as best as we
know are documented the way they're implemented) all that is being done
is creating new commands.

Nothing that currently exists (and works) is being eliminated.  The
draft doesn't say that NLST can't go on accepting wildcards if it
wants to (but nor does it say that it should).

Further, nothing here is mandatory, it is all optional, both for the
servers to implement (though I hope that most will) and for clients
to use (there, I'd hope that only clients that have a need for the
new functions will use them, not just everyone "because it is there")

  | :   |  2. But how is the client supposed to know whether it's a directory 
  | :   |     name when it's in the syntax of the server's file system?
  | : It defines what is what in the syntax it accepts.
  | How can it?

You're missing the point.   What I said is that the client defines the
syntax it accepts.   How can it not?   So, the client (user FTP implementation)
tells its users "when you give me a string that looks like ... I will
treat the yyyy part as a directory name, and the xxxx part as a file name
in that directory).

That's exactly what should always be done.   Expecting every user to
understand every syntax for every file system on the planet is absurd.

  | Either the FTP client knows the file system syntax of the
  | server host or it doesn't.

It doesn't, and shouldn't need to.   What it knows is what the
FTP spec says can be passed, and in 959 that's just a directory
name for CWD, and a file name for RETR (etc).

So, the client provides some syntax for the user to enter file names
and directory names, and it issues the correct commands to achieve the
effect the user has requested.

  | In common practice, most clients simply assume
  | the server to be Unix, which is no basis for an open protocol.

Of course not, that's what we're trying to get away from here.   And
that includes the expectation that the server implements some unix
server idea of what '*' really means.

  | If it doesn't know the host's syntax, it can't decompose strings like
  | "[.blah]*.jpg" into CWD and NLST commands (or whatever).

If the client says that the syntax in which it accepts paths is
	[.directory]filename
then of course it can.   The client is likely to do that if the
native file system on the system it is implemented on uses that
syntax, so the end-user can refer to files via FTP using the same
general syntax as is used to refer to files on disks.

On the other hand, a client on a unix system is likely to expect
its users to input
	/directory/*.jpg
which it (should) implement by doing
	CWD directory
	NLST
		<match files that have names that end in .jpg according
		to local matching policy>
	RETR a.jpg
	RETR b.jpg

(etc).   That the server would have referred to those files as
[.directory'a.jpg etc is completely irrelevant, unknown, and
unimportant.

Any modern client is also going to need to be able to understand
and parse
	ftp://server-name/directory/a.jpg
as well of course.   This provides a universal file nameing format
(which is what URLs and their kin were designed to do of course).

Again, the client turns that into the appropriate CWD and RETR
commands to fetch the file - regardless of the directory+file name
syntax at the server.

If the server happens to support TVFS, then the client gets more
options in the way it passes the file name to the server.

  | Or the possibility
  | that some file system uses "/" in identifiers.  Mapping rules can be
  | devised, but then of course users must be exposed to them.

No they needn't.   Old Mac filesystems (probably still current ones, but
it has been a while) used ':' as the separator, and '/' as just a character.

That means that the mac name volume:profit/loss account
can be (easily) mapped into
	/volume/profit:loss account
and the users don't have to know that at the server there's anything
different.

Of course, when advising someone of a name in the FTP namespace, its
FTP name needs to be given, but that's not all that hard to deal with
(only local users advertising files for FTP need be aware of that).

  | In other words, users are always going to have to know something about
  | the platform on the other end of the connection, even if wildcards were
  | abolished from the wire.

We still don't have solutions to all the possible wild file types, so
some knowledge may be required.   But to just transfer bit images, and
then send them back again, no knowledge of the server should be required.

  | : I can't actually see any very compelling reason why a command line style,
  | : traditional FTP client, would ever need to use MLSD...
  | For recursive downloads.  There's no other way to do it.

Recursive downloads aren't a requirement.   Traditional FTP clients
haven't offered any facility like that (though some have done it, by
LIST parsing I guess).   So, I guess yes, if you want to fetch everything
(or some definable subset of everything) then I suppose MLSD is a
reasonable way to do it.

  | I used FTP for decades without knowing that.

I wasn't talking about users of FTP, but implementors.   The user should
be able to use wildcards - I have no problem with that at all.   The client
should make it all work (just as the unix shell makes it all work for
unix command line users - and work the same way on unix filesystems, NFS
filesystems (that might have almost any semantics imaginable) and even
DOS filesystems (where wildcards typically get interpreted slightly
differently by the native OS).

But to do that, the client doesn't have to do it the stupid way.  And
client implementors should be aware that that way is unreliable, and
unsafe.

  | That's certainly a compelling argument.  Yet it is not applied consistently.

If you mean hasn't been, then yes, I'd agree, almost anything used to
be able to get published.   These days, I suspect it is more
consistent.

  | Even without wildcards, we have been sending undefined strings back and
  | forth forever, and this will continue with MLSD, unless TVFS notation
  | becomes mandatory.

No, the strings there aren't undefined.   Or not in the same way.   The
string given to RETR (etc) is a file name valid on the server.  True we
don't know precisely what is legal there, but we know what it means, it
is a file name.   And when we issue that name, the file named will be
retrieved, or an error will occur.

If you allow wildcards, without specifying what they mean, then you have
no idea what will happen.

That is, with MLSW assuming it is created, if I do MLSW * what happens?
With no spec for what is a wildcard, and what is just a character, I
might get a listing of the file/directory named '*' (or an error if there
isn't one), or I might get a listing of all files with no "extensions"
or I might get a list of everything (possibly even a list of everything
including everything).   I have no clue at all what will happen.  I don't
know whether '*' is a magic character or not.

  | But if that happened MLSD but not for other commands
  | (such as CWD), what would be the point?

If TVFS is implemented, it is implemented for every command that
accepts a file name.

  | Again, undefined strings are the "escape valve" that lets users accomplish
  | what they want when the protocol or software stands in their way.  They let
  | users to "go over the head" of the client and speak directly to the server.

But that is useless.   What would your client do if when you issue

	RETR abc.

The server just decided that names ending in '.' mean that you have
abbreviated the name (etc.) and what you're asking for is for every
file whose name starts with abc to be sent down the wire.

There are sometimes good reasons to allow the user to bypass what the
client things is correct, and talk to the server directly, but the
client should at least be aware that's what the user is doing, and not
be ambushed by something weird that the server implementor, and human
client user just happen to have agreed will be treated strangely.

  | Anyway, it might turn out that server-side
  | wildcard interpretation is needed in practice, and if that demands a
  | standard wildcard notation, then one will emerge.

Hmm...   One hasn't (really) emerged after all these years.
That suggests that there is no demand for a standard notation,
and that suggests that server side wildcard isn't really needed
in practice...

kre