Subject: Ftp-WG: NLST vs MLSD

From: Frank da Cruz <[email-masked]>
Date: Wed, 16 Oct 2002 10:34:28 -0500

OK, I think we're finished here.  At this point we are merely repeating
ourselves.  If I am able, I'll submit a new draft at some future time.

- Frank

---

From: "D. J. Bernstein" <[email-masked]>
Date: Wed, 16 Oct 2002 12:26:42 -0500

Frank da Cruz writes:
> OK, I think we're finished here.

No. We've discovered that there's an interoperability failure between
your client and some servers. The servers are complying with RFC 959,
so the fault lies with your client. Your client has to be fixed.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: Alun Jones <[email-masked]>
Date: Wed, 16 Oct 2002 13:11:36 -0500

>so the fault lies with your client. Your client has to be fixed.

Actually, it seems that server _and_ client are complying with RFC 959.

The client, in good faith, is passing what it believes is a "directory or 
other system-specific file group descriptor" given by its user, to the 
server.  The server sees an NLST command that holds such a "directory or 
other system-specific file group descriptor", and responds with a set of 
file names.  The client parses through each file name, and fetches them one 
by one.

RFC 959 does not specify that the "system-specific file group descriptor" 
is only to mean a folder, directory, or other containing unit.  It does not 
specify that it cannot be a wildcard.  "*.txt" would seem to be a valid 
"system-specific file group descriptor", describing, as it does, a group of 
files in a system-specific manner.  The original author may have intended 
it to mean "directory" only, but then why say "directory _or_ 
system-specific file group descriptor"?  Sadly, at least one of the authors 
of RFC 959 is beyond our questions.

Certainly it's risky passing a wildcard to the server - the risk is that 
the server says something akin to "I don't know what you're talking about", 
and nothing is transferred.  But I don't think it's counter to RFC 959.

If anything needs to be fixed to be compliant with RFC 959 (and I don't 
think it does), it's the user who thinks he can pass a wildcard spec in to 
his FTP client, and have it passed onto, and interpreted by, his server.

Alun.
~~~~

--
Texas Imperial Software   | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place   | http://www.wftpd.com or email [email-masked]
Cedar Park TX 78613-1419  | VISA/MC accepted.  NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.

---

From: Alun Jones <[email-masked]>
Date: Wed, 16 Oct 2002 13:11:37 -0500

Alun.
~~~~

---

From: "D. J. Bernstein" <[email-masked]>
Date: Wed, 16 Oct 2002 17:15:04 -0500

Alun Jones writes:
> other system-specific file group descriptor" given by its user

Incorrect.

Perhaps you're not familiar with the programs in question. We're talking
about the ``mget'' command supported by the BSD ftp program and its
imitators. The mget argument is a wildcard, not a directory name.

Or perhaps you're confused by the phrase ``other system-specific file
group descriptor'' in RFC 959. That's simply recognizing the fact that
some systems back then had other names for directories: disks, mounts,
etc. You would have already realized that this doesn't include wildcards
if you had read RFC 959 more carefully: the same phrase is used for CWD.

Bottom line: Certain client implementors have led the users to believe
that mget * will retrieve all files in the current directory, but they
are not implementing this function correctly.

---

From: Alun Jones <[email-masked]>
Date: Wed, 16 Oct 2002 18:19:05 -0500

>imitators. The mget argument is a wildcard, not a directory name.

The client has no way of knowing that.  The client takes what is passed as 
a parameter to mget by the user, and sends it exactly in NLST.  Then it 
takes the response and gets each file.  You seem to think that the client 
somehow "knows" that the user's parameter is flying in the face of RFC 
959.  It knows no such thing.

>if you had read RFC 959 more carefully: the same phrase is used for CWD.

I've also taken it on previous occasions to mean other, more exotic, 
things, such as zip files or other archives (really, what's the difference 
between one of those and a directory?).  A wild-card specification is no 
more or less valid, and a sufficiently perverse mindset could create an 
implementation wherein "CWD *.txt" might allow you to list/access only 
files containing ".txt" in their name.  It _is_ a descriptor for a group of 
files.

>are not implementing this function correctly.

Perhaps you can point me to the documentation that has led these users down 
the garden path.  One could just as easily say that certain server 
implementors have done this, by accepting and expanding wild cards.

Well, whatever, it's clear that there's a whole mess caused by the 
implementation of wild-card support in an environment that wasn't designed 
for that purpose.  MLSD specifically forbids wild-cards for just that 
reason, it seems.  However, there's no reason why an MLSW command couldn't 
be specified, that leaves the topic of wild-card expansion up to the 
server.  Users aren't all that confused by existing wild-card techniques, 
and they won't be all that confused by an MLSW command being used for mget 
in the future, as long as the command is designed specifically to address 
that need.

Alun.
~~~~

---

From: "D. J. Bernstein" <[email-masked]>
Date: Sat, 19 Oct 2002 14:48:27 -0500

> However, there's no reason why an MLSW command couldn't be specified,

It's awful protocol design. Here's the picture:

   Protocol   Servers           Clients without mget   Clients with mget
   --------   ---------         --------------------   ---------------------
   standard   no effort         no effort              parse wildcards, loop
   fantasy    parse wildcards   no effort              loop
   your mix   parse wildcards   no effort              parse wildcards, loop

The first line is how the standard FTP protocol works. By moving the
wildcard handling as close as possible to the user, it puts the burden
on the smallest number of programs---the programs that actually care.

The second line is how some confused implementors think FTP works. It
puts a burden on every server, including servers that would otherwise be
substantially simpler, for the benefit of some clients that have to be
fairly complicated anyway.

The third line, your MLSW proposal, is absurd. It has all the burdens of
the first two lines without any of the benefits.

> > The mget argument is a wildcard, not a directory name.
> The client has no way of knowing that.

The client's mget documentation states explicitly that wildcards are
allowed. RTFM.

> the garden path.

The Kermit FTP client mget documentation, for example, explicitly allows
wildcards. Its implementation of this feature fails to interoperate with
some compliant servers.