Subject: Ftp-WG: NLST vs MLSD

From: Frank da Cruz <[email-masked]>
Date: Tue, 15 Oct 2002 15:46:05 -0500

Alun Jones <[email-masked]>; Tue, 15 Oct 2002 13:49:10 -0500:

: At 09:04 AM 10/15/2002, Frank da Cruz wrote:
: >This is what bothers me.  Of course what you say is true, but I don't 
: >think it's a good idea to ignore implementation, existing practice, and 
: >widespread human behavior when modifying a key Internet protocol, if the 
: >modifications can affect all that.
: 
: To an extent.  MLST / MLSD was specifically supposed to ignore 
: implementation, and existing practice, because what is there at present
: is wildly unreliable.
:
From a theoretical standpoint, perhaps.  But people have been using
"mget <wildcard>" since time immemorial.  As far as I know, they are not
complaining.  It does not seem to be a problem in search of a solution
(and "if it's not broken...")

: How many versions of "ls -l" are there?
:
Personally, I would never use LIST as a basis for MGET.  I would never
even dream of parsing host-dependent directory listings.  I would rather
use NLST to get the list of names, and then get whatever info about each
file I could from the server by sending SIZE, MDTM, etc.  But that is so
gross that MLSD is far more attractive for this purpose.

: That was what MLST and MLSD were supposed to replace - LIST, not NLST...
:
It would be silly to say MLSD, when available, can't replace NLST.  If it's
there, it will be used.  Again, I agree MLSD is a fine LIST replacement,
but (and this time let me set this off for emphasis)...:

  --> MLSD is the only way to accomplish recursive MGET <---

: Apparently, there is already implementation of wildcards in MLSD on two 
: servers, and at least one client implementation - the question becomes 
: what, if anything, should be done about that?
:
I think we should consider MLSD already specified except for some details,
like the repertoire of facts and the ordering of the list.  No wildcards, no
filenames; the argument can be empty, or a directory name.  (But you might
want to say a bit more about exactly what format the directory name be in --
anything accepted by the host?  TVFS?  Relative but not absolute?... Can I
send "MLSD somedisk:[dir1.dir2.dir3]" to a VMS server?)

: Sadly, I don't have access at present to a server that implements 
: wildcards in MLSD...
:
There's a publicly accessible one at ftp.ipswitch.com (Robert says they
will be fixing it to comply with the draft, but the last time I looked a
few days ago, it still interpreted wildcards).

: >... Either the FTP client knows the file system syntax of the server
: >host or it doesn't.  In common practice, most clients simply assume
: >the server to be Unix, which is no basis for an open protocol.  If it
: >doesn't know the host's syntax, it can't decompose strings like
: >"[.blah]*.jpg" into CWD and NLST commands (or whatever).
: 
: Without TVFS, if a client user types "MGET foo", the client must "NLST 
: foo", and then RETR everything that it sees in response.
:
Whoa!  In other words you want to say that MLSD "MUST NOT" be used to
implement MGET.  See what I said about that just above, especially the
emphasized part.

: There's almost certainly room for a specification that says "this command 
: takes a specification that refers to one or more files or directories, and 
: lists the matching items in the following format".  MLSD isn't it - so 
: perhaps you'd like to write MLSW?
: 
I'd *like* to :-)  And I'd also like to write MLSR and MLSX.  In other words
all four combinations of who-interprets-wildcards versus whether-to-recurse.

I'd also be willing write a proposal for a standard wildcard syntax if
nobody else wants to take it on.

I'd also like to keep my job, and those of my group.  At the moment this is
enough of an issue that I can't commit to doing anything useful within a
specific time frame, as I normally would do.  In any case, I would like to
see at least some allusions to these issues in the draft so implementors
have some background, guidance, and perhaps an inkling of "as-yet unresolved
issues".

- Frank

---

From: Alun Jones <[email-masked]>
Date: Tue, 15 Oct 2002 17:12:13 -0500

>(and "if it's not broken...")

Perhaps you misunderstood.  MLST/MLSD is a replacement for LIST.  Why you 
think I was commenting on "mget <wildcard>", I can't imagine.  I'm not 
DJB.  What I was saying is grossly unreliable, and that MLSD hopes to fix, 
is the myriad of different formats returned by LIST - the format changes 
from one installation to the next (is your server running GNU's 
'ls'?  BSD's?  Whose?  Its own attempt to reproduce one of these?)

mget, as you note, does seem to work with wildcards, in part because 
everyone copies Unix's format.  However, even that's not exactly reliable, 
and many a server author comes across issues where his emulation of Unix's 
'ls' is not quite the same as the 'real thing'.  "mget *" versus "mget *.*" 
caused me quite a fuss in my early, Win16, days, when I insisted on 
interpreting the wildcards as DOS-style wildcards.  Similarly with "mget 
*/*", etc.  So, yes, people are complaining, both when the wildcard spec 
matches that of the server's operating system, and when the wildcard spec 
is forced to match Unix's.

However, as you say, it works for the most part, and doesn't need 
fixing.  Why are you so intent to use MLSD as a replacement for NLST?

>gross that MLSD is far more attractive for this purpose.

A step partway between the two appears more appropriate.  NLST, followed by 
MLST on each file that you are going to get, so that you can set the 
modification time, display progress information, etc, as you download.

>   --> MLSD is the only way to accomplish recursive MGET <---

And a recursive MGET is not going to benefit from server-side expansion of 
wildcards, unless I'm missing something obvious.  "MGET/RECURSE *.txt" 
isn't going to pass "MLSD *.txt" and find any directories to recurse.  The 
only way to get a listing of every directory that you'll be wanting to 
recurse into is to call MLSD with a parameter that allows you to list every 
directory within your source.  In other words, you're just going to be 
passing in a directory name to list, which is what MLSD, as it is 
documented, expects to see.

>like the repertoire of facts and the ordering of the list.  No wildcards, 
>no filenames; the argument can be empty, or a directory name.  (But you 
>might want to say a bit more about exactly what format the directory name 
>be in -- anything accepted by the host?  TVFS?  Relative but not 
>absolute?... Can I send "MLSD somedisk:[dir1.dir2.dir3]" to a VMS server?)

If the host doesn't support TVFS, you should be sending local format 
filespecs.  That's fairly simple.  As to what's displayed in the output 
from MLSD, it's not supposed to be read by the client and concatenated with 
anything else, so the server may choose to display only local format, only 
TVFS, or a combination of both.  Of course, what you can't say (and only 
the server knows to prevent) is "MLSD somedisk:[dir1...]" to a VMS server, 
because that represents more than one directory.

>few days ago, it still interpreted wildcards).

Do you have a Unix one to point to?  This one sidesteps the problem by 
matching wildcards only against the current directory - you can't do "MLSD 
*/*.txt" or similar atrocities.

>implement MGET.  See what I said about that just above, especially the

I think that NLST is adequate for MGET.  You want to use MLSD, and you want 
to use wildcards in MGET.  MLSD doesn't do wildcards, therefore is not 
appropriate for use in expanding wildcards for MGET; therefore it makes no 
sense to use MLSD for MGET.

>I'd *like* to :-)  And I'd also like to write MLSR and MLSX.  In other 
>words all four combinations of who-interprets-wildcards versus 
>whether-to-recurse.

Given the examples in the MLST draft refer to MLST / MLSD as "MLSx", 
perhaps that should be changed, or your choice of name should be revisited.

>I'd also like to keep my job, and those of my group.  At the moment this 
>is enough of an issue that I can't commit to doing anything useful within 
>a specific time frame, as I normally would do.  In any case, I would like 
>to see at least some allusions to these issues in the draft so 
>implementors have some background, guidance, and perhaps an inkling of 
>"as-yet unresolved issues".

We've all got day-jobs.  Some of us have night-jobs, too, and raise 
families.  It seems to me, and I know this may sound rude, that you're the 
one facing the problem.  This group loves to take drafts and knock out the 
bad bits, but nobody wants to write the draft.  I think a draft is 
required, if you want this problem to be solved, and it appears that you're 
the one interested in writing it.

Alun.
~~~~

--
Texas Imperial Software   | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place   | http://www.wftpd.com or email [email-masked]
Cedar Park TX 78613-1419  | VISA/MC accepted.  NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.

---

From: Alun Jones <[email-masked]>
Date: Tue, 15 Oct 2002 17:12:29 -0500

Alun.
~~~~

---

From: Alun Jones <[email-masked]>
Date: Tue, 15 Oct 2002 17:55:45 -0500

Sorry for the double-post, there - my mail reader puts "you wrote" in the 
attribution line if I don't do "Reply To All", so I've got into the habit 
of hitting "Reply To All" in mailing lists.  What I need to do, of course, 
is get into the habit of deleting the extra copy of the address from the 
headers before sending (goodness only knows why it shoves a copy into "To" 
and "Cc".

Alun.
~~~~