Subject: Ftp-WG: NLST vs MLSD

From: Frank da Cruz <[email-masked]>
Date: Thu, 10 Oct 2002 10:25:57 -0500

At Wed, 9 Oct 2002 23:17:50 -0500, Alun wrote:
> Does each MLSD output for a wildcard spec even produce a 'cdir' type?
>
No.

> After all, the wild card isn't going to match a directory, usually, yet
> 7.3.1 of the MLST draft notes that an MLSD response MAY include one or
> more "cdir" type entries, probably because the authors anticipated that
> MLSD's argument would always be a directory that would be listed in that
> 'cdir' entry.  "MLSD *.txt" could produce a line that's of 'cdir' type,
> and whose name definitely doesn't match "*.txt".
> 
> In the instance that an FTP server allows "MLSD */*.txt", it may even be 
> useful to define that a 'cdir' entry should be listed prior to the files 
> that match the wildcard spec in its directory.
>
I suggested this too.  It seems to me that the MLSD spec pushes a lot of
work off on the client, and something doesn't feel right about that.  One
server, many clients -- why should all the clients have to deal with the
fact that the server doesn't order its list in a useful way?

> So that if "*/*.txt" 
> matches "a/1.txt", "a/2.txt" and "b/3.txt", you'd have a 'cdir' line that 
> specified where 'a' is, followed by listings for "1.txt" and "2.txt", then 
> a 'cdir' line for 'b', and a listing for "3.txt".
> 
> This goes against the note in 7.3.1 that the 'cdir' entry can occur 
> anywhere in the MLSD output, because it would require that the 'cdir' line 
> precede all matched elements in that directory.  Would it be worth 
> specifying that all 'cdir' output lines must precede other lines relevant 
> to that directory?
> 
I have been advocating that.  If "cdir" can't be taken as a heading, what's
the point of including it in an MLSD listing if the client can't tell which
entries it applies to?  (The answer is evidently something like: "it can be
used as a heading, in the sense that a GUI might put the cdir name in the
title bar, but it isn't a heading in the sense that it comes at the head of
the output")

> Here's another thought - what should MLSD do if given a wildcard that 
> expands into several names, some of which are directories, others of which 
> are files?  Should it list the details of the directories, or the contents 
> of those directories?
> 
Since there are many possibilities, I think we might want to define a
family of MLSx commands; one to do each thing.  Or perhaps more rationally,
to allow MLSD to be accompanied by options ("switches") because you quickly
get into combinatorial problems if you have to define a separate command
for each combination of options.  Just off the top of my head, I think the
options we need include:

 . Whether the server should interpret wildcards in the MLSD argument.
 . Whether the server should return a recursive listing.

Well, if there are only two options, maybe it's not so bad:

         Interpret 
         Wildcards   Recurse
  MLSD      No         No 
  MLSR      No         Yes
  MLSW      Yes        No
  MLSX      Yes        Yes

So far I have dealt with recursion depth-first, by CWD'ing to any "dir" I
run across, and CDUP'ing when done (recursively).  If I had to deal with
cdir's to recurse, I'm not sure how I'd do it, e.g. to do a breadth-first
traversal (as Robert once suggested, to avoid the too-many-temp-files-open
problem).  After all, what does cdir tell you?  Not much -- usually it's
just "." or "/".  Even if I know that foo is bar's parent, how do I get
back back to foo from some arbitrary place without knowing foo's full path?

In the general case, where a directory is too deep for the straightforward
walk, I think the best approach is having MLSR or MLSX return filenames in
TVFS notation, relative to the server's current directory at the time it
received the MLSR or MLSX command.  Then we need only one temp file and we
don't have to clutter the connection with CWD or CDUP commands or do any
bookkeeping.  Thus perhaps MLSR and MLSX should be defined as returning
only filenames, only in TVFS notation.

As an aside, Kermit protocol has used TVFS-like notation for years to
convey directory structure between client and server, and it works very
nicely.  For example, a VMS server (whose path syntax bears no resemblence
to Unix's) can send a directory tree to a Windows client (Unix-like
directory structure but with different dirseps), switching automatically
between text and binary mode for each file.  The same tree can be uploaded
from Windows to another VMS computer with pretty decent results.  (Obviously
there's a lot I didn't say about preservation of VMS file properties, but
that's irrelevant here.)

But this does raise some interesting possibilities for MLSD facts:

 . Over the past few years, we have proven in the field (with Kermit
   software) that files can be categorized as text or binary for purposes
   of transfer by algorithm with near perfect accuracy.  Of course the
   algorithm varies by platform, but that's as it should be.  We use these
   same techniques in the Kermit FTP client when sending a group of files
   (MPUT).  The client can automatically switch between "ASCII" (text) and
   binary mode for each file.  But what about downloads?  If the server
   could employ these same algorithms, then it could include a
   "text-or-binary" subfact with the "type" fact, similar to the MIME
   Content-Type value.  This would be a hint to the client to use the
   appropriate mode for RETR, so as to avoid garbaging the file.
   (Just a hint -- of course the client can do as it wishes.)

 . For cross-platform (or even same-platform) directory-tree duplication,
   it might be desirable to have a standard notation for file permissions,
   as distinct from the "perm" fact, which applies only to the current
   user of the FTP connection.  Kermit protocol has this and it works out
   nicely -- for example, we set the execute bit automatically when
   receiving executable files, and we don't open up private files to world
   access unintentionally just by transferring them to another computer.

> >In any case, my concern is that the new spec conflicts with existing
> >practice.  For decades it has been possible for FTP clients to execute
> >commands like:
> >
> >   mget pics/mae-west-*.jpg
> >   ...(etc)
> >   mget pics/*/mae-west-*.jpg
> >
> >and get at least a semi-expected result.  This is about to change.
> 
> Hardly - NLST isn't going to go away.  It is still expected that NLST will 
> be used for MGET.  What you're hoping to do is to _use_ MLSD to get facts 
> about those files before transferring them, so that you don't have to do a 
> SIZE, MDTM, etc, for each file that you transfer (you could just do an 
> MLST, but I can see why you'd like an MLSD with wildcards).
> 
My point was really this: the use of NLST versus MLSD should be transparent
to the user.  Thus the client software would pick the best method in each
situation, according to the tradeoffs.  But as the spec currently stands,
there is no way to do that without risk.  Thus the hideously complicated and
obscure user interface described here:

  http://www.columbia.edu/kermit/newftp.html

to allow the user to get out of trouble when the client's choices are not
what the user needs.

> >That's a topic for another day.  For now, there is no standard wildcard
> >format, not even within one platform (with a few exceptions).  In Unix,
> >for example, every program that expands wildcards does it its own way,
> >either with its own custom code, or by calling some nonstandard library
> >or other.
> 
> Actually, to judge from the recent problems with "../*/../*" overloads, 
> they mostly use glob() from libc.
> 
Glob is not universal, nor is ftw, nftw, nor fts, nor, for that matter,
Unix.  Internet standards are platform-independent and must not assume
anything about what platforms are in play at the client or server end.

By the way, since there is so much confusion over what NLST should do, I
think there should be statement of clarification, to the effect that "in
view of widespread, if not universal, current practice..." NLST is
expected to expand wilcards and return a list of the files that match.

And that wildcard syntax and semantics depend on the software (client,
server, libraries, OS, whatever) -- there is no standard, no universal
syntax.  Unless we want to tackle that one too.  Personally, I think it's
better to "let the 100 flowers bloom" in this area.  Or at least let
sleeping dogs lie.

- Frank

---

From: Robert Elz <[email-masked]>
Date: Sat, 12 Oct 2002 10:31:40 -0500

Date:        Thu, 10 Oct 2002 10:25:57 -0500
    From:        Frank da Cruz <[email-masked]>
    Message-ID:  <CMM.0.90.4.1034262924.fdc@watsol>

  | At Wed, 9 Oct 2002 23:17:50 -0500, Alun wrote:
  | > Does each MLSD output for a wildcard spec even produce a 'cdir' type?
  | >
  | No.

You mean it wouldn't.   As defined, MLSD is only for directories, the 'D'
there is definitely "directory".

  | It seems to me that the MLSD spec pushes a lot of work off on the client,

Absolutely.   Which is where it ought to be.   As a client author I know
that means more work for you, but I'd much rather have you do the work,
than have server implementations doing more work all the time.   Busy servers
already have too much to do, many would like to be able to offer more, but
are already out of processing resources.   On the other hand, most of the
time, the client is just spinning, waiting for a file to transfer.

  | One
  | server, many clients -- why should all the clients have to deal with the
  | fact that the server doesn't order its list in a useful way?

One reason is because there's no definition of what is "useful".   I expect
listings to be ordered by file size, by modification time, by file "type",
by name, ...   The server has no idea what will be useful, so either it has
to be told, in which case it has to do the ordering every time - and in that
scenario, just as above, it is better for the client to do it.   Or it
simply has to decide "this is the correct order", in which case clients need
the ability to reorder anyway, to get the order that the user prefers.

  | If "cdir" can't be taken as a heading, what's
  | the point of including it in an MLSD listing if the client can't tell which
  | entries it applies to?

It applies to all of the entries, by definition.   The output is a listing
of the contents of one directory, cdir gives the name of that directory.

  | (The answer is evidently something like: "it can be
  | used as a heading, in the sense that a GUI might put the cdir name in the
  | title bar, but it isn't a heading in the sense that it comes at the head of
  | the output")

No, it isn't a heading at all.   It (when TVFS is supported anyway) can be
used to construct a path to a file in the directory, from the current working
directory of the (instance of) the server.

That is, it supplies information useful for naming the files listed.

Sure, some GUI clients might decide that they can use it in a title bar, or
something similar, but that's a by product, not the purpose.

cdir not coming first is just because that's another constraint upon the
server, which in some odd cisumstance, might mean that the server would need
to generate the listing, beore transmitting any of it.   Servers are busy.
The less restrictions we impose upon them, the better.

  |  . Whether the server should interpret wildcards in the MLSD argument.

Not just whether, but also what he wildcards actually mean, otherwise your
client is going to have to interpret the user's spec, and re-code it into
some standard wildcard language.

Personally, I believe that wildcards have no place at all in the FTP spec.
They will cause far more problems than they'd ever solve.   All we need is
for clients to start acting more reaosnably.

  |  . Whether the server should return a recursive listing.

If this was to exist, it would certainly want to be a new command, rather
than a switch, as the output format is necessarily going to need to differ
(so it is possible to distinguish one directory from the next).

Note that you cannot simply use a cdir line, require it to come first, and
define that as the beginning of a new directory, or you would be unable to
distinguish the case of a directory having multiple cdir lines (which is a
very reasonable thing to do - giving both a fully qualified path, and a 
relative path from the PWD) from an empty directory, followed by another one.

  | After all, what does cdir tell you?  Not much -- usually it's
  | just "." or "/".

If you CWD first, and just "MLSD" all the time, then yes, "." is quite
often going to be the result.   But if you give an arg to MLSD, so you're
referring to some other directory, then "." would simply be wrong (well,
assuming traditional unix type semantics anyway, there's nothing in the MLSx
spec that requires this, "." could easily be the name of some sub-directory
of the current directory).

  | Even if I know that foo is bar's parent, how do I get
  | back back to foo from some arbitrary place without knowing foo's full path?

There's no guaranteed way of doing that, other than establishing a new
connection, in FTP.   CDUP is optional.   MLSx type pdir is optional.
FTP servers aren't required to provide you with a way to get back out of
a directory they have let you into.   Once you're there, you can be confined
there.

When TVFS exists (which is probably going to be most of the time), that's
why it makes more sense to construct paths, and use them, rather than CWD
around all the time.

This is true regardless of the MLSx spec.

  | In the general case, where a directory is too deep for the straightforward
  | walk, I think the best approach is having MLSR or MLSX return filenames in
  | TVFS notation, relative to the server's current directory at the time it
  | received the MLSR or MLSX command.

That would be an option where TVFS exists.   But what do we do with the
servers that can't support TVFS?

  | Then we need only one temp file

You only need one temp file anyway.   You can easily seek around in that
as you need to.   Sure, seek pointers (and an accompanying size) are a little
more overhead for the client, than a file descriptor, but they're much less
overhead for the system that is running the client.

  |    But what about downloads?  If the server
  |    could employ these same algorithms, then it could include a
  |    "text-or-binary" subfact with the "type" fact, similar to the MIME
  |    Content-Type value.  This would be a hint to the client to use the
  |    appropriate mode for RETR, so as to avoid garbaging the file.
  |    (Just a hint -- of course the client can do as it wishes.)

Sounds reasonable to me, write a definition for the fact type....  Once
this draft gets published (eventually) we will have a fairly standardised
method of adding new facts.   Certainly it was never imagined that the
set that is in the doc will cover everything (even if we are still hanging
onto the completely useless, as far as I can tell, create-time ...)

  |  . For cross-platform (or even same-platform) directory-tree duplication,
  |    it might be desirable to have a standard notation for file permissions,
  |    as distinct from the "perm" fact, which applies only to the current
  |    user of the FTP connection.

Im not sure how exactly that will work, as on almost all systems, the
permissions vary from user to user.   To describe anything more general
than the current user, you need to be able to identify users in some way.
And that then means some kind of general user namespace management problem.

  |    Kermit protocol has this and it works out
  |    nicely -- for example, we set the execute bit automatically when

Yes, "executable" is perhaps one thing that is currently missing, though
it is not clear to me whether executable is a permission, or a file type,
or both.

We could perhaps do with another fact type, perhaps attributes, to describe
things that aren't really permissions, and aren't really types either.  The
binary/ascii file transfer mode hint, whether the file is an executable one
(which is not the same as "the current user should be allowed to execute it")
etc, could perhaps fit there.

  | My point was really this: the use of NLST versus MLSD should be transparent
  | to the user.

To the user, yes.   Not to the user's software though.

  | Thus the client software would pick the best method in each
  | situation, according to the tradeoffs.

Yes.

  | But as the spec currently stands,
  | there is no way to do that without risk.

There is no way at all to use NLST the way you do, at all, without risk.
You have just decided that the risk is low enough that you have decided
to ignore it.   As server load gets higher, I fully expect to see servers
start looking at the spec, seeing that they're not required to handle
wildcards, and simply stop doing that (just as many that used to do on the
fly gzip or ungzip, no longer do, because it just costs too much for the
server, regardless of how much it would save the network or the client).

  | Thus the hideously complicated and
  | obscure user interface described here:
  | 
  |   http://www.columbia.edu/kermit/newftp.html
  | 
  | to allow the user to get out of trouble when the client's choices are not
  | what the user needs.

This is all because your'e attempting to optimise things which very
rarely ever need optimising.   Just fetching the directory listing, and
doing your own wildcard processing would work fine in just about every
case.   It is likely to make your code bigger, and more complex, but
someone has to do all the work, and the client is the place to do it,
not the server.

  | Glob is not universal, nor is ftw, nftw, nor fts, nor, for that matter,
  | Unix.  Internet standards are platform-independent and must not assume
  | anything about what platforms are in play at the client or server end.

Absolutely.   So, when I say "mget *.txt" I should get all the files that
end in .txt on a dos or unix platform, but the file whose name is "*.txt"
on a macintosh (I think there is no magic '*' there, at least before MacOS X)
Then if I'm on a unix client and do 'mget [a-c]*' I should get all files
that start with a b or c, on a dos client I should probably get an error
as '[' means nothing to dos users (I believe), and also is illegal in file 
names.

Anything other than this means that file names given to ftp don't mean the
same as file names given to other commands on the system, and that's a very
bad UI design.

  | By the way, since there is so much confusion over what NLST should do, I
  | think there should be statement of clarification, to the effect that "in
  | view of widespread, if not universal, current practice..." NLST is
  | expected to expand wilcards and return a list of the files that match.

I would strongly object to that, and in fact, am quite likely to disable
wildcard processing in my server.

  | And that wildcard syntax and semantics depend on the software (client,
  | server, libraries, OS, whatever) -- there is no standard, no universal
  | syntax.  Unless we want to tackle that one too.  Personally, I think it's
  | better to "let the 100 flowers bloom" in this area.  Or at least let
  | sleeping dogs lie.

You mean you're proposing to require a mechanism that no-one can use safely
as no-one knows what it might mean?   That would be an "interesting" design
choice.

kre

---

From: Alun Jones <[email-masked]>
Date: Sat, 12 Oct 2002 16:03:33 -0500

>as '[' means nothing to dos users (I believe), and also is illegal in file

Better still, at least as regards your argument:

X:\>copy con [a-c].txt
xx
^Z
         1 file(s) copied.

X:\>dir [a-c]*
  Volume in drive X has no label.
  Volume Serial Number is 0000-0000

  Directory of X:\

10/12/2002  03:27 PM                 4 [a-c].txt
                1 File(s)              4 bytes
                0 Dir(s)     192,708,608 bytes free

Alun.
~~~~

--
Texas Imperial Software   | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place   | http://www.wftpd.com or email [email-masked]
Cedar Park TX 78613-1419  | VISA/MC accepted.  NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.