Subject: Ftp-WG: NLST vs MLSD

From: Frank da Cruz <[email-masked]>
Date: Sat, 12 Oct 2002 13:49:37 -0500

On Sat, 12 Oct 2002 10:31:40 -0500, Robert wrote:
: At Thu, 10 Oct 2002 10:25:57 -0500, Frank wrote:
:   | It seems to me that the MLSD spec pushes a lot of work off on the 
:   | client,
: 
: Absolutely.  Which is where it ought to be.  As a client author I know
: that means more work for you...
:
I'm sure you know my interest in this is not avoiding work.  I've already
done most of the work, so if nothing changes the Kermit FTP client will
fit in, but how will users receive it?  My concerns are:

 1. That the spec be as clean, simple, and elegant as possible to
    promote the widest implementation; not just superficially (because
    it already is clean and elegant on the surface) but also at the
    deeper level where it must be implemented.

 2. That it not force choices upon the user that s/he is not equipped
    do deal with.  This not only frustrates users, but exerts back
    pressure on help desks.  That is, it has human consequences.

I understand there's a tradeoff between the complexity of the server and
the client, but we're talking about the interface between them.  The
interface should be usable in a straightforward, obvious way or it won't
take off and we'll still be parsing "ls -l" listings for another 30 years.

:   |  . Whether the server should interpret wildcards in the MLSD argument.
: 
: Not just whether, but also what he wildcards actually mean, otherwise your
: client is going to have to interpret the user's spec, and re-code it into
: some standard wildcard language.
: 
Wildcard syntax and semantics are, by nature, up to the software that
implements them.  I don't believe this can be legislated, given the
free-for-all nature of software development.  You also can't legislate away
30 years of practice.  The protocol specification might be cleaner and
clearer by avoiding the issue, but neither users nor developers will be
happy.  Nor can you safely design a standard wildcard language because you
can't enumerate all the known and future file-system and wildard syntaxes,
and therefore can't avoid introducing conflicts and ambiguities.  At some
point, if only as a final resort, the user must be exposed to the server
host's notation -- it can't be totally hidden without interfering with
access.

As noted previously and agreed by all, RFC959 was not clear in this area,
This resulted in widespread if not universal server-side wildcard
interpration.  The cat has been loose and breeding for decades; you can't
undo this.

New servers that support MLSD as specified (and perforce, continue to also
support NLST) will force arcane choices upon the user, as noted in my paper.
We might consider it unlikely that a user might want to get 3 files out of a
directory containing a very large number files, but that does not mean the
protocol should ignore this case.  It's same argument that was made when
designing the address space of the early networks, which was outgrown faster
than any expert foresaw (more than once).

: Personally, I believe that wildcards have no place at all in the FTP spec.
: They will cause far more problems than they'd ever solve.
:
You can't ignore them.  RFC959 did and look what happened.  The spec needs
to say something about which side handles them and, since there are
legitimate and even compelling arguments to made for each side, a choice
should be offered.  Suppose we have two commands, MLSD and MLSW that are
implemented in every server.  There is a use for each.  GUI drag-and-drop
FTP clients will choose MLSD; command-line clients (which we must not
pooh-pooh) will tend to prefer MSLW because they are used by professionals
who care about -- and need -- detailed control and automation capabilities.

: Note that you cannot simply use a cdir line, require it to come first, and
: define that as the beginning of a new directory, or you would be unable to
: distinguish the case of a directory having multiple cdir lines (which is a
: very reasonable thing to do - giving both a fully qualified path, and a
: relative path from the PWD) from an empty directory, followed by another
: one.
: 
Then "cdir" needs a stricter definition; it should do one thing or the
other.  Add another fact if it's necessary to do both (and I believe it is;
the simple relative form for GUI title bars or whatever, and the fully
qualified form for use by the client in navigating the tree).

In fact, this one of the most annoying things about FTP -- you can't find
out (programmatically) the server's current directory.  The PWD command is
useless.  CDIR should be an improvement.

Ditto for PDIR.

:   |    But what about downloads?  If the server
:   |    could employ these same algorithms, then it could include a
:   |    "text-or-binary" subfact with the "type" fact, similar to the MIME
:   |    Content-Type value.  This would be a hint to the client to use the
:   |    appropriate mode for RETR, so as to avoid garbaging the file.
:   |    (Just a hint -- of course the client can do as it wishes.)
: 
: Sounds reasonable to me, write a definition for the fact type....
:
FTP protocol requires the client to control everything, but only the
server knows about its own files.  The server needs a way to suggest the
appropriate STRU and TYPE for each file to the client.  This could be done
in various ways, but the cleanest is to add new facts:

  mode=<any-mode-value>   (S, B, C)
  stru=<any-stru-value>   (F, R, P)
  xtype=<any-type-value>  (A[ x], E, I, L x)

"xtype" distinguishes this new fact from the existing "type" fact.  If a
value contains spaces (as TYPE values can), we have to work around the
prohibition against facts containing spaces, such as replacing space by
(say) an ASCII punctuation character, e.g.:

  xtype=L:8
  xtype=A:C

Presumably we won't have to confront "quoting hell" since MODE, STRU, and
TYPE values should always be alphanumeric (but watch out; I don't see this
stated explicitly anywhere).

The wording should be to the effect that each of these facts can have any
value defined for the corresponding FTP command, thus not committing
ourselves to enumerate them, which would require this spec to change any
time a new MODE, STRU, or TYPE value was added somewhere else.  But it
should list the existing values as examples.

:   |  . For cross-platform (or even same-platform) directory-tree 
:   |    duplication, it might be desirable to have a standard notation for
:   |    file permissions, as distinct from the "perm" fact, which applies
:   |    only to the current user of the FTP connection.
: 
: Im not sure how exactly that will work, as on almost all systems, the
: permissions vary from user to user.  To describe anything more general
: than the current user, you need to be able to identify users in some way.
: And that then means some kind of general user namespace management
: problem.
: 
We've had some experience with this in Kermit.  Clearly you can't cover
everything (ACLs spring to mind), and there's not much you could reasonably
expect to do about namespace (file owner and group).  But a simple model
could accommodate most things that people care about: user and world read,
write, delete, execute, list, and append permission.  Maybe group too.  This
takes in VMS, Unix, DOS, TOPS-20 (gone but not forgotten because it
pioneered many concepts some of which still need to catch on).  In any case
this is just information provided by the server to the client in a new
fact, say XPERM.  The client can use it however it wants.

In the reverse direction it is also often desirable to set permissions of
uploaded files.  FTP protocol does not give a way to do that.  Defining a
standard notation for file permissions could open the door to this.

: Yes, "executable" is perhaps one thing that is currently missing, though
: it is not clear to me whether executable is a permission, or a file type,
: or both.
: 
It's a permission because it grants execute permission.  In TOPS-20, a
program could have execute but not read permission and you could still run
it.  But you couldn't copy it, save it, or look at its contents.

Of course conceptually it's also a file type, and perhaps it might be useful
to distinguish between (say) a shell script that has execute permission and
an executable binary program (that might or might not have execute
permission), but I'm not sure I see any need for this.

Of course some file systems overload execute permission to save bits, e.g. a
Unix directory with x permission means you can list its contents, but that's
a separate concept.

: We could perhaps do with another fact type, perhaps attributes, to
: describe things that aren't really permissions, and aren't really types
: either.
;
We've done this in Kermit, but far from perfectly.  The biggest problem
comes in defining how to describe highly-structured files: fixed-length
record files, variable-length record files (as in VMS, VOS, CMS), nonstream
files (relative, indexed), various record formats (e.g. with carriage
control).  Ultimately what we settled on was a special OPTIONAL mode of file
transfer in which the file's control block is sent (in a specially coded
segment) along with the file.  If the file transfer partner is on the same
platform, it can use it to reconstruct the file; if it's not, it can archive
the information in a way that can be used if the file is subsequently
transferred to a computer with compatible FCBs.

But in the general case, I think you've already defined most of the commonly
used attributes: date, size, character-set, etc, and FTP's existing MODE,
STRU, and TYPE commands cover most of the rest.  (Any VMS FTP implementors
read this?)

:   | But as the spec currently stands,
:   | there is no way to do that without risk.
: 
: There is no way at all to use NLST the way you do, at all, without risk.
:
It's not me, it's everybody.

: You have just decided that the risk is low enough that you have decided
: to ignore it.   As server load gets higher, I fully expect to see servers
: start looking at the spec, seeing that they're not required to handle
: wildcards, and simply stop doing that (just as many that used to do on the
: fly gzip or ungzip, no longer do, because it just costs too much for the
: server, regardless of how much it would save the network or the client).
:
You're overloading MLSD.  It is at once a new file-listing format and a
new discipline to reduce load on the server -- two unrelated concepts.  It
should be possible to get the new listing format without having a whole
new set of rules about who does what.

Perhaps what we need is a way for the server to set policy.  First of all,
we have to agree to accept the fact that NLST expands wildcards because it
does; you can't change it.

Then suppose the server supports MLSD, and announces this in its FEAT
response.  The client sends "NLST [abc]*.zip".  The server says:

  550 Too busy for NLST - try MLSD

The client can switch to its MLSD handler and try again (sending "MLSD"
and then holding "[abc]*.zip" locally for matching).

:   | Thus the hideously complicated and
:   | obscure user interface described here:
:   | 
:   |   http://www.columbia.edu/kermit/newftp.html
:   | 
:   | to allow the user to get out of trouble when the client's choices are
:   | not what the user needs.
: 
: This is all because your'e attempting to optimise things which very
: rarely ever need optimising.
:
I've been in this business long enough to know that using words like
"rarely" is tempting fate.

As for the rest, we're going around in circles; I've said my piece and I
don't care that much.  If anybody else cares about these issues, please
speak up; if not, don't.

Meanwhile, in case I still have anybody's attention, here's another item
that might be worth looking at:

RFC959's STOU specification is inadequate.  First, it does not accept an
operand, which is unreasable.  You should be able to "stou foo.bar" and have
it stored as foo.bar if there is no conflict, or foo.bar.1 or whatever if
there is, and in fact many clients and many servers do allow this, despite
the spec.  Second, and more serious, there is NO WAY for the client to find
out the name picked by the server for the file.  Why should we care?  FTP is
used increasingly for transaction processing, EDI, etc (for example at IBM
Info Exchange and Advantis).  "Atomic file movement" and feedback is
essential for transaction processing.

Summary:

 . Add MLSW, MLSR, MLSX.
 . Define a standard notation for permissions.
 . Add XTYPE, MODE, and STRU facts.
 . Add an XPERM fact for sending file permissions (as distinct from PERM).
 . Add a command (PERM?) for setting server file permissions.
 . Do something about STOU: augment its definition or define a new command.

Oops, one more thing....  The SIZE command returns a different result
depending on the prevailing TYPE.  But the SIZE fact always returns the
actual length of the file in octets, right?  I just wanted to make sure
this is intentional; in fact I prefer the latter behavior.  I discovered
that if you don't switch to TYPE I before sending a SIZE command then
you can wind up waiting an awfully long time for the answer!

- Frank

P.S. Sorry for sending a "help" command to the list -- I meant to send
it to the -request address...

---

From: "D. J. Bernstein" <[email-masked]>
Date: Sat, 12 Oct 2002 16:14:55 -0500

Frank da Cruz writes:
> widespread if not universal server-side wildcard

Certainly not universal. High-quality clients don't rely on it. You are
overgeneralizing when you claim that NLST expands wildcards.

> new discipline to reduce load on the server -- two unrelated concepts.

The discipline is not new. If you want to handle wildcards, you should
be relying on the standard protocol that works with all servers, rather
than on the behavior of certain FTP servers. Load is a red herring; the
real issue is reliability.

> At some point, if only as a final resort, the user must be exposed to
> the server host's notation -- it can't be totally hidden without
> interfering with access.

False. For example, NFS and ftpfs users access files just fine without
having any clue about the server's favorite wildcard notation.

> than any expert foresaw (more than once).

No, it isn't the same argument.

Obvious difference #1: Address limits are much more serious than speed
limits. Speed is multiplied by a renewable resource, namely time, while
addresses are not.

Obvious difference #2: Limits on the size of the entire system are much
more serious than limits on local branching factors. I don't care
whether I can put a million files into a single directory; I do care
whether I can put a million files onto a single disk.

> The server needs a way to suggest the
> appropriate STRU and TYPE for each file to the client.

No. It works much more smoothly to transfer a binary stream, along with
enough out-of-band information for the client to understand the stream,
as in HTTP. This is another good example of client-side processing being
better than server-side processing.

> FTP is used increasingly for transaction processing, EDI, etc (for
> example at IBM Info Exchange and Advantis).

Why? Do they have no clue how to design a sane protocol?

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: Alun Jones <[email-masked]>
Date: Sat, 12 Oct 2002 19:33:34 -0500

At 01:49 PM 10/12/2002, Frank da Cruz wrote:
>protocol should ignore this case.

Actually, yes, it may very well mean that.  There are people out there, 
only a few, who want to use FTP to stream video.  Does that mean that the 
general standard of FTP should be updated to include a command that lists 
the frame-rate of an MPG file?  The more unlikely a usage is, the less it 
needs to be made a part of the general protocol.

>should be offered.

The spec, as far as I was aware, _does_ say something about which side 
handles them:

2.2.2. Wildcarding
...
    Clients that desire some form of pattern matching functionality must
    obtain a listing of the relevant directory, or directories, and
    implement their own file name selection procedures.

That sounds pretty clear.  It may not be right, but it is currently 
present.  I'd like to see how some of the MLST implementations you've cited 
that handle wildcards cope with extreme cases - how well thought-out they 
are.  It may very well be the case that these MLST implementations are 
sufficiently broken that the advantage of wildcard support is lost among 
the disadvantage of unreliable output.  If wildcards are to be supported, 
then these extreme cases must be supported _in_the_spec_.

>Suppose we have two commands, MLSD and MLSW that are

If we're going to haggle about MLSW, then let's wait until we've got _this_ 
draft to RFC status; if MLSW is going to be a separate command, then it 
needs to stop holding up this draft, already long overdue in many respects.

>qualified form for use by the client in navigating the tree).

Okay, so maybe the draft doesn't say that "cdir should not be used as a 
heading".  It also doesn't say that it shouldn't be used as a dessert 
topping.  The spec does say what cdir's purpose is, it doesn't say what its 
purpose isn't.  Any 'cdir' form can be used by the client in navigating the 
tree, as "CWD <cdir-name>" will get you to the directory listed, from the 
current directory.

>useless.  CDIR should be an improvement.

Since RFC 959, PWD is supposed to respond:
257<space>"<directory-name>"<space><commentary>

However, maybe there is room for a record in MLST that specifies the 
absolute path, if one exists.  In a TVFS file system, of course, that's 
_very_ easy to generate and recognise - it's any cdir entry where the name 
starts with "/".

>   mode=<any-mode-value>   (S, B, C)

Why a fact for MODE?

>   stru=<any-stru-value>   (F, R, P)

No objections here, it would seem that this property would come directly 
from the filing system - of course, in many popular filing systems, it's 
going to be completely extraneous as it'll always be 'F', but there are 
systems that might want to communicate that a file is a 'record' format file.

>   xtype=<any-type-value>  (A[ x], E, I, L x)

This is a tricky one to defend, though.  How do you tell whether a file 
needs Image or ASCII mode?  Reliably, I mean, and without using up a whole 
heap of resources by scanning every file just to produce a listing.

>(say) an ASCII punctuation character, e.g.:

Or, since the first parameter is a single character, omitting the space 
would do just as well.

>standard notation for file permissions could open the door to this.

If this is done, of course, it's important to note that the permissions you 
get back out may not be exactly the same as the ones you put in.  Ask to 
mark a file as readable but not executable, and what do you then do on a 
file system where the two facts are the same?  Server-dependent?  Probably, 
but it's likely that clients won't be happy to discover that their commands 
aren't interpreted exactly.

>it.  But you couldn't copy it, save it, or look at its contents.

Why is it of importance to FTP, though?  FTP is not about executing 
files.  It's about reading and/or writing them, with a touch of deleting, 
renaming, etc.  But there's no provision for execution (and it's a bad idea 
when there is!)

>It's not me, it's everybody.

"Everybody" is using NLST in a way that is not documented.  It's also 
liable to a few freaky behaviours.  I can remember the first time someone 
complained to me that "MGET *" didn't get all the files.  Well, no, because 
on Win16 (the platform I was working on at the time) doesn't match "*" to 
"every file", it specifically only matches "every file that doesn't have an 
extension".  Wildcard expansion at the server is not as unambiguous and 
safe as all that.

>new set of rules about who does what.

The format, however, requires a new set of rules.  The format as it stands 
is of no use when it comes to listing a wildcard that expands to match 
files in multiple directories - it's also of no use to a listing that 
matches files and directories.  If MLSD should accomodate wildcards, then 
it needs a rewrite of the format; if the format stays as it is, then the 
rule must be that MLSD does not accomodate wildcards.

>does; you can't change it.

I don't think that's a good argument for you to be proposing.  I'd rather 
approve a standard on the basis of "because it's useful to do so", instead 
of "because everyone expects it to behave in the way that one vendor 
implemented it".  Need I raise the spectre of the "Big M" vendor as an 
example as to why it's a good idea to specify, rather than follow the most 
popular implementation?

There are arguments why wildcard expansion is good.  One is that it allows 
a server to sacrifice an amount of processor time to save an amount of 
bandwidth and transmission time.  This may be a good trade, even for the 
server.  An FTP server that's just an FTP server may very well spend most 
of its time simply accessing disk and network - not a lot of processor 
operations (unless it's encrypting, in which case it's probably spending a 
lot of time there).  Now, that's a better argument than "we'll do it this 
way because it's the way it's been done so far".  After all, the way it's 
been done so far is with an undefined de-facto "everyone does it like Unix" 
listing format :-)

>and then holding "[abc]*.zip" locally for matching).

A client may well want to choose to evaluate wildcards locally for other 
reasons - for instance, it may know better how to expand certain 
combinations; "[abc]" as a file specification in Windows, for instance, 
means a file called "[abc]", not "a single letter file name, either a, b or 
c".  Similarly, as someone noted earlier (possibly you?), there may be 
times when it is appropriate for the user who knows the wildcard syntax of 
the server, to ask the server to expand.

>:   | obscure user interface described here:

If we can avoid words like "hideous" or "obscure", we might avoid making 
this into an overly emotional discussion, and get down to the issues at 
hand.  Thanks.

>"rarely" is tempting fate.

Okay, so let's look at the scenarios.  You've suggested a possible 
situation that might cause server wild-card expansions to be a useful thing 
- does this occur frequently enough in practice that it's worth carrying 
the implementation over to a new command?  As you've noted, every server 
currently does wildcard handling in NLST, so is there sufficient saving 
from current use of NLST wildcards to justify insisting on its retention?

Does anyone here have means of gathering the sort of statistics that might 
be useful in determining this?

>the spec.

I'll admit it - even my own server allows a parameter for STOU.  But part 
of the point of STOU is that we are expecting there to be a collision, in a 
significant number of cases.

>Second, and more serious, there is NO WAY for the client to find
>out the name picked by the server for the file.  Why should we care?

Apparently, because we haven't read RFC 1123 :-)

          4.1.2.9  STOU Command: RFC-959 Section 4.1.3

             The STOU command stores into a uniquely named file.  When it
             receives an STOU command, a Server-FTP MUST return the
             actual file name in the "125 Transfer Starting" or the "150
             Opening Data Connection" message that precedes the transfer
             (the 250 reply code mentioned in RFC-959 is incorrect).  The
             exact format of these messages is hereby defined to be as
             follows:

                 125 FILE: pppp
                 150 FILE: pppp

             where pppp represents the unique pathname of the file that
             will be written.

>  . Add MLSW, MLSR, MLSX.

In a separate draft, _please_.

>  . Define a standard notation for permissions.

Wow.  Good luck.

>  . Add XTYPE, MODE, and STRU facts.

MODE?  No.  XTYPE?  How to do it reliably?

>  . Add an XPERM fact for sending file permissions (as distinct from PERM).

Should be handled by OS-specific facts.  File permissions as they are on 
the file system are file-system dependent.

>  . Add a command (PERM?) for setting server file permissions.

Again, this is an OS-specific issue.  A SITE command should be used; 
_maybe_ a set of OS-dependent or FS-dependent commands.  I don't think 
you'll get a generic format.

>  . Do something about STOU: augment its definition or define a new command.

You know, I think you've got enough here that it's worth starting a new draft.

>you can wind up waiting an awfully long time for the answer!

Not only is this intentional, but it is documented.  Since MLSD doesn't 
generally precede a downloading of every file in the directory, it would be 
wasteful to calculate the transfer size of each file.  SIZE, on the other 
hand, takes _one_ file name as parameter (hey, why not make that take 
wildcards?  Just kidding!), and as such it's expected that the client user 
knows that the operation will take time.

Alun.
~~~~

--
Texas Imperial Software   | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place   | http://www.wftpd.com or email [email-masked]
Cedar Park TX 78613-1419  | VISA/MC accepted.  NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.