Subject: Ftp-WG: MLST nits

From: Robert Elz <[email-masked]>
Date: Thu, 14 Jan 1999 03:47:55 -0500

I am just cleaning up some oddments in my MLST implementation
- done mostly so I can include some real examples in the draft,
rather than to be useful for anyone - it is littered with code
like
	static char pbuf[1040];
which should never see any real use anywhere, but it works for my
private testing (if I wanted to break it, I have other ways than
concocting long strings, etc).

Anyway, while doing it, I encountered a few questions.  Most of these
don't necessarily involve anything that would change the mlst draft
in any way, though in a few, if we get soem kind of clear agreement
that additions to, or changes to, the text are needed, I will do that.
(A new version of the draft with more examples will appear sometime
soon anyway).

The first, and perhaps the most likely to perhaps result in a change
relates to the following text in the draft...

   The "l" permission applies to the directory file types, and indicates
   that the listing commands, LIST, NLST, and MLSD may be applied to the
   directory in question, and that MLST, LIST, NLST, and STAT may be
   applied to objects in the directory.

The relevant part is the "and that ..." clause at the end.   Perhaps this
is obvious, but it means that a directory which can be listed, but for
which the contents can't be referenced at all (on unix, a directory with
444 mode for example) can't be indicated as being listable, even though it
is.   Now such things are quite rare, almost useless, and perhaps deserve
to not be marked as being listable, in which case everything can stay just
the same as it is (including my code, which currently tests for both read
and exec perm bits before it sets the 'l' indicator).

Or, perhaps we might split those two features into separate indicators,
keep 'l' for what it is, and make a new one for "information can be obtained
about objects in the directory".   On unix the 'e' flag has the same
meaning as this new flag would have, but I would not like to overload it
in general (that is, for a unix server, 'e' and 'new' would both be set
when either was, just as 'c' 'p' and 'm' are either all set, or all clear.
On some other underlying filesys, 'l' and 'new' might always both be set
or clear together - all of that is OK).

Opinions?

Next, I hope it is clear in the draft already that implementations are not
required to report any facts which make no sense to report, and that that
is a file by file decision that can be made, not just a "my implementation
has no way to make the unique fact, so I never report it" type of decision
(though that one is acceptable as well).   If anyone wasn't clear about that
then let us know, so the draft can be made clearer - if you can point to a
place in the draft which suggests otherwise, or where more clarity could be
added, that would help.

The query relates to reporting the size of a directory object.   I was just
reporting the st_size value that unix supplies (number of bytes occupied by
the directory on disc) - but this really makes no sense at all, it is a number
that relates to almost nothing (except approximately the sum of the lengths of
the filenames the directory once contained, typically rounded up).   Now, my
implementation omits the size fact for directories (and consequently unless
I alter it, so will the examples that go in the draft).

The question is whether anything should be said in the draft about how the
size fact relates to directories?   It could be recommended to be omitted
(said not to apply to objects of type dir pdir or cdir), or it could be
defined in some other way for directories (perhaps a count of the number of
entries in the directory, or something).

Opinions?  Comments?

I also omit the size for objects that happen to be unix block or character 
special files - not that I couldn't (sometimes) find a reasonable size to
return there (discs have sizes anyway), but I'm too lazy, permitted to
omit the size, and so I will...   A really good implementation probably would
not, but mine is not that.

These however raise another question entirely.

We already have in the draft ...

   Files types that are specific to a specific operating system, or file
   system, can be encoded using the "OS." type names.  The format is:

        os-type   = "OS." os-name "=" localtype
        os-name   = <an IANA registered operating system name>
        localtype = 1*RCHAR

which is fine - I can, and do, return OS.UNIX=...

(UNIX is one of the IANA registered OS names, and the most appropriate for
this purpose, as block and char special files etc are common to all of the
unix variants I have ever heard of).

The question now is whether we need any form of commonality or registration
for the "localtype".   We can just ignore this, and allow the market to
reach some common usages as required, or we could request the IANA to make
a registry of OS dependent types (I don't think we want to attempt to
list everything here - that would be insane).

What I do currently is ...

Type=OS.unix=chr-13/29;Modify=19990104143839;Perm=;Unique=keVO1+5G4 device

(and yes, that is an output line from my test environment, "device" is the
file name of the char dev I created for the test it has no permissions
because I didn't care what major/minor numbers I gave it, and wanted to
make sure I didn't accidentally damage something by reading or writing
whatever it happens to be ... ie: I did a "chmod 0 device" or something).

The "chr-13/29" is supposed to indicate character device, major 13,
minor 29.   The same is done for block devices (using "blk") - I'm not
doing the "OS.Unix=BLOCK" thig which is in the draft purely as a means
to illustrate that the "localtype" is case independent.

So, do we want to request of the IANA that they register these OS
dependent file types or not?   If we do, I would expect that all that
should be needed for a registration would be message from someone
giving the OS name, the localtype they're registering, and a definition
for what it means.   That is, the idea would be just to make the information
available, not for anyone particular to debate it (the IANA would
verify that there weren't dozens of registrations of the same thing).

The same should be asked of the OS dependent facts.   My FEAT output
says (currently anyway) ...

 MLST Type*;Size*;Modify*;Perm*;Unique*;UNIX.mode;UNIX.chgd

The last two (which are off by default) are OS specific.  When I turn them
on (and turn off most others to keep the line here shorter), for the
same device above, I get ...

 Type=OS.unix=chr-13/29;UNIX.mode=020400;UNIX.chgd=19990104145937 device

(that's actually from MLST, hence the leading space).

UNIX.mode is the st_mode value (as is - so you can see I actually made the
device 0400 mode, it will be owned by root), UNIX.chgd is the "st_ctime"
value - which indicates when (almost) anything associated with the file was
last altered (which can be useful for mirroring type applications, perhaps).
Unix.mode has an octal integer as its value (standard unix representation),
and UNIX.chgd has a time-val as its value.

In the FEAT output above you'll see that I don't support the lang, charset,
or media-type facts (though they are recognised by the command parser as
valid fact names, so if you try "OPTS MLST charset" it won't be an error,
whereas "OPTS MLST foobar" would be - the "charset" fact request will just
be ignored).   This is all as the draft says should be done I believe.

The OS specific facts come from the text in the draft ...

   Further operating system specific keywords could be specified by
   using the IANA operating system name as a prefix (examples only):

        OS/2.ea   -- OS/2 extended attributes
        MACOS.rf  -- MacIntosh resource forks
        UNIX.mode -- Unix file modes (permissions)

The question is whether this is enough of a specification of OS specific
facts, and whether we should ask the IANA to maintain a registry of these
kinds of facts?

I believe that there are one or two other implementations being done.
Do the people doing those (if they're on this list) have any other issues
or comments to raise?

If so, now would be an excellent time to make comments.   I'll be doing an
update of the draft, including MLST examples, soon - if there aren't any
more issues to resolve, then that might very well be the last of the mlst
draft series.

kre

---

From: Alun Jones <[email-masked]>
Date: Thu, 14 Jan 1999 08:07:02 -0500

>and exec perm bits before it sets the 'l' indicator).

Correct me if I'm wrong, but such a directory isn't listable by MLSD,
either, right?  Hence, it's not markable with an 'l' in the first place.
As a side-issue, of course, some file systems (FAT comes to mind) have the
concept of a 'hidden' file, which should not be listed, even if the
directory is listable.  The 'l' shouldn't suggest that all these hidden
files are listable.

>place in the draft which suggests otherwise, or where more clarity could be

I'm sure that, however clear we make it, someone will write a client that
expects exactly the same range (and ordering) of facts to be presented on
each line of an MLSD/MLST output.

>entries in the directory, or something).

In the past, we've seen that some items that are undefined (such as listing
formats) will always be expected to match a de-facto standard.  In this
case, we have the advantage that not supplying the fact is a valid response
- since it is just as valid to suggest that the size reported be a total of
the sizes of each item in the directory (which could give away the presence
of hidden files!), as it is to report the number of files within the
directory (again - only non-hidden files?), maybe we should state that the
size fact does not apply to directories, and clients should not expect it
to appear, or carry meaningful information if it does.

>verify that there weren't dozens of registrations of the same thing).

That sounds like it would be useful, although it should be emphasised that
the creation of such facts should generally be in some manner related to
the OS, and not for the creation of pseudo-facts to reflect a 'virtual file
system' or some other artifact that the FTP server builds to supplement or
supplant a feature of the OS.  Is there an OS type that could be used for
such non-OS features?  I know that often a feature (such as file ownership
on FAT volumes) will be mimicked to provide similarity to other systems,
despite that such a feature is not provided by the OS.

>facts, and whether we should ask the IANA to maintain a registry of these

If they are to be usefully registered at all, then the format of those
facts must be registered as well.

I'll be taking a look back at the HOST section in the MLST draft shortly
and let you know if I have any further comments on it.

Alun.
~~~~

--
Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find it
1602 Harvest Moon Place | at web site http://www.wftpd.com or email
Cedar Park TX 78613     | us at [email-masked].  VISA / MC accepted.
Fax +1 (512) 378 3246   | NT based ISPs, be sure to read details of
Phone +1 (512) 378 3246 | WFTPD Pro, NT service version - $100.
*WFTPD and WFTPD Pro now available as native Alpha versions for NT*

---

From: Stephen Tihor <[email-masked]>
Date: Thu, 14 Jan 1999 09:21:25 -0500

> each line of an MLSD/MLST output.

Probably. But its one reason to include examples where the range and ordering
of the facts varies arbitrarilly from line to line :)

---

From: Robert Elz <[email-masked]>
Date: Thu, 14 Jan 1999 09:57:11 -0500

Date:        Thu, 14 Jan 1999 08:07:02 -0500
    From:        Alun Jones <[email-masked]>
    Message-ID:  <[email-masked]>

  | Correct me if I'm wrong, but such a directory isn't listable by MLSD,
  | either, right?

My server will list it correctly if no facts are requested (none
at all).   In that case it doesn't need to stat() the files in the
directory, and so doesn't.   Then read permission is enough.

It acts a little oddly if any facts are requested, acting almost
as if the directory is empty...

Thus ...

---> PORT 127,0,0,1,4,188
200 PORT command successful.
---> MLSD .
150 BINARY connection open for MLSD .
Type=pdir;Modify=19990104143638;Perm=e;Unique=keVO1+d?3 ..
Type=file;Size=0;Modify=19990104143932;Perm=awr;Unique=keVO1+8G4 writeable
Type=dir;Modify=19990114065955;Perm=cpmel;Unique=keVO1+7G4 promiscuous
Type=dir;Modify=19990114142538;Perm=;Unique=keVO1+1t2 no-exec
226 MLSD completed

Note no-exec which has no perms (no 'l' as, as currently implemented,m
it requires that the directory have the unix 'x' bit).   (There were a
few more output lines there which aren't relevant, snd so I deleted).

---> PORT 127,0,0,1,4,193
200 PORT command successful.
---> MLSD no-exec
150 BINARY connection open for MLSD no-exec
Type=cdir;Modify=19990114142538;Perm=;Unique=keVO1+1t2 no-exec
226 MLSD completed

That's with the default fact set (that was shown in the FEAT output
of my last message).

If then ...

---> opts mlst
200 Done
ftp> mlsd no-exec
---> PORT 127,0,0,1,4,194
200 PORT command successful.
---> MLSD no-exec
150 BINARY connection open for MLSD no-exec
 a
 b
 c
 d
226 MLSD completed

This is kind of similar to what a LIST command does...

---> PORT 127,0,0,1,4,211
200 PORT command successful.
---> LIST no-exec
150 Opening ASCII mode data connection for /bin/ls.
no-exec/a: No permission
no-exec/b: No permission
no-exec/c: No permission
no-exec/d: No permission
total 0
226 Transfer complete.

Pretty weird ...  but you still get to discover what files are
in the directory.   If I use a different client that I can (pretty
stupidly) convince to use the NLST command for other than an MGET
or similar, what I see is ..

200 PORT command successful.
---> NLST no-exec
150 Opening ASCII mode data connection for file list.
no-exec/a
no-exec/b
no-exec/c
no-exec/d
226 Transfer complete.

Then if I go back to my newer client ...

---> opts mlst type;modify;perm
200 Done

---> PORT 127,0,0,1,4,225
200 PORT command successful.
---> MLSD promiscuous
150 BINARY connection open for MLSD promiscuous
Type=cdir;Modify=19990114065955;Perm=cpmel promiscuous
Type=pdir;Modify=19990114142445;Perm=el ..
226 MLSD completed

That's what a listing of an empty directory would normally look like.
With no facts requested ...

---> opts mlst
200 Done
---> PORT 127,0,0,1,4,219
200 PORT command successful.
---> MLSD promiscuous
150 BINARY connection open for MLSD promiscuous
226 MLSD completed

Note no Type=cdir or type=pdir output there, as without the type
fact, there's no way to recognise them for what they are (there
might be an argument for keeping the type=pdir in this case, as it
is in fact an entry in the directory, the type=cdir cannot be
left in the output - perhaps I should make that explicit in the
doc?

(Aside: the reason the relative paths in the Type=cdir, rather than
fully qualified is because in this particular location in my ftp
tree, the server cannot determine fully qualified path names, it
lacks permission to do so.   Elsewhere...

---> PORT 127,0,0,1,4,233
200 PORT command successful.
---> MLSD .
150 BINARY connection open for MLSD .
Type=cdir;Modify=19990112030453;Perm=el /iana
Type=pdir;Modify=19990112030508;Perm=el ..
Type=dir;Modify=19990114073438;Perm=el assignments
226 MLSD completed

---> MLST assignments
250- Listing assignments
 Type=dir;Modify=19990114073438;Perm=el /iana/assignments
250 End

That comment is just for anyone who wondered).

  | Hence, it's not markable with an 'l' in the first place.

This may very well be the best solution though, certainly the
listing, even if available, isn't terribly useful!

  | As a side-issue, of course, some file systems (FAT comes to mind) have the
  | concept of a 'hidden' file, which should not be listed, even if the
  | directory is listable.  The 'l' shouldn't suggest that all these hidden
  | files are listable.

Does it now?    Can you suggest clarifying text?

  | I'm sure that, however clear we make it, someone will write a client that
  | expects exactly the same range (and ordering) of facts to be presented on
  | each line of an MLSD/MLST output.

I will try to avoid some of that by what is in the examples.  Unfortunately
my client never varies the order in which it presents the facts (doing
that would require some pretty weird coding I think), so I will have to
fake the responses a little to get that example.   But I think that is
worth while.   Thanks for the reminder on that one.

  | maybe we should state that the
  | size fact does not apply to directories,

That certainly suits me.   Avoids a bunch of work to implement anything
else.

  | That sounds like it would be useful, although it should be emphasised that
  | the creation of such facts should generally be in some manner related to
  | the OS,

OK.

  | and not for the creation of pseudo-facts to reflect a 'virtual file
  | system' or some other artifact that the FTP server builds to supplement or
  | supplant a feature of the OS.

I'm not sure I can imagine anyoen adding features to their virtual
filestore just so they can invent a new OS dependent fact to report it.
That one seems a bit far fetched.   I can certainly see implementors
inventing data to fit the standard facts, because it seems important
to support them all (I could have pretended that all my filenames are
Lang=English;Charset=utf-8 if I wanted to (or Charset=ascii).   That
happens to be true, but my server doesn't really have any means of
knowing it, the data is not recorded anywhere - not implementing the
facts is a better choice.   I will try to make that clear in the examples.

  | Is there an OS type that could be used for| such non-OS features?

Not that I recall from my scan of the OS type listing.   But someone can
always ask the IANA to add one...

  | I know that often a feature (such as file ownership
  | on FAT volumes) will be mimicked to provide similarity to other systems,
  | despite that such a feature is not provided by the OS.

Fortunately, we have no concept of ownership in the FTP NVFS (with or
without TVFS), so there's no problem of that one with MLST at least.
(I don't think it makes any sense for implementors to even add OS
specific facts to report that data, as over a FTP connection there
can't really be any expectation that the identity information revealed
means anything at all).

  | If they are to be usefully registered at all, then the format of those
  | facts must be registered as well.

Yes, of course.

  | I'll be taking a look back at the HOST section in the MLST draft shortly
  | and let you know if I have any further comments on it.

Thanks, for this, and for that.

kre

---

From: Robert Elz <[email-masked]>
Date: Thu, 14 Jan 1999 10:14:13 -0500

Date:        Thu, 14 Jan 1999 09:57:11 -0500
    From:        Robert Elz <[email-masked]>
    Message-ID:  <[email-masked]>

Apologies for replying to my own mail, but ...

  | ---> PORT 127,0,0,1,4,193
  | 200 PORT command successful.
  | ---> MLSD no-exec
  | 150 BINARY connection open for MLSD no-exec
  | Type=cdir;Modify=19990114142538;Perm=;Unique=keVO1+1t2 no-exec
  | 226 MLSD completed

I have just realised that is crazy, what it should do, and now actually
does, is ...

---> PORT 127,0,0,1,5,84
Type=pdir ..
 a
 b
 c
 d
226 MLSD completed

That is, it lists all the facts it can list for each file, omiting
any that it can't find.

So, should the 'l' flag still be off for my no-exec directory (as you
can see above that it is), or should it be on, with some other flag
indicating that information about entries in the directory will be
available?   Obviously it doesn't matter mush in this particular case,
it is obvious that no info is available, but if one does ..

---> mlst no-exec
250- Listing no-exec

One might like to know that listing the directory won't be
terribly enlightening, but is possible?

Similarly, from a listing of the parent directory.

kre

---

From: "Mike Gleason" <[email-masked]>
Date: Thu, 14 Jan 1999 13:23:13 -0500

At 03:47 AM 1/14/99 -0500, you wrote:

>Now, my implementation omits the size fact for directories (and
consequently unless
>I alter it, so will the examples that go in the draft).

This is what NcFTPd Server (UNIX) does and it seems to work well in
practice.  NcFTP Client simply leaves that field blank when it prints
directory listings obtained via MLSD.  Although admittedly, those two use
an implementation that is several months older than the latest spec.

>for the "localtype". [...] The same should be asked of the OS dependent
facts. 
>My FEAT output says (currently anyway) ...
>

FWIW, NcFTPd implements UNIX.mode, UNIX.owner, and UNIX.group.  I do think
the working group needs to help standardize these OS-specific items in some
form.  For example, NcFTPd's UNIX.mode only provides owner/group/other
(i.e. st_mode & 0777).  It also prints the mode as an octal number, using a
leading zero.  However, I have seen another implementation that simply
prints the whole st_mode in decimal.  A client wishing to use this fact
can't do this effectively if every implementation is different.

The spec should probably not directly define OS-specific facts, but if
OS-specific standards have evolved it should point to those documents when
possible.  This would cover the case where someone reads the spec but does
not participate directly in this group.

Mike Gleason

---

From: "Mike Gleason" <[email-masked]>
Date: Thu, 14 Jan 1999 13:43:19 -0500

At 03:47 AM 1/14/99 -0500, you wrote:

>for the "localtype".  [...]
>
>file name of the char dev I created for the test

I may have dozed off during discussion of this, but do we really need OS
specific types at all?  FTP clients and their users probably don't care
about device "files", etc.  IMHO, for FTP there are only two useful items
to the client, a data container (directory/folder) or a data object (file).
 If a user wants to see device files, socket files, named pipes, etc., let
them login by telnet.

Even an element in the filesystem that could be useful to FTP like a named
pipe could still be represented as a "file" if you can read from it or
write to it.  (However, in that case it could have indeterminate size, so
no size fact would be reported.)  Otherwise, you can't do anything to the
object otherwise, so from the user's perspective, it might as well not
exist (and therefore not be listed).

I apologize if my sleep deprivation is causing me to be short-sighted and
miss something blatantly obvious...

---

From: Robert Elz <[email-masked]>
Date: Thu, 14 Jan 1999 18:31:56 -0500

Date:        Thu, 14 Jan 1999 13:43:19 -0500
    From:        "Mike Gleason" <[email-masked]>
    Message-ID:  <[email-masked]>

  | I may have dozed off during discussion of this, but do we really need OS
  | specific types at all?

I suspect they're useful for mirroring, and similar.   FTP is more
than anon FTP (or ought to be).

  | Otherwise, you can't do anything to the
  | object otherwise, so from the user's perspective, it might as well not
  | exist (and therefore not be listed).

You may be able to recreate it, if what you're trying to so is make
a direct image of a filesys (a mirror).  True, named pipes and sockets
of various types aren't likely to be useful that way, but other things
can be (including devices).   I see less (but not zero) application for
the owner/group info than for being able to list the entire filesys
some way or other.

It seems that (so far anyway) there is agreement that we ought ask the
IANA to maintain a registry of OS specific facts and their definitions,
and from that, and if we do not agree to remove OS specific file types,
I'd assume (so far) agreement to ask the IANA to create a registry for
those as well.

I will write some text for the IANA considerations section, probably
over the weekend, and send it to the list for comment.

kre

---

From: Stephen Tihor <[email-masked]>
Date: Fri, 15 Jan 1999 00:06:38 -0500

Some data objects are quite complex.  For file system mirroring lots of
additional information needs to be moved around.  For a elegant transmission of 
complex objects (multipart files with complex typing information outside the 
pure body) it is also useful since one can use a much more natural
decomposition between header and body.   For FTP based network file systems it
is very useful indeed. 

There are other much less elegant ways to hack around these issues but they are
just hacks.   Even in the simpel UNIX space clean handling of hard and soft
links benefits from OS specific facts. 

Registering their semantics seems a good idea.  Pre-registering a setof POSIX
facts is probably politically correct and a useful suggestion for how to handle
most common UNIXisms.