Subject: Ftp-WG: Last Call on MLST is over ...

From: Robert Elz <[email-masked]>
Date: Tue, 18 Dec 2001 03:52:40 -0500

There were just two comments submitted (our AD managed to count none,
but he's busy...) that I know of.

The first, from [email-masked] asked for clarification of a few issues...

[email-masked] said:
  | Section 2.4 states that "unless stated to the contrary, any reply to
  | any FTP command ... may be of the multi-line format". 
[...]
  | This makes those parts of the specification that provide a more
  | restricted syntax for certain replies ambiguous: it isn't clear where
  | multi-line responses are permitted.  This happens in section 3.1
  | (defining a successful MDTM reply) and 4.1 (SIZE): both give ABNF that
  | on its face allows only for a single-line reply,

If the WG approves, to handle that, I propose putting the following short
paragraph at the end of each of sections 3.1 and 4.1 ...

   Note that when the 213 response is issued, that is, when there is no
   error, the format MUST be exactly as specified.  Multi-line responses
   are not permitted.

That is then "stating to the contrary" and should solve the problem I believe.

Zefram, is that sufficient?

Also ...

[email-masked] said:
  | Section 3.1, on the MDTM command, says "Attempts to query the
  | modification time of files that are unable to be retrieved generate
  | undefined responses.".  The phrase "undefined responses" sounds like
  | it permits what would otherwise be violations of the protocol,
[...]

For that, I propose changing:

   The "pathname" specifies an object in the NVFS which may be the
   object of a RETR command.  Attempts to query the modification time of
   files that are unable to be retrieved generate undefined responses.  

from section 3.1 of mlst-13.txt into:

   The "pathname" specifies an object in the NVFS which may be the
   object of a RETR command.  Attempts to query the modification time of
   files that are unable to be retrieved may generate an error-response,
   or can result in a positive response carrying a time-val with an
   unspecified value, the choice being made by the server-PI.

in what will become mlst-14.txt once we have agreed on the changes (doing
it this way to save on turnaround time for the I-D).

Zefram, is that OK?

Anyone in the WG have any problems with those changes?  If so, what would
be better wording?

The second comment (sent only to the IESG and Paul and myself as editors)
was from the RFC Editor.   They don't like the abstract, it doesn't meet
their current guidelines (which didn't exist when -13 was published I think,
certainly not -12, which was the last draft with any editorial changes).
Some of what they want certinly makes sense - having the abstract refer to
references in the doc makes the abstract much less useful when red apart from
the doc - which is really what the abstract should be for.

To handle this, I suggest making the abstract be ...

Abstract
   
   This document specifies new FTP commands to obtain listings of remote
   directories in a defined format.
   
   In order to allow consenting clients and servers to interact more
   freely, a quite basic, and optional, virtual file store structure is
   also defined.
   
   This proposal also extends the FTP protocol to allow character sets
   other than US-ASCII by allowing the transmission of 8-bit characters
   and recommending the use of UTF-8 encoding.

   Much implemented, but long undocumented, mechanisms to permit
   restarts of interrupted data transfers in STREAM mode, are also
   included here.

I am awaiting a reply from the RFC Editor as to whether this would meet
their guidelines or not.   Assuming that it does, does the WG have any
problems with that change?   Or suggestions for something better?
The new guidelines can be found from the rfc editor's web page.

Aside from those, I made one cosmetic change - the list of old FTP commands
in section 8 that are now required to handle UTF-8 path names has switched
from being one column down the left of the page in semi-random order, to
two columns, alphabetically sorted (column major I think it is - that is, down
the left column, then continue down the right column).  This actually ends
up saving a page in the final doc... (or does as an I-D - actual final
pagination for RFC format won't get done until the doc is IESG approved).

Also, since the only mention of reference [1] (US-ASCII spec) used to be
in the abstract, I added a couple of mentions of it other places...

You will, of course, get time to read the revised draft and comment (perhaps
not much time - or perhaps if the IESG decides to be dumb again, a lot more
time...) after it is published - the point of this mail is to avoid having to
submit too many more iterations of this doc if at all possible.

If no-one objects to the changes here, I'll send in the new I-D in a day or
two, then if there are still no objections, our chair can ask the IESG to
progress the doc to a PS RFC (I will send the rfc editor the nroff sources
for the doc, so once IESG approved, it should not take too long to appear).

kre

---

From: "Brad Clements" <[email-masked]>
Date: Tue, 18 Dec 2001 10:00:27 -0500

>    unspecified value, the choice being made by the server-PI.

Is there some particular reason why an MDTM call on a non-existing object should not 
always fail with a positive permanent error?

I think I've seen some clients that use MDTM as a simple existance check, so I do not 
think that a random time-val response is appropriate if the file cannot be subsequently 
LIST,NLST (etc)

In my opinion, if the file object will not be listed by NLST or LIST (etc), then MDTM 
should return a permanent error response.

Note that I'm not tying this to RETR, since an object may exist, but be write-only for the 
currently authenticated object.  Hmm.. In NetWare, we allow write-only without LIST or 
READ access as anonymous "drop boxes". In that case, MDTM would also be 
expected to return a failure response, even though the authenticated user could write to 
the object. Since MDTM returns information about an object (existance, and modtime), 
which is a subset of info returned from LIST (typically), it should be tied to LIST rights 
from an authentication standpoint.

In all cases though, if the file doesn't exist or the user doesn't have that right, then I 
think an error response is the only appropriate response, not an unspecified timeval.

Brad Clements,                [email-masked]   (315)268-1000
http://www.murkworks.com                          (315)268-9812 Fax
netmeeting: ils://ils.murkworks.com               AOL-IM: BKClements

---

From: Alun Jones <[email-masked]>
Date: Tue, 18 Dec 2001 13:19:18 -0500

>always fail with a positive permanent error?

Not non-existing, but "unable to be retrieved".  Non-existant files should, 
of course, give the usual 550 error.

>should return a permanent error response.

Again, the language you quote says "the modification time of files that are 
unable to be retrieved", not "the modification time of files that do not 
exist / that will not be listed"

>from an authentication standpoint.

A little further in the draft (the next paragraph), we see "The server-PI 
will respond to the MDTM command with ... a 550 reply if the file does not 
exist, ..." - apparently your objections have already been addressed.  The 
possibility of a "213 <random-valid-but-not-necessarily-correct-time-val>" 
response seems to me to be more appropriate for an item which exists, can 
be listed, but whose modification time cannot reliably be established, to 
avoid the problem of an FTP client screwing up when it tries to MDTM a file 
as an existence check.  F'rinstance, a volume root on Windows won't give 
you a valid date from CFile::GetStatus (which itself simply calls 
::FindFirstFile).

I suspect that the aim is that anything that shows up as an output line in 
NLST should also serve as an input to MDTM and result in a 213 response - 
anyone else care to contribute on this?

Alun.
~~~~

--
Texas Imperial Software   | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place   | http://www.wftpd.com or email [email-masked]
Cedar Park TX 78613-1419  | VISA/MC accepted.  NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.

---

From: Robert Elz <[email-masked]>
Date: Tue, 18 Dec 2001 21:53:08 -0500

Date:        Tue, 18 Dec 2001 10:00:27 -0500
    From:        "Brad Clements" <[email-masked]>
    Message-ID:  <3C1F11E9.30977.3848AD2E@localhost>

  | Is there some particular reason why an MDTM call on a non-existing object
  | should not always fail with a positive permanent error?

What Alun said, but, I could make the text be ...

	Attempts to query the modification time of files that exist
	but are unable to be retrieved may generate an error-response, [...]

if that would make it more clear?

  | READ access as anonymous "drop boxes". In that case, MDTM would also be 
  | expected to return a failure response,

No, that's exactly the case where MDTM is expected to return whatever
you decide to make it return.   That is, you can check to see that it
is un-retrievable if you like, and return an error, or you can return the
last time someone dropped something in there, or you can return the epoch,
or just about anything else that happens to be easy.

  | In all cases though, if the file doesn't exist or the user doesn't have
  | that right, then I think an error response is the only appropriate
  | response,not an unspecified timeval.

Yes, the unspecified case was really meant to handle situations where
the file does exist, the user has permissions, but what is there isn't
a normal file (doing MDTM on a directory is one such situation).   That
is, the server isn't required to check that the object is one where it
makes sense to look for a modification time.

If you have a suggestion for improved wording, please send it soon.

kre

---

From: Robert Elz <[email-masked]>
Date: Tue, 18 Dec 2001 21:55:13 -0500

Date:        Tue, 18 Dec 2001 03:52:40 -0500
    From:        Robert Elz <[email-masked]>
    Message-ID:  <[email-masked]>

  | I am awaiting a reply from the RFC Editor as to whether this would meet
  | their guidelines or not.

The RFC Editor (or a part of it) has suggested ...

Abstract
   
   This document specifies new FTP commands to obtain listings of
   remote directories in a defined format, to define a virtual file
   storage structure, to allow character sets other than US-ASCII, and
   to permit restarts of interrupted data transfers in STREAM mode.

which looks fine to me - much shorter, and still covers pretty much
everything (MDTM and SIZE can just be considered part of what is needed
for restarts - for the purposes of the abstract anyway).

Any problems with that version?

kre

---

From: "Brad Clements" <[email-masked]>
Date: Wed, 19 Dec 2001 10:40:57 -0500

> if that would make it more clear?

I think the issue is being somewhat clouded because we are overloading the MDTM 
command with too many "uses".

1. MDTM on objects that exist, but don't have the concept of a modtime, such as a 
volume. I suppose the returned modtime should be up to the server implementation and 
could be any valid timeval. Why a client would request this I don't know, perhaps 
because the object was an element of a path? 

2. MDTM on objects that do not exist should return an error response.

3. MDTM on objects that exist, but the authenticated user does not have "rights" to..

I'm not sure how best to handle this, seems that no single answer suites everyone. Our 
server implementation treats option 3 like option 2 if the user would not "see" the object 
in a LIST command. That is, if LIST,NLST, MLST would not return the object, then 
MDTM returns an error response.

This is independent of read and write, rights.

I think requiring that MDTM take "LIST access rights" into account is beyond the scope 
of this document.

Not specific wording, but in an earlier post was this text:

>   modification | time of files that are unable to be retrieved generate
>   undefined | responses.".  The phrase "undefined responses" sounds like

1. I'm objecting to "unable to be retrieved" - Personally I think it should be tied to LIST 
or NLST if, indeed, the use of the word "retrieved" was implying RETR, rather than 
meaning that the OS could not provide the object modtime to the server 
implementation.

If you meant "retrieved" as in, the OS couldn't provide the value, then I think the 
wording should be: "files for which a modification time is not available".

If you meant "retrieved" as in, a subsequent RETR on the file would fail, then I think this 
needs to be cleared up.

2. undefined response should be: either an error response or a timeval whose value is 
implementation specific.

You may also wish to add a cautionary statement that using MDTM to determine the 
existance of an object or the validity of a path is not appropriate. Or is it? If we expect 
clients to use MDTM as a shortcut to determine if a path is valid, or if a file 'exists', then 
we should clarify how that mechanism is supposed to work.

---

From: Robert Elz <[email-masked]>
Date: Thu, 20 Dec 2001 14:46:08 -0500

Date:        Wed, 19 Dec 2001 10:40:57 -0500
    From:        "Brad Clements" <[email-masked]>
    Message-ID:  <3C206CD8.26761.3D93ECF1@localhost>

  | I think the issue is being somewhat clouded because we are overloading
  | the MDTM command with too many "uses".

Actually, "we're" not overloading it at all - its purpose is to check whether
a file that was RETR'd previously which failed part way through, has been
changed since.  If not, then restarting the transfer (using REST) is probably
going to be OK.  If it has changed, then it isn't.   And yes, there's still
the race condition (the file might be altered between the MTDM and the
subsequent REST/RETR).

  | 1. MDTM on objects that exist, but don't have the concept of a modtime,
  | such as a  volume. I suppose the returned modtime should be up to the
  | server implementation and could be any valid timeval.

That is what the text is trying to say.

  | Why a client would request this I don't know, perhaps 
  | because the object was an element of a path?

I don't know either ... and if the text makes it clear that what happens
if a client does this is more or less unspecified, then clients will probably
stop doing it (if any do).

  | 2. MDTM on objects that do not exist should return an error response.

That's OK.

  | 3. MDTM on objects that exist, but the authenticated user does not
  | have "rights" to..

I don't care about this case at all.   As MDTM is a support function for
REST, it really doesn't matter.  Whether the client restarts or not, the
RETR that follows will fail...   That's all that really matters.

  | If you meant "retrieved" as in, a subsequent RETR on the file would fail,
  | then I think this needs to be cleared up.

That was the intention - that is, for MDTM's purpose, if RETR won't work,
then what MDTM returns is simply irrelevant (it can return an error, or it
can return any random time it likes).

  | You may also wish to add a cautionary statement that using MDTM to
  | determine the existance of an object or the validity of a path is not
  | appropriate.

Perhaps.   I'd much prefer to simply get this doc finished and stop
continually fiddling with it...

One might hope that when MLST gets implemented more, clients will start
using that to find out about files.  They get better information, just
as cheaply.  Specifying that MDTM can do anything it likes, except in
the circumstance where it should be used should help dissuade people from
using it inappropriately..

This is one of the drawbacks of having facilities like MDTM implemented,
and in wide circulation, but undocumented.  People know they exist, but
there's no way to discover what is an appropriate use, and what isn't.
So, if it seems to work, it must be OK...

Note, the most common use of MDTM today is, I think, undoubtably that of
setting the mod time on a fetched file to match the mod time of the file
on the server.   That is supported by this spec - the file was able to be
RETR'd, so it should have an MDTM response that is sane.

kre

---

From: "Brad Clements" <[email-masked]>
Date: Thu, 20 Dec 2001 14:54:10 -0500

> a file that was RETR'd previously which failed part way through, has been

Many of our customers are using it to determine if the file has changed, a poor man's 
replication, even in STOR mode, but I digress

> continually fiddling with it...

I've been lurking on this list for quite and while and so far have had nothing to say. Sorry 
my comments have come so late in the game. I concur, get the doc finished and ship it.