Subject: Ftp-WG: Please comment

From: "sobhan sunil dasari" <[email-masked]>
Date: Mon, 27 Sep 1999 09:25:22 -0400

Dear Sir
I am attaching a txt file containing a draft for a possible extension 
to FTP.I would be glad to know your comments on the draft.
Thanks
With warm regards
D.Sobhan Sunil
Final Year B.E
DEPT OF METALLURGY
RECT,INDIA

-----------------------------------------------------------------------------
Submitted by
Final Year B.E 
Dept of Metallurgy
Regional Engineering College	
Trichy - 620015
Tamil Nadu
India 

Introduction

  This document amends the File Transfer Protocol (FTP) . The aim 
  of this document is

  a) To define the procedures for automatic restart of a 
  interrupted transfer using commands which have already been 
  proposed like "REST", "SIZE" and "MDTM".

  b) Introduce a new command "INFO" which can be used to obtain 
  details about a file in a machine independent format. 

  c) Standardize the method of approach for recursive downloads 
  using "INFO". 

  d) Suggest the usage of "INFO" for automatic restart of 
  interrupted transfers.

  e) Explore the advantages and potential applications using 
  "INFO".

  File Modification Time (MDTM) command

  The FTP command MDTM is used to find the modification time of 
  the file in the server file system . This command is being 
  supported by many ftp servers at present.This command returns 
  the time at which the file given in its argument field was last
  modified.The time is returned in ISO 3307 "Representation of
  the Time of Day" format.

  This format is

  YYYYMMDDHHmmSS or YYYYMMDDHHmmSS.xxx, where

  YYYY is the year
  MM is the month (01-12)
  DD is the day of the month (01-31)
  HH is the hour of the day (00-23)
  mm is the minute of the hour (00-59)
  SS is the second of the hour (00-59)
  xxx if present, is a fractional second and may 
  be any length

  Time is expressed in GMT.

  SIZE OF FILE (SIZE)

  This command returns the size of the file in bytes named in its
  argument field. This value depends on the TYPE of transfer. 

  Automatic Restart mechanism for interrupted downloads:

  In conventional ftp clients if a transfer is interrupted a 
  restart is possible using the restart command followed by the 
  get call.(REST followed by RETR).The client program finds out
  the size of the local interrupted file and issues a 
  " REST <size of local file>" to the server and then uses the
  RETR <file>. However we assume that the file at the server end
  has not undergone any modifications.Some ftp clients try to use 
  a mechanism for ensuring that the server side file has not
  been modified after the partial transfer has been carried out. 
  They use the modtime command (i.e MDTM) and compare the 
  local file's creation time with the file at the server side.
  If there is a modification at the server of the file after the 
  local transfer we find that the file modification time is 
  greater than than the local file modification time hence we 
  have to transfer the entire file again. However if the time 
  difference is less we can proceed with the restart as mentioned
  above.Also additional checking is done to ensure that the size 
  of the partially transferred file is less than the original file.

  A new mechnism as descibed below is being proposed which 
  automates the restart process. To implement the restart 
  mechanism for an ftp client the commands the ftp server should
  support are MDTM,SIZE and REST. Before the actual transfer 
  occurs the client first requests the server the size of the
  remote file (SIZE <remote file>) , the modification time of the 
  remote file (MDTM <remote file>). The client program then gives 
  the request for the download of the file (RETR <remote file>). 
  If the transfer is interrupted say by the user or due to the
  connection getting broken a footer is written in the partially 
  transferred local file which will now be called the local 
  shadow file in our further discussions.

  The format and contents of the footer are as follows 
  - size of remote file
  - modification time of remote file
  - modification time of partially transferred local file 

  The local shadow file is of the form "Ftp$<Local File Name>$.
  For example if we are downloading a remote file "test.c" and 
  storing in local file "test1.c".If the transfer is interrupted
  the local shadow file will be "Ftp$test.cpp$". 

  We consider the problem by assuming that after interruption of 
  the file transfer process client gets the details of the 
  modification time of the partially transferred local file i.e 
  time 'T'. Now it writes the footer containing the size
  of the remote file , modification time of the remote file
  (whose details it got before the start of the file transfer) 
  and the modification time of the partially transferred local
  file whose details it had just now got. After writing the footer
  the modification time of the partially transferred file with 
  footer written is not 'T' but 'T + t' where 't' is a small
  interval which is the difference of the modification time of
  partially transferred local file with footer written and the 
  modification time of the local file it had written in the 
  footer. 

  Now we have to ensure that the modification time of the local 
  shadow file is same as that of the modification of the local 
  file we had written in the footer. So we update the modification 
  time stamp of the local shadow file to be equal to that of the 
  modification time we had written in the footer ( eg. using 
  utime system call in unix).

  Now consider the case when we again try to download the same 
  remote file and try to store it in a local file having the same 
  local name as that used in the previous interrupted ftp process.
  The ftp client first checks if there exists a file having a 
  file name of the format "Ftp$<Local File Name>$".Note that this 
  format is the same format used to create the local shadow file
  for a interrupted download. If such a file exists it might be 
  possible that this file is a partially downloaded file with a 
  valid footer. So we read the last 'n' bytes of the file where
  'n' is the size of the footer and form the structure of
  the footer whose details are mentioned above. We now get the 
  size and modification time of the remote file compare it with 
  the data got from the footer. We also get the modification time
  of the local file and compare it with the modification time of 
  the partially downloaded file which is present in the footer.If
  the data in footer does not match with the details got we
  conclude that the file has to be downloaded from beginning of 
  the file. However if the data in footer matches with the data 
  we have got we issue a restart command(REST) with the restart
  point as 'size of the local shadow file minus the size of the
  footer'. We also have to check for for other possible conditions 
  and the flow chart below represent all the steps.

  ================================================================ 

                        --------------------
                       | Request for a file |
                        -------------------- 
                       /                      \
  ---------------------------------         --------------------
 | Partial transfer only completed |       | Transfer completed |
 | when transfer is interrupted    |        --------------------
  ---------------------------------
               |
   --------------------------------
  | Footer written and file closed | 
   -------------------------------- 
                |
   --------------------------------
  | Updation of modification time  |
  | stamp of local shadow file     |
   --------------------------------

  ================================================================

                   ----------------------------------------
                  | New request for transfer of same remote| 
                  | file to be stored in same local file   |
                   ---------------------------------------- 
                                    |
                    --------------------------------------
                   | Data in footer compared with data got| 
                    -------------------------------------- 
                   /                                    \ 
   ---------------------------         ---------------------------
  | Comparision shows restart |       | Comparision shows complete | 
  | mechanism can be applied  |       | download of file required  |
   ---------------------------          --------------------------
              |                                    |
   -------------------------------          ----------------------
  | Restart point is local shadow |        | Download entire file | 
  | file size minus footer size   |         ----------------------
   ------------------------------- 
              |
   ---------------------------------
  | Issue request with restart point |
   ---------------------------------
         /                           \ 
   --------------------------        -------------------------- 
  | If remote size - present |      | If remote size - present |
  | local size >= footer size|      | local size < footer      |
   --------------------------        -------------------------- 
             |                                    |
  -------------------------------        -------------------------------
 | Write in file from position   |      | Write in file from position   | 
 | local file size - footer size |      | local file size - footer size |
  -------------------------------       | till EOF & Truncate file      | 
                                         -------------------------------

  ======================================================================

  The added advantage in this method is that using this method 
  the following cases which cannot be implemented using the 
  conventional methods can be implemented here. 

  1.) If the local file undergoes modification after the 
  interrupted transfer there is still a positive difference 
  between the modification at server end and modification 
  time at client end due to which the change in the local 
  file cannot be recorded. 

  2.) If the remote file is modified and also the local file is 
  modified but after the remote file has been modified. 

  The ftp clients now available donot have any fool proof 
  mechanism for a automatic restart for interrupted downloads. 

  INFO command

  A new command "INFO" is being proposed. The syntax of the "INFO" 
  command is 

  INFO SP PATH CRLF

  where PATH may be a name of a remote file. Also wild cards can be 
  used to represent multiple files. The INFO command is used to get 
  information about a single or a group of remote files.Also the 
  client program can write the output of the input command in a 
  file if the user gives an extra argument. Ex: INFO A.TXT TEST 
  writes the information about a.txt in a file named 'test'. The 
  information is of a specific format and resembles the output from 
  the "stat" command in unix. The format of output for a request 
  using "INFO" command is given below.

  ===============================================================

  213 Total : [no of files matching required request field]

  (the below info is got for each matching file) 

  NAME : [name of file]
  TYPE : [file type code] ([file type name])
  OWNER : [owner id] ([user name])
  PERMISSIONS : [XXX] 
  SIZE : [Fle size in bytes]
  MODTIME : [File modification time]
  LINKS : [No oflinks to file]

  ===============================================================

  If no files are matching the pattern of the argument given with 
  INFO command the output will be as follows.

  550 No such file or directory

  If permission is not available to get details of files we get the 
  output as follows

  550 Permission denied 

  ================================================================== 

  TYPE : The codes for the file type with the file type name in 
  brackets in the format used in the info reply are as follows.

  1 (TEXT)
  2 (IMAGE)
  3 (BINARY)
  4 (DIRECTORY)
  5 (DEVICE)
  6 (SPECIAL)
  //OTHER CODES CAN BE GIVEN
  20 (UNKNOWN)

  PERMISSIONS : The file permission field consists of 3 digit with 
  the first digit giving details of the file permissions for the 
  owner , the second digit giving details of the file permissions 
  for the members of the group and third digit contains details of 
  file permissions for everyone. Read is represented by 4 , Write 
  by 2 and Execute by 1 as in unix. So [641] represents read and 
  write rights for owner , read rights for members of the group and 
  execute permissions for others. If the permission details for 
  either the owner, group members or others cannot be known the 
  digit is replaced by 9.

  MODTIME : The file modification time is obtained in the format of 
  the output of MDTM command described above.

  The output of the INFO command can be understood by the following 
  examples.

  'C>' represents the request at client side.
  'S>' represents the response from 

  ====================================================================

  Example 1. 

  C>INFO a.c

  S>213 Total : 1

  NAME : a.c [name of file]
  TYPE : 1 (TEXT) [file type code and file type name in bracket]
  OWNER : 231 (sobhan) [owner id followed by user name]
  PERMISSIONS : 700 [1st for owner 2nd for group 3rd for others]
  SIZE : 1200
  MODTIME : 19990226114512 
  LINKS : 2

  226 Transfer complete 

  =================================================================

  ================================================================

  Example 2.
  ---------

  C>INFO i*

  S>Total : 2

  NAME : inet.cpp [name of file]
  TYPE : 1 (TEXT) [file type code and file type name]
  OWNER : 231 (sobhan) [owner id followed by user name]
  PERMISSIONS : 700 [1st for owner 2nd for group 3rd for others]
  SIZE : 8200
  MODTIME : 19990321103225 
  LINKS : 1

  NAME : india [name of file]
  TYPE : 4 (DIRECTORY) [file type code and file type name]
  OWNER : 21 (kalyan) [owner id followed by user name]
  PERMISSIONS : 777 [1st for owner 2nd for group 3rd for others]
  SIZE : 1204
  MODTIME : 19980112061521 
  LINKS : 1

  226 Transfer complete 

  ================================================================

  Example 3.
  ---------

  C>INFO ftpd

  S>550 No such file or directory

  ================================================================

  Example 4.
  ---------

  C>INFO *.pl a

  This command writes the output of INFO command into local file 
  "a". On success we get message "226 Transfer complete" . If there 
  is a error we get "550 " followed by error statement like "550 
  Disk full" , "550 No such file or diectory" etc.

  ================================================================

  RECURSIVE DOWNLOADS USING INFO 

  We can standardize the process of recursive downloading of direc-
  tories using INFO command. In the file transfer protocol as 
  defined by RFC 959 . It is not possible to download a directory 
  using RETR. If a user now gives "get <file name>" at client 
  program the client first gets information about the file using 
  INFO command and gets to know about the type of the file. If the 
  file type is a directory then the client program proceeds as 
  follows. 

  1. Create the directory and change current working directory at 
  the client side to the directory created. Change the current 
  working directory at server side to the directory requested. 

  2. Request the information about files in the directory using 
  "INFO *" and save it in a file "TempInfo$1$". Start 
  downloading files by reading file information from 
  "TempInfo$1$" and issuing "RETR <file name>".

  3. If you encounter a file with the file type as "DIRECTORY" in 
  the "TempInfo$1$" file , repeat the process as mentioned from 
  step 1 but save the output from "INFO" command in file which 
  follows the format "TempInfo$" followed by the depth of our 
  subdirectory with respect to original directory and follow by
  "$". For exmaple: If a file of type directory is encountered 
  in the file "TempInfo$1$" then the output for INFO command 
  in next step will be stored in "TempInfo$2$". 

  4. After all the files as read from the output file from INFO 
  downloaded delete the INFO output file . 

  5. If the file that has just been deleted is "TempInfo$1$"
  then the recursive download process has been completed.

  6. If the file that has just been deleted is not "TempInfo$1$" 
  but is a INFO output file of greater depth like "TempInfo$2$"
  "TempInfo$3$" etc then we change the current working directory 
  at both client and server sides by one step backward.Continue 
  downloading files using the names after the directory name 
  which has just been downloaded. If all files are downloaded 
  folow pocedure from step 4. If a file of type directory is 
  encountered follow the procedure as from step 1.

  All the steps given above can be represented by the following 
  flow chart. 

  FLOW CHART FOR RECURSIVE DOWNLOADING USING INFO

                 -------------------------------- 
                | DOWNLOAD REQUEST BY FTP CLIENT |
                 -------------------------------- 
                            |
                   ------------------------------
                  | FTP CLIENT SENDS INFO <file> |
                  |    AND KNOWS FILE TYPE       |
                   -------------------------------
                       /              \
   ---------------------            -----------------
  | FILE TYPE DIRECTORY |         | FILE TYPE OTHER | 
   ---------------------           ----------------- 
            |                              |
   ---------------------------      -----------------
  | Create the new directory. |    | Download file   | 
  | Change current directory  |     -----------------
  | at both client & server.  |
   --------------------------
            |
   ---------------------------
  | Get the info of all files |
  | in the directory and save |
  | them in local info file   |
   ---------------------------
              |
   ----------------------------------
  | Parse contents of local info     |
  | file & get file name & file type |
   ---------------------------------- 
                |
   ---------------------------------------------
  | Proceed as above depending on the file type |
   --------------------------------------------- 

  =================================================================

  RESTART MECHANISM USING INFO

  The INFO command can be used to implement the restart mechanism 
  which has been described above. Before downloading a file the 
  client gets the INFO about the remote file using INFO command. If 
  the transfer is aborted in the middle the client writes the 
  footer in the partially transferred file. The remote file size 
  and modification time which is obtained by the INFO command is 
  used in the footer. Thus the use of INFO comand obviates the 
  usage of MDTM and SIZE . Also the extra information that we get 
  from INFO like file permissions can be used for processing on the 
  client side rather than the server side.

  POTENTIAL APPLICATIONS OF INFO

  The usage of INFO command obviates many commands like SIZE, MDTM, 
  MLST,LIST and STAT (INFO without arguments can be made to give 
  output similar to that got by STAT with no arguments) that are 
  widely used in the file transfer protocol implemenatations. The 
  user name that is obtained from INFO can be used to regulate 
  users on the client side . The restart mechanism and recursive 
  downloading mechanism can be implemented with INFO . The usage of 
  INFO now enables validation to be carried out at client side 
  rather than at server side. 

  ================================THE END=========================

Get your FREE Email at http://mailcity.lycos.com
Get your PERSONALIZED START PAGE at http://my.lycos.com

---

From: Stephen Tihor <[email-masked]>
Date: Mon, 27 Sep 1999 12:24:35 -0400

> to FTP.I would be glad to know your comments on the draft.

Well in general, the examples are all too UNIX centric.  

Half of it is
an implementation which sounds more like an implementation note FYI item 
than text for a real RFC.  If put into the FTP RFC as a suggestion or example
or the like I would stull want the code to be complete and generic. 

The proposed implementation stores data in a file system in ways that are not
appropriate if (a) its hard to backup in a file or (b) any user anyplace has
file names starting with $FTP$.  A better implementation would store this
information in a parallel file structure or in the O/S supplied spaces for 
in tree notations or in-tree with explicit code to handle local/remote file 
name translation, probably a quoting mechanism.

Also I note the algorithm proposed assume the file system is a single acyclic 
tree.  Recent work on FTP, especially the listing proposals include the key 
unique facts needed to handle that.   

Also this work seems not to reflect any of the draft and near draft work in
this mailing lists archives.  A perusal of them should allow a more targetted
proposal that focuses on things missing or additional standard facts needed.

---

From: Alun Jones <[email-masked]>
Date: Mon, 27 Sep 1999 17:25:06 -0400

>to FTP.I would be glad to know your comments on the draft.

Main impression is that it's a larger output even than MLST, doesn't have
configuration to set or return what items will or even _can_ be listed
(number of links?  always 1 on FAT!  Owner doesn't exist on FAT), and of
course replicates much of what MLST is trying to do [but then, MLST also
replicates much of what EPLF was trying to do, and merely achieved
popularity over EPLF.]

MLSD is certainly able to do the recursive directory listing that you're
looking for, in almost exactly the manner you describe.  This'll perhaps be
a little ruthless to say, but I'm sure there are others here thinking it -
MLST/MLSD is so close to being final (as far as I can tell) that any entry
right now has to be unbelievably stellar in order to get anything more than
a cursory glance from many of us.  There are already implementations of
MLST/MLSD in test right now.  I don't see enough in INFO that would cause
me to investigate it more, rather than continuing with pulling bugs out of
the MLST implementation I've got.

[Please note - I'm not taking issue with the re-inclusion of the MDTM, REST
and SIZE specs, and calling that a replication of MLST - IMHO, they should
have been spun off anyway into a separate RFC or BCP, but there you are.]

Alun.
~~~~

--
Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find it
1602 Harvest Moon Place | at web site http://www.wftpd.com or email
Cedar Park TX 78613     | us at [email-masked].  VISA / MC accepted.
Fax +1 (512) 378 3246   | NT based ISPs, be sure to read details of
Phone +1 (512) 378 3246 | WFTPD Pro, NT service version - $100.
*WFTPD and WFTPD Pro now available as native Alpha versions for NT*