Subject: draft-bryan-ftp-hash-00 -- items for discussion

From: Alfred Hönes <[email-masked]>
Date: Thu, 1 Apr 2010 21:54:26 +0200 (MESZ)

I have been asked to respin on apps-discuss a summary of concerns
raised in the pre-draft stage for draft-bryan-ftp-hash-00 and
communicated to the authors after draft submisison.

Here we go ...

-------- forwarded content of message to the authors -------

It looks like the concerns raised so far have not been adressed.
[ Thus, the first part of Appendix C, which pretends "no known issues",
  sounds ironical.  :-)  ]

In particular:

a) untypical use of a command with a file path argument *and*
   another argument -- consider using:
        OPTS HASH <hash>
        HASH <filepath>

b) underspecification of the representation of the file that shall
   undergo the hash calculation; a canonical form of the file is
   essential here for interoperability; I already have discussed
   some of the options that came to my mind: using the current
   transfer mode settings, defining a specific overriding set of
   mode settings, or requiring a single specific set of mode
   settings to be active (or a small choice of such).
   I cannot imagine how verification would be possible without
   a well defined canonical file representation in a heterogenous
   systems environment where e.g. one system uses record-oriented
   file storage and another one uses byte-stream storage with either
   <NL> or <CRLF> convention, one system pads records with blanks
   to an alignment boundary and another system truncates trailing
   blanks on file lines, ... (all that behavior exists in my office!)

Further, I'd like to draw your attention to RFC 5797 and the
FTP Commands and Extensions IANA Registry it has established now, at
  <http://www.IANA.ORG/assignments/ftp-commands-extensions/
          ftp-commands-extensions.xhtml> ).
So you might wish to amend your proposal with substantial IANA
considerations for the registration of the proposed extension and
the associated FEAT strings.

Also, it might make sense to give reference to prior work;
I'm aware of an old expired I-D, draft-twine-ftpmd5-00, May 2002.

Typos:
-  Abstract:   s/intergrity/integrity/
-  Sect.1:  s/from a FTP server/from an FTP server/

Kind regards,
  Alfred Hönes.

--

+------------------------+--------------------------------------------+
| TR-Sys Alfred Hoenes   |  Alfred Hoenes   Dipl.-Math., Dipl.-Phys.  |
| Gerlinger Strasse 12   |  Phone: (+49)7156/9635-0, Fax: -18         |
| D-71254  Ditzingen     |  E-Mail:  [email-masked]                     |
+------------------------+--------------------------------------------+

---

From: Anthony Bryan <[email-masked]>
Date: Thu, 1 Apr 2010 17:11:15 -0400

> communicated to the authors after draft submisison.

Thanks, Alfred.

>   blanks on file lines, ... (all that behavior exists in my office!)

We'll work on these.

Other issues Daniel Stenberg has mentioned that we will address:

- Should there be a way to request a partial file hash (similar to
Content-MD5)? [At least one of the nonstandard hash commands supports
this.]

- Correct response code to use for completion and errors.

- Need to include some more general advice on algorithms just in case
algorithm X is found to be broken the day after this draft is
released.
- The FTP server's right to refuse to calculate the hash is of course
important to help against DOS risks.

> the associated FEAT strings.

Ah yes, here's what we have now:

5.  IANA Considerations

This new command is added to the "FTP Commands and Extensions"
registry created by [RFC5797].
Command Name: HASH
Description: Cryptographic Hash of a file
FEAT String: HASH
Command Type: Service execution
Conformance Requirements: Optional
Reference: This specification

> I'm aware of an old expired I-D, draft-twine-ftpmd5-00, May 2002.

Didn't know about that, added it. Apache Ftp Server supports it. I've
also been told that wuftpd supports SITE CHECKSUM.
Here is the expanded list of software that supports a nonstandard FTP
hash type command (XMD5, XSHA1, SITE SHOHASH, etc):

    * Akamai NetStorage
    * Apache Ftp Server (supports draft-twine-ftpmd5)
    * Cerberus FTP server
    * FileCOPA FTP Server
    * FireFTP
    * Gene6 FTP Server
    * GoldenGate FTP (Ftp Full Java Server)
    * IceWarp FTP Server
    * JAFS
    * MOVEit DMZ
    * Nofeel FTP server
    * Null FTP
    * ProFTPD module mod_digest
    * SmartFTP client
    * Starksoft Ftp Component for .NET / Mono
    * RaidenFTPD32 FTP server
    * WS_FTP client / server
    * wuftpd ('SITE CHECKMETHOD' and 'SITE CHECKSUM')
    * zFTPServer

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Tim Kosse <[email-masked]>
Date: Mon, 05 Apr 2010 23:43:24 +0200

Hello,

>    another argument -- consider using:

we are currently preparing a new version of the draft which uses this
proposed syntax.

>    essential here for interoperability
>    [...] some of the options that came to my mind: using the current

Unfortunately neither option is perfect. Considering the most common use
cases however I would go with the latter options, using a specific set
of mode settings.

A related issue are partial hashes. In some situations it might be
useful to only get the hash of a portion of a file. It could for example
be used to find out where two files differ, in order to only transfer
the mismatching part instead of the whole file.

Having two many parameters however would make caching difficult and
might impose heavy load on servers which is not desirable.

>           ftp-commands-extensions.xhtml> ).

The "IANA Considerations" section will be updated in the next version of
the draft.

Regards,
Tim Kosse

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 6 Apr 2010 10:15:35 +0200 (CEST)

On Mon, 5 Apr 2010, Tim Kosse wrote:

> A related issue are partial hashes. In some situations it might be useful to 
> only get the hash of a portion of a file. It could for example be used to 
> find out where two files differ, in order to only transfer the mismatching 
> part instead of the whole file.
>
> Having two many parameters however would make caching difficult and might 
> impose heavy load on servers which is not desirable.

Perhaps it will make sense to allow the server to deny answering to partial 
hashes? Or even include the range in the response so that the client can see 
what data range the checksum is for. It wouldn't be completely strange to me 
to use the REST position as start position for the hash calculation.

-- 

  / daniel.haxx.se

---

From: Tim Kosse <[email-masked]>
Date: Sat, 10 Apr 2010 21:08:05 +0200

On 2010-04-06 10:15, Daniel Stenberg wrote:
> Perhaps it will make sense to allow the server to deny answering to
> partial hashes?

Yes, that should be allowed if there is support for partial hashes.

> Or even include the range in the response so that the
> client can see what data range the checksum is for.

Seems redundant to me, if a client would query a specific range, it
already knows that range.

> It wouldn't be
> completely strange to me to use the REST position as start position for
> the hash calculation.

This would probably limit us to MODE S as the semantics of REST are
somewhat more complex in the other modes.

Also, have a look at how REST is defined in RFC 3659. In particular:

> The REST command must be the last command issued before the data
>    transfer command that is to cause a restarted, rather than a
>    complete, file transfer.  The effect of issuing a REST command at any
>    other time is undefined. [...]
> Because of this, a user-PI that has issued a REST command, but that
>    has not successfully transmitted the following data transfer command
>    for any reason, should send another REST command before the next data
>    transfer command.  If that transfer is not to be restarted, then
>    "REST 0" should be issued.

We would have to extend the semantics of REST in this case.

Note that many clients send the REST command as keep-alive command. As
such changing REST may have unintended side effects with such clients.

Tim

---

From: Daniel Stenberg <[email-masked]>
Date: Sat, 10 Apr 2010 23:02:21 +0200 (CEST)

On Sat, 10 Apr 2010, Tim Kosse wrote:

>> It wouldn't be completely strange to me to use the REST position as start 
>> position for the hash calculation.
>
> This would probably limit us to MODE S as the semantics of REST are somewhat 
> more complex in the other modes.
>

Oh, obviously I hadn't done my home work and retract that idea. I think all 
those additional details make REST a bad choice to (ab)use for partial hashes.

Thanks!

-- 

  / daniel.haxx.se

---

From: Anthony Bryan <[email-masked]>
Date: Fri, 16 Apr 2010 15:01:49 -0400

>   another argument -- consider using:

Our current draft uses this. Thanks for the feedback!

>   blanks on file lines, ... (all that behavior exists in my office!)

Our current draft includes this:

The HASH command is useful for files transmitted in Image type mode
(TYPE I) and Stream transfer mode (MODE S).

Does that address this issue sufficiently?

> the associated FEAT strings.

Added to our current version.

> I'm aware of an old expired I-D, draft-twine-ftpmd5-00, May 2002.

I was able to find 2 servers that support this method.

In total, there are at least nine non-standard FTP checksum/hash commands:
XCRC, MD5/MMD5, XMD5, XSHA, XSHA1, XSHA256, XSHA512, SITE SHOHASH, SITE CHECKSUM

Over 45 FTP servers/clients support these 9 commands.

Also, interested parties might want to read Robert McMurray's blog
post on IIS, "Automatically Creating Checksum Files for FTP Uploads":

"I had a great question in the publishing forums on forums.iis.net,
where someone was asking if FTP 7 supported the XCRC command. The
short answer is that the XCRC command is not supported, but I came up
with a way to create an FTP provider that supports something like it.
"

http://blogs.msdn.com/robert_mcmurray/archive/2009/07/30/automatically-creating-checksum-files-for-ftp-uploads.aspx

And finally, here is some feedback on our draft from those
implementing the nonstandard hash commands.

zFTPServer: Interesting... a nice complement (or even replacement) for
the XCRC and XMD5 commands...
Apache FTP server: This looks good. We should have these defined years
ago, but better late than never!