Subject: [ftpext] I-D Action:draft-ietf-ftpext2-hash-00.txt

From: [email-masked]
Date: Wed, 24 Nov 2010 14:45:01 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the FTP Extensions, 2nd edition Working Group of the IETF.

	Title           : File Transfer Protocol HASH Command for Cryptographic Hashes
	Author(s)       : A. Bryan, et al.
	Filename        : draft-ietf-ftpext2-hash-00.txt
	Pages           : 15
	Date            : 2010-11-24

The File Transfer Protocol does not offer any method to verify the
integrity of a transferred file, nor can two files be compared
against each other without actually transferring them first.
Cryptographic hashes are a possible solution to this problem.  In the
past, several attempts have been made to add commands to obtain
checksums and hashes, however none have been formally specified,
leading to non-interoperability and confusion.  To solve these
issues, this document specifies a new FTP command to be used by
clients to request cryptographic hashes of files.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-ftpext2-hash-00.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

---

From: Lothar Kimmeringer <[email-masked]>
Date: Thu, 25 Nov 2010 01:15:22 +0100

Hello,

just being added to the maillist and already having a first question ;-)

> against each other without actually transferring them first.

[...]

> http://www.ietf.org/internet-drafts/draft-ietf-ftpext2-hash-00.txt

I read the draft and tried to find the reason by searching the
archive. Why are startpoint and endpoint (SP and EP) missing?
The options are even mentioned in Appendx B. Adding these
options would e.g. allow to implement distributed downloads where
parts of the same file can be downloaded from different mirror-
servers at the same time checking the hash of each part.

In case you want to add another entry to Appendix B:
Lobster IntegrationServer (supports XCRC, XSHA1, XMD5; all with
SP and EP).

Best regards,

Lothar Kimmeringer
-- 
Lothar Kimmeringer                E-Mail: [email-masked]
                PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
                  questions!

---

From: Anthony Bryan <[email-masked]>
Date: Thu, 25 Nov 2010 18:52:25 -0500

> just being added to the maillist and already having a first question ;-)

welcome, thanks for joining us!

> archive. Why are startpoint and endpoint (SP and EP) missing?

good question!

as you say, some of the other commands (XCRC, XMD5, XSHA1) support
partial file hashing.

I remember this was on our list of known issues, and we authors
brought it up a number of times. we never received any feedback on it
(except for Mat from SmartFTP on his forum who was for it), so it was
dropped.

[for everyone's info, some of the older FTP drafts like HASH that were
around before this mailing list were discussed on apps-discuss (going
back to March for HASH):
http://www.ietf.org/mail-archive/web/apps-discuss/current/maillist.html
]

> servers at the same time checking the hash of each part.

that's exactly what we're doing in Metalink/XML & Metalink/HTTP (part
of the reason we needed HASH). mirrors, hashes, & signatures, are
offered over HTTP headers:
http://tools.ietf.org/html/draft-bryan-metalinkhttp (see section 7.1)

the Content-MD5 header which does partial hashes for HTTP is being
phased out I believe. we have the partial file hashes in the XML and
different ranges can be downloaded from mirrors...

> Lobster IntegrationServer (supports XCRC, XSHA1, XMD5; all with

done! it'll appear in the next version.
-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: "Ángel González" <[email-masked]>
Date: Fri, 26 Nov 2010 01:35:12 +0100

> http://www.ietf.org/mail-archive/web/apps-discuss/current/maillist.html

I also think it is interesting to have hashes of partial content.
They key piece is 'partial verification'. You could have distributed
downloads
from several mirrors without being able to separatedly hash each piece.
Hash the file on all the mirrors (so it's the same), download the pieces,
join, hash the final file. If it's good, ok. If not, redownload
*everything* again.
Which is exactly what you would have to do if you did a normal download from
the ftp.
You download a big file (eg. a DVD iso) and it fails verification (the
server hash
command is analogous to the checksums usually provided in the dowload page).
That file is quite worthless now, even if you know that the error will
probably be
just a couple of bytes.
Let's assume now that the ftp server allos a hash range command. Then,
instead
of redownloading everything, your client starts bisecting the file,
finds a 1K block
where your local copy differs, and redownloading it you're done.

A couple of requirements:
* The user should not be required to know the file size in advance.
* The protocol should allow a server allowing full file hashes but
rejecting partial hashes.

---

From: Anthony Bryan <[email-masked]>
Date: Fri, 26 Nov 2010 03:47:06 -0500

> where your local copy differs, and redownloading it you're done.

yep, the partial file hashes are for download repair.

that's what we use them for in Metalink, but instead of being able to
request the hash for a specific range, the hashes are pre-computed &
stored in XML for (usually 256k but can be whatever size) chunks,
instead of dynamically requested.

as mentioned, Content-MD5 (RFC 1864) requests partial hashes for HTTP,
and Instance Digests (RFC 3230) for full file hashes.

> * The user should not be required to know the file size in advance.

you mean the client? why? it should just assume from a starting point
to the end of the file?

> rejecting partial hashes.

right, partial file hashing would be OPTIONAL.
-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: "Ángel González" <[email-masked]>
Date: Fri, 26 Nov 2010 17:13:45 +0100

> you mean the client? why? it should just assume from a starting point
Sorry, I wasn't clear here. The client should not be required to know
the file size
for getting *the full hash*. If it is intended to convert the full file
hash case into
'partial hash from 0 to filesize', there should be some magic value for
the end of file.

---

From: Anthony Bryan <[email-masked]>
Date: Mon, 29 Nov 2010 21:10:22 -0500

> 'partial hash from 0 to filesize', there should be some magic value for

what would be the reason for that? someone could try to HASH a file
that is still being uploaded very slowly.

so, people seem to be interested in partial file hashing. I think it
is worth considering, even though it makes things more complicated and
there are potential security issues.

any suggestions how to add this to the current draft?
-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads