Subject: Re: [ftpext] New Version Notification for draft-bryan-ftp-hash-05

From: Tim Kosse <[email-masked]>
Date: Sat, 03 Jul 2010 12:15:59 +0200

On 2010-06-30 01:34, Anthony Bryan wrote:
> after reading draft-somers-ftp-mfxx-04, maybe we should import
> pathname from section 2.2 of RFC 3659?

Yes, that makes sense.

Tim

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 12 Jul 2010 11:59:32 +0200

A more efficient and less DoS-prone way to implement this would be for the client to tell the server the hash, and for the server to tell the client whether the file matches the hash.

To optimize this, the hash wouldn't be a straigh hash over the entire file, but consist of three parts: the file length, a hash over a small part of the file (say, the first and last kilobyte) and finally a hash over the entire file. So if the length doesn't match, the file doesn't match the hash and there is no need to access the file. If the hash over the small part doesn't match, there is no match and hashing the entire file is avoided.

Nitpick: repeating the first part of the abstract in the introduction verbatim is a bit annoying.

---

From: Paul Ford-Hutchinson <[email-masked]>
Date: Mon, 12 Jul 2010 23:03:22 +0100

> Iljitsch van Beijnum  wrote on 12/07/2010 10:59:32:
> 
> A more efficient and less DoS-prone way to implement this would be 
> for the client to tell the server the hash, and for the server to 
> tell the client whether the file matches the hash.

Not sure how this reduces DoS - if a client can STOR/RETR a file then they 
know the length and 'mini-hash' and thence force the server to repeatedly 
calculate the full hash.  (And if the server is caching the hash, then 
this isn't a DoS issue in the first place).

> 
> To optimize this, the hash wouldn't be a straigh hash over the 
> entire file, but consist of three parts: the file length, a hash 
> over a small part of the file (say, the first and last kilobyte) and
> finally a hash over the entire file. So if the length doesn't match,
> the file doesn't match the hash and there is no need to access the 
> file. If the hash over the small part doesn't match, there is no 
> match and hashing the entire file is avoided.

my 2c - this is an unnecessary  complication to address a possible threat 
that can be managed by other mechanisms (throttling HASH requests from a 
given client, caching the hash, ...)

Paul

---

From: Anthony Bryan <[email-masked]>
Date: Mon, 12 Jul 2010 18:39:20 -0400

> To optimize this, the hash wouldn't be a straigh hash over the entire file, but consist of three parts: the file length, a hash over a small part of the file (say, the first and last kilobyte) and…

Iljitsch, interesting approach.

I believe one of the draft's earlier incarnations also had a CHKHSH
command which did just this.

unless I'm misunderstanding, this would be MORE indirect, unoptimized,
complex and DoSy than the current draft.

to be able to answer whether a hash is correct, the server still needs
to know the correct hash?
when you meet someone, do you ask "is your name Bob? no, how about
Jim? no, Ed?" or do you just come right out and ask "what's your
name?" which is simpler?

MirrorBrain is an example of a server, although HTTP not FTP, that is
a hash/mirror cache/provider: http://www.mirrorbrain.org/
it also might be worth examining what the other FTP servers that
implement the nonstandard hash commands do.

you mention partial file hashes, some of those nonstandard commands
like XCRC and XMD5 support it with a byte range.
do you think HASH should support partial file hashing?

> Nitpick: repeating the first part of the abstract in the introduction verbatim is a bit annoying.

sure, we'll work on that.

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 13 Jul 2010 01:01:32 +0200

> complex and DoSy than the current draft.

I'll grant you more indirect and more complex. But don't forget the first rule of client-server: let the client do the hard work.

> Jim? no, Ed?" or do you just come right out and ask "what's your

Ah, but the question isn't "what's your name" but something that requires a rather expensive operation, which is always a nice DoS vector. 

> do you think HASH should support partial file hashing?

No, because that way the client could ask the server to do an almost infinite number of hashes on a single file, and it makes it impossible for the server to precompute the hashes.

---

From: Anthony Bryan <[email-masked]>
Date: Mon, 12 Jul 2010 20:12:22 -0400

> I'll grant you more indirect and more complex. But don't forget the first rule of client-server: let the client do the hard work.

from what I understand, your proposal involves potentially more work
on the server?

> Ah, but the question isn't "what's your name" but something that requires a rather expensive operation, which is always a nice DoS vector.

I don't get it...either way, the server and client both have to
perform the expensive operation, right?

the server caches hashes or can limit the use of HASH, as Paul mentioned...

I haven't heard of this as a problem for Instance Digests (8 years
old), Content-MD5 (15 years), or the current 10 nonstandard FTP
hash/checksum commands (9 years), even though I've brought it up
before.

> No, because that way the client could ask the server to do an almost infinite number of hashes on a single file, and it makes it impossible for the server to precompute the hashes.

right, there are reasons against partial file hashes/Content-MD5. does
everyone think they are insurmountable?
I'm personally against it as it involves more work! :)

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 13 Jul 2010 13:10:17 +0200

On 13 jul 2010, at 2:12, Anthony Bryan wrote:

>> don't forget the first rule of client-server: let the client do the hard work.

> from what I understand, your proposal involves potentially more work

If the hashes match then the extra check for the size and the small hash are unnecessary extra work, yes. But you only have to be able to skip hashing a multi-GB file once in a while to make up for that.

> perform the expensive operation, right?

A malicious client could ask the server for hashes for all files it has. In my scheme, the amount of work done by the server is limited to computing the short hash, unless the malicious client has the full file or its full hash, in which case the client can get the server to compute hashes over the entire files.

In the non-malicious case you still get to save work some of the time.

BTW, try this for fun: type md5 <gigabyte+ file> and try to use your computer as normal: load a big web page, start an application.

> hash/checksum commands (9 years), even though I've brought it up

As commercials for financial services keep telling us: past performance doesn't guarantee future performance.

---

From: Richard Koenning <[email-masked]>
Date: Tue, 13 Jul 2010 15:13:19 +0200

> A malicious client could ask the server for hashes for all files it has. In my scheme, the amount of work done by the server is limited to computing the short hash, unless the malicious client has…

The malicious client needs only the short hash to get the server into 
computing the hash of the entire file, because the server recognizes 
only after doing this, that the client hasn't the correct hash of the 
complete file. Therefore i don't see the gain of your method regarding 
DoS attacks.
-- 
Dr. Richard W. Könning
Fujitsu Technology Solutions GmbH