Subject: Re: [ftpext] A few comments on draft-bryan-ftp-hash-05

From: Tim Kosse <[email-masked]>
Date: Sat, 03 Jul 2010 13:14:02 +0200

Hi,

thank you very much for your extensive feedback Paul.

On 2010-06-30 10:50, Paul Ford-Hutchinson wrote:
> [...]
> C> HASH filename.ext
> S> 213 SHA-1 80bc95fd391772fa61c91ed68567f0980bb45fd9 filename.ext

I like this suggestion.

> [...]

> Where the draft states "the raw octet data of the file", it probably needs 
> to be making fewer assumptions about the filesystem of the server.  A 
> "file" is not necessarily a byte stream.  It may have a record structure 
> for example.  I'd remove that sentence and leave it as "The returned hash 
> MUST be calculated as if a client were to download the full file using 
> TYPE I and MODE S and were to calculate the hash on the received octet 
> data."

Makes sense.

> In which case, one of the replies to the HASH command has to be "5xy I 
> cannot calculate the HASH as I would not be able to deliver that file to 
> you with TYPE I and MODE S"

551 perhaps?

> Should there be an allowance for "private" hash algorithms? not just those 
> defined in the IANA registry named "Hash Function Textual Names" 

Should be fine to allow provided algorithms provided that there is a
prefix to distinguish between registered and private algorithms, X_ perhaps.

Also, it needs to be made clear that the private algorithms are not
interoperable, as two servers might implement two completely different
algorithms using the same private name.

> Section 6:
> 
> [...]
> Not sure that a protocol specification is the right place to place a 
> SHOULD on an implementation decision like this. 
> [...]
> These two SHOULD words should be MAY.  The protocol impact of this is that 
> a client MUST allow that a HASH command might take a reasonably long time 
> to complete.

Agreed.

> Also, if you do start 
> dipping toes into the murky waters of caching, you really need to specify 
> how to update the cached value [...]

What about "A server MAY cache the calculated hashes [...] provided it
updates or invalidates the cached hash when content of the corresponding
file changes."

> [...]
> As a protocol spec, I would re-write something like ...
> 
> "A server may refuse to process a HASH command for many reasons, one of 
> which may be a suspected denial of service attack, a client MUST be able 
> to understand that refusal to process HASH commands may be transient (if 
> indicated by a 4xy response) and MAY be honoured later if the server so 
> decides."

Sounds good.

>> In addition, the HASH command can be used to draw conclusions about
>> the contents of a file.  If the hash of a file on some server matches
>> the hash of some known file, then both files are likely identical.
>> To prevent this scenario it suffices to limit use of the HASH command
>> to users who would already be able to download the file.
> 
> I think the issue of which files a server should process a HASH command 
> for a logged in user for is a bit bigger than this ...
> [...]

Yes, that paragraph needs to be improved.

> - The draft probably needs to state what should happen if somebody 
> attempts to HASH something that isn't a file (e.g. it's a directory)

Ack.

Regards,
Tim

---

From: Anthony Bryan <[email-masked]>
Date: Fri, 9 Jul 2010 19:44:57 -0400

> thank you very much for your extensive feedback Paul.

yes, thanks, Paul!

>> attempts to HASH something that isn't a file (e.g. it's a directory)

the above things are in the -06 draft.

Remaining known issues concerning this draft:

    * Should HASH support partial file hashes, similar to the
Content-MD5 HTTP Header.
    * Section needs improvement: which files should a server process a
HASH command for a logged in user?
    * What should happen if somebody attempts to HASH something that
isn't a file (e.g. it's a directory)?

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Robert McMurray <[email-masked]>
Date: Tue, 13 Jul 2010 22:01:15 +0000

In response to the following question that Anthony had posted:

>>  What should happen if somebody attempts to HASH
>>  something that isn't a file (e.g. it's a directory)?

I have a suggestion - why not return a list of filenames with their associated hashes for the directory using the algorithm that was specified by an earlier OPTS command? For example:

C> HASH /foo/
S> 125 Data connection already open; transfer starting.
S> MD5 F0E1D2C3B4A5968778695A4B3C2D1E0F file1.txt
S> MD5 78695A4B3C2D1E0FF0E1D2C3B4A59687 file2.txt
S> MD5 0F1E2D3C4B5A69788796A5B4C3D2E1F0 file3.txt
S> 226 Transfer complete.

This has obvious benefits when considering an FTP client that is trying to synchronize a list of files between locations, and it's much more reliable than file sizes and date/time stamps. In addition, allowing for the HASH command to be requested for a directory would be much better than trying to implement hashes in the response to a LIST request.

---

From: Tim Kosse <[email-masked]>
Date: Wed, 14 Jul 2010 00:40:32 +0200

Hi,

On 2010-07-14 00:01, Robert McMurray wrote:
> why not return a list of filenames with their associated hashes for the directory [...]? For example:
> 
> C> HASH /foo/

Are you proposing to use a data connection just like with RETR/STOR and
the likes? If so, how to distinguish between getting the hash for a
single file (as sent over the control connection) and getting the hashes
for an entire directory?

Either all data needs to be sent over the transfer connection or the
transfer connection must never be used for HASH, otherwise the client
would not know when to send PORT/PASV... in preparation.

In case everything goes over the transfer connection, I see a problem
with directories containing only a single file with a name identical to
> S> MD5 F0E1D2C3B4A5968778695A4B3C2D1E0F file1.txt

Is it a file named file1.txt in the current directory or a file named
file1.txt in the file1.txt directory? This this getting even more
complicated considering that the FTP specs do not mandate a particular
path syntax.

The other possible solution would be to use a multi-line response.

Last but not least, what should happen if the client issues HASH on a
directory containing over a million files?

Tim

---

From: Robert McMurray <[email-masked]>
Date: Wed, 14 Jul 2010 00:57:52 +0000

Thanks, Tim.

>> why not return a list of filenames with their associated
>> hashes for the directory? For example:
>> 
>> C> HASH /foo/
> Are you proposing to use a data connection just like with
> RETR/STOR and the likes? If so, how to distinguish
> between getting the hash for a single file (as sent over
> the control connection) and getting the hashes for an
> entire directory?
> 
> Either all data needs to be sent over the transfer
> connection or the transfer connection must never be used
> for HASH, otherwise the client would not know when to
> send PORT/PASV... in preparation.

Yes - I am proposing to use the data channel. It's slightly more expensive in terms of resources (e.g. opening the data channel) but the tradeoff is worth it when dealing with large directories, which I'll explain later.

> In case everything goes over the transfer connection, I
> see a problem with directories containing only a single
> file with a name identical to that of the directory:
> 
> Is it a file named file1.txt in the current directory or
> a file named file1.txt in the file1.txt directory? This
> this getting even more complicated considering that the
> FTP specs do not mandate a particular path syntax.

I am suggesting the same behavior that you would expect with a LIST command:

- If you send LIST or HASH with no parameters then the server will assume the current directory.
- If you send LIST or HASH with a single parameter of "foo" then the server determines if "foo" is a file or a subdirectory of the current directory and then responds accordingly.
- If you send LIST or HASH with a single parameter of "/foo/bar/" then the server knows that you are requesting a path that is based off the server's root path and responds accordingly.

To answer your question more directly, only files hashes are returned in the response to a HASH command, so a line in the server response that listed "MD5 F0E1D2C3B4A5968778695A4B3C2D1E0F file1.txt" would mean a file named "file1.txt" in the current directory; this is the same as if the client sent a "LIST file1.txt" command that would return the directory listing for that file in the current directory. Currently a client has to keep track of the current working directory when it sends a LIST command, and there would be no difference for HASH. The LIST command has the additional overhead of enumerating directories in a response, but HASH would have no need to do that, so there is no confusion as to whether the server response meant a file or a directory.

> The other possible solution would be to use a multi-line

That is certainly a possibility, but I would think that the data channel would work better. (And once again it would be consistent with commands like LIST.)

> Last but not least, what should happen if the client issues
> HASH on a directory containing over a million files?

This is where issuing a HASH command for directory really helps out - you currently have the same situation with the LIST command, meaning that you can issue a LIST command for a directory with a million files and then your server has to respond with that directory listing - there would be no difference with the HASH command, except for the additional overhead for computing hashes if they are not already pre-hashed or cached somehow by the FTP server implementation. But think about trying to synchronize a client and server with a million files - the client will need to get a directory listing from the server to get the list of files that exist at each location, then issue a HASH command for _every_ file individually (which is an enormous amount of client/server round-trips) instead of issuing a single HASH command for the whole directory.

Thanks again.

---

From: Paul Ford-Hutchinson <[email-masked]>
Date: Thu, 15 Jul 2010 07:56:29 +0100

This is why I asked the question in the first place, I think this 
discussion needs to happen and the group needs to agree on a decision.

Robert McMurray wrote on 14/07/2010 01:57:52:

> >> hashes for the directory? 

Why not? - because it is complicated and, arguably unnecessary (MLST and 
repeated HASH can achieve the same result, agreed with different network 
overhead)
 
> Yes - I am proposing to use the data channel.

Which is a major change from the existing proposal where the response is 
sent on the control connection.

Whatever we decide, we must agree that 'HASH filename' and 'HASH dirname' 
must operate in the same manner with respect to whether the reply comes 
back on control or data.  To have different mechanisms depending on what 
the type of thing you are HASHing is (which a client doesn't necessarily 
know) is clearly wrong.

> I am suggesting the same behavior that you would expect with a LIST 
command:

Which is certainly possible.  Perhaps one way forward is to create another 
I-D defining a new HLST command which specifies just this.

> > The other possible solution would be to use a multi-line

Personally, I don't like this idea - it's creating yet another pattern for 
the FTP protocol to use.

> > HASH on a directory containing over a million files?

And this is where I see the real issue.  We are already having a 
discussion about server resource and the potential of DoS attacks by the 
use of the HASH command.  To compound this by allowing a single command to 
HASH 'millions' of files is surely even more contentious.  If we do think 
it's necessary, and we also agree that the 'The server MAY refuse to 
calculate a HASH for any reason including "I am too busy right now"' - 
then how to convey this on a file-by-file basis as part of a directory 
HASH command is a non-trivial issue.

I propose that this draft specifically excludes HASH over a directory 
(i.e. returns a 4xy error) and, if there is felt to be a need for a 
multi-file HASH command, create a new draft defining an HLST command that 
does just that.  The main reason is to keep the focus on the work done so 
far with HASH and not open a whole new can of worms at this stage. 

I believe that to try to create a single draft with both approaches is 
asking for IETF trouble.

Paul

---

From: Tim Kosse <[email-masked]>
Date: Sun, 25 Jul 2010 22:41:38 +0200

> HASH command is a non-trivial issue.

"Too busy right now" is just one of many possible reasons. Others
include permission denied, filesystem error, harddisk failure and so on.
All these would have to be communicated on a file-by-file basis.

There is another huge problem that would definitely arise if we were
using the data connection to return results: Firewalls and
(consumer-grade) NAT routers. I had lots of problems with those when I
switched FileZilla to use MLSD to obtain directory listings. These
products fail to understand the semantics of new commands (as expected)
but do not filter the FEAT response. So a data connection based hash
command be very problematic.

> (i.e. returns a 4xy error) 

Yes. Intuitively I would have implemented HASH without directory
support, but I concur that it needs to be explicitly mentioned in the draft.

Tim

---

From: Anthony Bryan <[email-masked]>
Date: Mon, 26 Jul 2010 14:30:33 -0400

> support, but I concur that it needs to be explicitly mentioned in the draft.

the draft says this already:

"The pathname argument MUST represent a file path, not a directory path."

what should we add?

further, Tim adds that this would be a permanent negative completion
(5yz) instead of transient negative completion (4yz) and suggests 553.

---

From: Paul Ford-Hutchinson <[email-masked]>
Date: Tue, 27 Jul 2010 11:37:09 +0100

> "The pathname argument MUST represent a file path, not a directory 
path."

Oops, must have missed that (even though I was looking for it).  Maybe 
resurrect the <blink> tag :-)

In my defence I would expect to see this restriction on the server 
behaviour and not on the content of a client created value (the client may 
have no idea that /x/y/z is a directory and not a file).  i.e. I would 
expect to see it in section 3.4 (where I don't spot 553)

> (5yz) instead of transient negative completion (4yz) and suggests 553.

The reason I put a 4xy was that path /x/y/z might be a directory one day 
and a file the next.  Revisiting the discussion on RFC959, I agree that it 
should be 5xy.  (55y to be more precise)

Cheers,
Paul

---

From: Anthony Bryan <[email-masked]>
Date: Tue, 27 Jul 2010 16:20:06 -0400

> resurrect the <blink> tag :-)

hehe :)

> In my defence I would expect to see this restriction on the server behaviour
> and not on the content of a client created value (the client may have no
> idea that /x/y/z is a directory and not a file).  i.e. I would expect to see
> it in section 3.4 (where I don't spot 553)

we'll add this in the next draft in 3.4:

"The server-PI SHOULD reply with a 553 reply if the user requests the
HASH of a directory, which is not allowed."

do you think we should have more clarifications elsewhere? I guess
it's implicit, but the draft mentions file, file, and doesn't mention
directory except to say that it's not for that. if it doesn't
communicate that well, maybe we should make it more explicit.

> The reason I put a 4xy was that path /x/y/z might be a directory one day and
> a file the next.  Revisiting the discussion on RFC959, I agree that it

great.