Subject: [ftpext] COMB command IETF draft proposal

From: Robert Oslin <[email-masked]>
Date: Wed, 8 Jun 2011 11:14:47 -0500

"Any submission to the IETF intended by the Contributor for publication as all or part of an IETF Internet-Draft or RFC and any statement made within the context of an IETF activity is considered an "IETF Contribution".

It only took me 10+ years, but here's my contribution the FTP ext community (attached).

Essentially an early draft to ratify the commonly used COMB command for support of multi-part (a.k.a accelerated) uploads. I welcome comments/questions from the community. Ideally redlining the document (or via comments in the doc). I will resubmit using formal IETF draft (in ASCII) once I've addressed all comments/questions in the MS Word version (hope that's ok and doesn't break and IETF rules).

Thanks,

Robert Oslin
Director of Product Management
Tel: 1 (210) 293-7902
Fax: 1 (210) 690-8824
Send me large files securely<https://robertoslin.cutesendit.com/>

www.globalscape.com<http://www.globalscape.com/>

   (NYSE Amex:GSB)<http://www.globalscape.com/cgi-bin/links.cgi?page=gsb>

*This communication, including attachments, is for the exclusive use of the addressee and may contain proprietary, confidential or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this communication and destroy all copies.

---

From: Robert McMurray <[email-masked]>
Date: Wed, 8 Jun 2011 19:11:07 +0000

Thanks, Robert.

I like the idea behind the COMB command; I could see where this could be quite useful for transferring large files.

One consideration that I have in your draft is verbiage that "The server-FTP process SHOULD delete each of the parts identified by the COMB command once the target file is created." I would think that the cleanup behavior for partial files would be left to the client, since the remaining parts of the draft place the responsibility on the client for splitting the file and keeping track of the parts. For example:

      C> STOR foo.txt
      S> 226 Transfer complete.
      C> STOR bar.txt
      S> 226 Transfer complete.
      C> COMB foobar.txt foo.txt bar.txt
      S> 250 COMB command successful.
      C> DELE foo.txt
      S> 250 DELE command successful.
      C> DELE bar.txt
      S> 250 DELE command successful.

As you mentioned in your draft, it would be advantageous if COMB was used with the HASH command to verify data integrity before the client takes care of the deletion tasks. For example:

      C> OPTS HASH MD5
      S> 200 MD5
      C> STOR foo.txt
      S> 226 Transfer complete.
      C> STOR bar.txt
      S> 226 Transfer complete.
      C> COMB foobar.txt foo.txt bar.txt
      S> 250 COMB command successful.
      C> HASH foobar.txt
      S> 213 MD5 426f62526f636b73 foobar.txt
      C> DELE foo.txt
      S> 250 DELE command successful.
      C> DELE bar.txt
      S> 250 DELE command successful.

Thanks again.

Robert McMurray
[email-masked]

------------------------------
From: Robert Oslin [mailto:[email-masked]] 
Sent: Wednesday, June 08, 2011 9:15 AM
To: [email-masked]
Subject: [ftpext] COMB command IETF draft proposal

Thanks,

Robert Oslin
Director of Product Management
Tel: 1 (210) 293-7902
Fax: 1 (210) 690-8824
Send me large files securely

www.globalscape.com
    (NYSE Amex:GSB) *This communication, including attachments, is for the exclusive use of the addressee and may contain proprietary, confidential or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this communication and destroy all copies.

---

From: Robert Oslin <[email-masked]>
Date: Wed, 8 Jun 2011 14:59:27 -0500

Thanks for the feedback! Issuing a hash (or XCRC until HASH is formally ratified) makes a lot of sense and will be inserted as a strongly recommended practice (i.e. SHOULD). As for the delete operation, I would be careful to make it mandatory of the client, rather leaving it open to interpretation, as there are many ways for the server to execute the combine operation, which may or may not result in the original parts being available for deletion. For example a rudimentary concatenate operation by the server on ascii type files (which would be quite slow) would result in each part being removed by nature of the concatenation operation, hence a subsequent delete operation would result in 5xx errors returned to the client on each DELE request. Perhaps it would be wise to recommend that the client check for parts post operation and do cleanup if any parts remain? In our particular implementation (and in the handful of servers that support COMB so far), it is the server that does the cleanup as a function of the combine operation. Let me know if that makes sense, or if not, then how to deal with the myriad ways in which servers may go about combining files and how to deal with corner cases.

Robert Oslin

-----Original Message-----
From: Robert McMurray [mailto:[email-masked]] 
Sent: Wednesday, June 08, 2011 2:11 PM
To: Robert Oslin; [email-masked]
Subject: RE: [ftpext] COMB command IETF draft proposal

Thanks, Robert.

      C> STOR foo.txt

      C> OPTS HASH MD5
      S> 200 MD5
      C> STOR foo.txt

Thanks again.

Robert McMurray
[email-masked]

Thanks,

Robert Oslin

www.globalscape.com

---

From: Van Glass <[email-masked]>
Date: Wed, 8 Jun 2011 14:15:29 -0600

I think that having the server perform the delete tasks is preferred
for the simple reason that this would reduce the number of commands
going across the command channel.  We have not yet implemented COMB
but would like to pursue it.  With that said I'm curious to know your
experience COMBing a large number of files or very large files.  Have
you experienced any issues where the COMBine process on the server
side takes a long amount of time, possibly resulting in a command
channel timeout on the client side?

Thanks,

Van Glass
JSCAPE
Managed File Transfer and Network Solutions
http://www.jscape.com

On Wed, Jun 8, 2011 at 1:59 PM, Robert Oslin <[email-masked]> wrote:
> Thanks for the feedback! Issuing a hash (or XCRC until HASH is formally ratified) makes a lot of sense and will be inserted as a strongly recommended practice (i.e. SHOULD). As for the delete operation, I would be careful to make it mandatory of the client, rather leaving it open to interpretation, as there are many ways for the server to execute the combine operation, which may or may not result in the original parts being available for deletion. For example a rudimentary concatenate operation by the server on ascii type files (which would be quite slow) would result in each part being removed by nature of the concatenation operation, hence a subsequent delete operation would result in 5xx errors returned to the client on each DELE request. Perhaps it would be wise to recommend that the client check for parts post operation and do cleanup if any parts remain? In our particular implementation (and in the handful of servers that support COMB so far), it is the server that does
>  the cleanup as a function of the combine operation. Let me know if that makes sense, or if not, then how to deal with the myriad ways in which servers may go about combining files and how to deal with corner cases.
>
> Robert Oslin
>
> _______________________________________________
> ftpext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ftpext
>

---

From: Robert Oslin <[email-masked]>
Date: Wed, 8 Jun 2011 15:57:24 -0500

"Have you experienced any issues"

- yes, with larger (multi-GB) size files; however there are ways to combine the file so that timeouts can be avoided in most cases. I made a couple of fleeting references to those methods in the spec, but really it would be up to the server vendor to research and implement the optimal recombine technique so that timeouts could all but be eliminated for everything but exceedingly large files... and even that could be avoided by the client implementing logic that extends or temporarily ignores timeouts when waiting for the result of a COMB operation, perhaps extending the timeout proportional to the file size that was uploaded (same could apply for HASH operations on very large files). I can update the draft to make note of such cases and recommended workarounds; however I wouldn't make any mandatory requirements out of those. BTW, one possibility is for the server to return 200 OK once the operation is approved but not yet consummated, i.e. making it more of an asynchronous command, but that may lead to even more problems down the road (like if client decides to do something with the resulting file even though it has not yet been fully created/recombined).

Robert 

-----Original Message-----
From: Van Glass [mailto:[email-masked]] 
Sent: Wednesday, June 08, 2011 3:15 PM
To: Robert Oslin
Cc: Robert McMurray; [email-masked]
Subject: Re: [ftpext] COMB command IETF draft proposal

I think that having the server perform the delete tasks is preferred for the simple reason that this would reduce the number of commands going across the command channel.  We have not yet implemented COMB but would like to pursue it.  With that said I'm curious to know your experience COMBing a large number of files or very large files.  Have you experienced any issues where the COMBine process on the server side takes a long amount of time, possibly resulting in a command channel timeout on the client side?

Thanks,

Van Glass
JSCAPE
Managed File Transfer and Network Solutions http://www.jscape.com

On Wed, Jun 8, 2011 at 1:59 PM, Robert Oslin <[email-masked]> wrote:
> Thanks for the feedback! Issuing a hash (or XCRC until HASH is 
> formally ratified) makes a lot of sense and will be inserted as a 
> strongly recommended practice (i.e. SHOULD). As for the delete 
> operation, I would be careful to make it mandatory of the client, 
> rather leaving it open to interpretation, as there are many ways for 
> the server to execute the combine operation, which may or may not 
> result in the original parts being available for deletion. For example 
> a rudimentary concatenate operation by the server on ascii type files 
> (which would be quite slow) would result in each part being removed by 
> nature of the concatenation operation, hence a subsequent delete 
> operation would result in 5xx errors returned to the client on each 
> DELE request. Perhaps it would be wise to recommend that the client 
> check for parts post operation and do cleanup if any parts remain? In 
> our particular implementation (and in the handful of servers that 
> support COMB so far), it is the server that does

---

From: Lothar Kimmeringer <[email-masked]>
Date: Thu, 09 Jun 2011 09:34:15 +0200

Am 08.06.2011 22:57, schrieb Robert Oslin:
> - yes, with larger (multi-GB) size files; however there are ways to combine the
 > file so that timeouts can be avoided in most cases. I made a couple of fleeting
 > references to those methods in the spec, but really it would be up to the server
 > vendor to research and implement the optimal recombine technique so that timeouts
 > could all but be eliminated for everything but exceedingly large files...

I have to find a way to open docx files so I might repeat something
that is already supposed but a simple way would be to respond with
the equivalent of a HTTP-100-response:

C: COMB filea fileb filec
S: 250-Start COMB
    250-COMB for filea finished
    250-COMB for fileb finished
    250-COMB for filec finished
    250 COMB command successful

Since a client should in general start over with a timeout every time
a line of the response has been received, it should help avoid timeouts
with very large files. In addition to that you receive some feedback,
that the server is still talking with you.

Regards, Lothar
-- 
Lothar Kimmeringer                E-Mail: [email-masked]
                PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
                  questions!

---

From: Van Glass <[email-masked]>
Date: Thu, 9 Jun 2011 08:07:30 -0600

To expand on Lothar's idea, perhaps this could be achieved using
positive preliminary (1XX) replies.

ttp://en.wikipedia.org/wiki/List_of_FTP_server_return_codes

e.g.

C> COMB "dest" "part1" "part2"
S> 1XX part1 complete
S> 1XX part2 complete
S> 250 success

Van Glass
JSCAPE

---

From: Van Glass <[email-masked]>
Date: Thu, 9 Jun 2011 08:12:51 -0600

scratch that last commment ... forgot that 1XX replies are only on per
command sent by client.

Van Glass
JSCAPE
Managed File Transfer and Security Solutions
http://www.jscape.com

---

From: Richard Koenning <[email-masked]>
Date: Thu, 09 Jun 2011 17:18:07 +0200

>     250 COMB command successful

How do you know already after the first partial file that the combining 
of the files will be ultimately successful? (The reply code of the first 
and the last line of the reply must be the same, see RFC 959, p. 36.)

With the SIZE command we have already the same problem (at least 
implementors have it, who don't limit the SIZE command to cases where 
the file stat information delivers already the answer.)

Ciao,
Richard
-- 
Dr. Richard W. Könning
Fujitsu Technology Solutions GmbH

---

From: "Ángel González" <[email-masked]>
Date: Thu, 09 Jun 2011 17:29:07 +0200

Robert Oslin wrote:
>
> "Any submission to the IETF intended by the Contributor for 
> publication as all or part of an IETF Internet-Draft or RFC and any 
> statement made within the context of an IETF activity is considered an 
> "IETF Contribution".
>
I still have some cuteftp binary ~12 years old :)

> Essentially an early draft to ratify the commonly used COMB command 
> for support of multi-part (a.k.a accelerated) uploads. I welcome 
> comments/questions from the community. Ideally redlining the document 
> (or via comments in the doc). I will resubmit using formal IETF draft 
> (in ASCII) once I've addressed all comments/questions in the MS Word 
> version (hope that's ok and doesn't break and IETF rules).
>
Libreoffice opens it fine, but the Table of contents is garbled, and 
there's a footer 'Oslin Expires 
666666000000FailFailFailFailFailFailDecemberDecemberDecemberDecemberDecemberDecember 
9/06/11 6201192009201020102011' which I doubt is present in the original 
file.

I would change the text
>
> in the rare case COMB is used as an append rather than a create 
> operation, which is why a single (source) part pathname is supported.
>
to
 > in such case COMB appends the partial pathnames to the existent file.

Regardign the Client or Server delete question:
a) Which is the behavior of current implementors?

b) IMHO "The client MUST check that no partial files are left after they 
are no longer needed."
Note that some interrupted upload could have left temporary files, so 
the client should always check. I'd suggest in the rfc using some 
specific pattern that the can application can easily recognise, maybe 
even containing the hash of what's supposedly there.

So if connecting a new session, the client detects a file called 
partial-8418bfdb63bc5d6ccc434b33d6017cd4.tmp it could prompt the user 
about deleting temporary files it has found. But also, on resuming a 
temporary upload, that same file could be reused without needing to 
reupload it (eg. the connection dropped when uploading the 7th piece, on 
next run it can skip the first 6 blocks), by knowing which piece it is 
(from the hash) and verifying that the content indeed matches (with HASH 
command).
Another option would be basing the temporary names on the target name, 
eg: mybigfile.iso.1 mybigfile.iso.2... Same resuming capabilities are 
available, by knowing the file sizes (and the block size it uses when 
splitting) and usage of the hash command.

I would provide the server the ability of deleting files after COMB, 
specially in cases where copying the final file would exceed the quota, 
in which case it COULD generate the big file and remove the partial ones.

I think there should be an error code for "I won't perform the action in 
this mode (text)" for cases where combining would involve some kind of 
transliteration.

---

From: Lothar Kimmeringer <[email-masked]>
Date: Thu, 09 Jun 2011 21:04:50 +0200

Am 09.06.2011 17:18, schrieb Richard Koenning:
 > of the files will be ultimately successful?

That happens if you write emails before drinking your first coffee ;-)
You're correct, that way it can't work but I'd like to keep that direction.
How about working in a similar way when opening a data connection. There
the server sends two responses instead of one.

C: COMB filea fileb filec
S: 150-Start COMB
    150-COMB for filea finished
    150-COMB for fileb finished
    150-COMB for filec finished
    150 COMB finished
    250 COMB command successful

Or in case of an error:

C: COMB filea fileb filec
S: 150-Start COMB
    150-COMB for filea finished
    150-COMB for fileb finished
    150 COMB finished
    552 Requested file action aborted: Exceeded storage allocation

So in general more or less what Van said (after reading his
mail again).

Regards, Lothar

---

From: Robert Oslin <[email-masked]>
Date: Thu, 9 Jun 2011 14:10:47 -0500

This assumes one method of recombining the file where each part is sequentially tacked on to the next, e.g. open, seek, write, open, seek, write, ad nauseam. There are likely OS and File System specific techniques for combining multiple chunks (parts) into a single file that may only be able to provide an all or nothing "finished" vs. a partial success for each chunk appended in a sequential write operation. I'll provide more on that in a bit (currently writing my reply to the various comments)...

-----Original Message-----
From: Lothar Kimmeringer [mailto:[email-masked]] 
Sent: Thursday, June 09, 2011 2:05 PM
To: [email-masked]
Cc: Robert Oslin; [email-masked]
Subject: Re: [ftpext] COMB command IETF draft proposal

 > of the files will be ultimately successful?

That happens if you write emails before drinking your first coffee ;-) You're correct, that way it can't work but I'd like to keep that direction.
How about working in a similar way when opening a data connection. There the server sends two responses instead of one.

Or in case of an error:

So in general more or less what Van said (after reading his mail again).

Regards, Lothar

---

From: Robert Oslin <[email-masked]>
Date: Thu, 9 Jun 2011 16:05:42 -0500

Angel, I agree with text changes.

I've answered (b) and (c) in a separate message.

From: Ángel González [mailto:[email-masked]]
Sent: Thursday, June 09, 2011 10:29 AM
To: Robert Oslin
Cc: [email-masked]
Subject: Re: [ftpext] COMB command IETF draft proposal

Robert Oslin wrote:

Essentially an early draft to ratify the commonly used COMB command for support of multi-part (a.k.a accelerated) uploads. I welcome comments/questions from the community. Ideally redlining the document (or via comments in the doc). I will resubmit using formal IETF draft (in ASCII) once I've addressed all comments/questions in the MS Word version (hope that's ok and doesn't break and IETF rules).
Libreoffice opens it fine, but the Table of contents is garbled, and there's a footer 'Oslin Expires 666666000000FailFailFailFailFailFailDecemberDecemberDecemberDecemberDecemberDecember 9/06/11 6201192009201020102011' which I doubt is present in the original file.

I would change the text

in the rare case COMB is used as an append rather than a create operation, which is why a single (source) part pathname is supported.
to

b) IMHO "The client MUST check that no partial files are left after they are no longer needed."
Note that some interrupted upload could have left temporary files, so the client should always check. I'd suggest in the rfc using some specific pattern that the can application can easily recognise, maybe even containing the hash of what's supposedly there.

So if connecting a new session, the client detects a file called partial-8418bfdb63bc5d6ccc434b33d6017cd4.tmp it could prompt the user about deleting temporary files it has found. But also, on resuming a temporary upload, that same file could be reused without needing to reupload it (eg. the connection dropped when uploading the 7th piece, on next run it can skip the first 6 blocks), by knowing which piece it is (from the hash) and verifying that the content indeed matches (with HASH command).
Another option would be basing the temporary names on the target name, eg: mybigfile.iso.1 mybigfile.iso.2... Same resuming capabilities are available, by knowing the file sizes (and the block size it uses when splitting) and usage of the hash command.

I would provide the server the ability of deleting files after COMB, specially in cases where copying the final file would exceed the quota, in which case it COULD generate the big file and remove the partial ones.

I think there should be an error code for "I won't perform the action in this mode (text)" for cases where combining would involve some kind of transliteration.

---

From: Anthony Bryan <[email-masked]>
Date: Thu, 9 Jun 2011 17:14:12 -0400

> the equivalent of a HTTP-100-response:

it would help to have a text version available.

> that the server is still talking with you.

for HASH we have this:

   Depending on multiple conditions, the final server response to a HASH
   command could take long time, so a server could output a "213-" line
   every 5-10 seconds to avoid the connection being idle and silent.

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Robert Oslin <[email-masked]>
Date: Thu, 9 Jun 2011 16:18:57 -0500

Thank you (all) for the feedback. It appears there three concerns so far:

1.       Timeouts

2.       Deletion

3.       Integrity checking
First let me state that it is my desire to keep the scope of the COMB command RFC as narrow as possible by leaving application specific logic to the client or server application, and dealing only with protocol specific issues.

Timeout is one example of application level logic that should be handled by the client in whichever manner it deems best, much like client implementations must do today when dealing with long running operations, such as waiting for the result of a custom command, hash operation, directory listing, etc. It is up to the client vendor to provide the necessary mechanisms and logic for specifying timeout values, when those values are ignored or temporary extended, and how to deal with a timeout when/if one occurs. It should suffice to mention in the RFC that client vendors should take into consideration long running combine operations and design their client solutions accordingly.

Deletion is also an application level decision that should be completely up to the server implementation. Why? Because whether or not parts are left over largely depends on the method employed by the server to combine the file, which may vary between server implementations on the same or different operating systems and/or underlying file systems. Therefore I believe it would be more correct to only require that the combine command be issued to the server indicating the client's intention, and if permissions allow, the server would then be solely responsible for the process of reassembling the file, and if necessary, deleting, archiving, or doing nothing with the constituent parts as a by-product of the reassembly process (see notes on sparse files - bottom). Keep in mind the example in the draft of the overall.log and today.log that were appended using the COMB command. In that case it may be desirable for the server to do nothing with the parts once the COMB takes place (i.e. leave all log parts intact), or perhaps archive the log parts after the combine operation. Regardless of the server's decision, a 200 OK reply should be returned to the client once the entire operation (as defined by the server) has completed successfully.

Integrity checking is a best practice and should certainly be recommended; however it should not be mandatory, just as hash validation is not mandatory for any other file creation or append operations today.

Some of the discussion has centered on whether the server should reply with a 250 for each part recombined, with the stated goal of "help[ing] avoid timeouts with very large files ". There are three reasons I recommend against this approach.  The first is because it only defers the problem, as extremely large files can have constituent parts that are in the multi-GB size range. Hence the client's timeout threshold could be exceeded even while waiting for one of the parts to be combined. The second reason was already stated (top), which is that it should be up to the client logic to determine how best to deal with long (or potentially long) running operations. The third reason is because we have no idea what method the server will use to combine the parts, whether sequential (and slow) method or using advanced techniques such as sparse files, which brings me to my next point and answer to Ángel 's question on the behavior of current implementations.

Our implementation of COMB matches the draft. I.e. our client uploads each part to the server (multiple parallel SEND commands), then sends the COMB command, after which the server combines the parts into the destination file, and then it (the server) deletes the parts and sends a 200 reply to the client. The client then (optionally) checks the integrity of the file using our XCRC command, another proprietary command that will likely be replaced by the HASH command. Unfortunately we (for historical reasons beyond this discussion) implemented a rather crude technique for combing the source parts into the destination file, essentially using a sequential copy of each source file part to the new (destination) file at a certain byte offset, which can take a horribly long time if each part is  "large". The problem with this isn't so much the potential for client timeouts (easily dealt with as mentioned earlier), but rather the adverse effect of this long running operation in that it negates the time gained from transferring the file in multiple parts to begin with! In some rare cases it can take longer to combine the file than it took to upload the same - again, this is not by fault of the COMB command logic, but rather our poor choice of technique for combining the file on the server.

To that extent we have researched alternate methods for combining the file that will all but eliminate this problem (and by nature and possible timeouts). Those include the use of sparse files and similar techniques, which likely exist for non-Windows operating systems as well. This brings me to my final point, which is that I missed something very important in the draft that that will require a bit of draft re-work...  tomorrow or Monday I'll write up a summary of that change and why I think it helps augment the current draft so that vendors can use advanced techniques for the combine process if they so choose that may not be possible by just using the COMB command, as those advanced techniques require foreknowledge of certain information (byte size and number of parts) prior to receiving/writing the parts data, in order to work properly. More to follow.

Robert Oslin
GlobalSCAPE

---

From: Lothar Kimmeringer <[email-masked]>
Date: Thu, 09 Jun 2011 23:51:21 +0200

Am 09.06.2011 21:10, schrieb Robert Oslin:
> This assumes one method of recombining the file where each part is sequentially
 > tacked on to the next, e.g. open, seek, write, open, seek, write, ad nauseam.
 > There are likely OS and File System specific techniques for combining multiple
 > chunks (parts) into a single file that may only be able to provide an all or
 > nothing "finished" vs. a partial success for each chunk appended in a sequential
 > write operation. I'll provide more on that in a bit (currently writing my reply
 > to the various comments)...

What the specific text after "150-" doesn't matter, it can also be
150-I'm still here, please wait
or whatever you (the server) want. The only important thing is
that something is given out at all that allows to reset the
timeout on the client side.

So if the server is working sequntially it might give out some
kind of information that helps seeing some progress, but it
might be something completely different like a poem ;-)

In principle this technique is already in use, e.g. with "tarpits"
(http://en.wikipedia.org/wiki/Tarpit_%28networking%29).

Regards, Lothar

---

From: Robert Oslin <[email-masked]>
Date: Thu, 9 Jun 2011 16:57:15 -0500

" So if the server is working sequntially it might give out some kind of information that helps seeing some progress, but it might be something completely different like a poem ;-)"

Agreed. Sending the 150 at regular intervals (and not dependent on some predetermined event, such as part1 combined), would work nicely.

A server NOOP if you will.

Robert

-----Original Message-----
From: Lothar Kimmeringer [mailto:[email-masked]] 
Sent: Thursday, June 09, 2011 4:51 PM
To: Robert Oslin
Cc: [email-masked]; [email-masked]
Subject: Re: [ftpext] COMB command IETF draft proposal

Am 09.06.2011 21:10, schrieb Robert Oslin:
 > There are likely OS and File System specific techniques for combining multiple  > chunks (parts) into a single file that may only be able to provide an all or  > nothing "finished" vs. a partial success for each chunk appended in a sequential  > write operation. I'll provide more on that in a bit (currently writing my reply  > to the various comments)...

What the specific text after "150-" doesn't matter, it can also be 150-I'm still here, please wait or whatever you (the server) want. The only important thing is that something is given out at all that allows to reset the timeout on the client side.

So if the server is working sequntially it might give out some kind of information that helps seeing some progress, but it might be something completely different like a poem ;-)

Regards, Lothar

---

From: Richard Koenning <[email-masked]>
Date: Fri, 10 Jun 2011 15:07:05 +0200

As Van Glass already pointed out in an answer to his own similar 
proposal only one 1yz message is allowed per command (see RFC 959, 
p.37). Even when RFC 959 is not engraved into stone one should think 
twice before violating/changing rules layed down in this RFC.
Best regards,
Richard

---

From: Robert Oslin <[email-masked]>
Date: Fri, 10 Jun 2011 08:53:52 -0500

I totally missed/forgot about that: "The server-FTP process may send at most, one 1yz reply per command.", which again gives favor to the argument that the client app should take care of / handle timeouts, which can be done through configuration by the user or by adapting timeout thresholds dynamically. 

We should still include a recommendation (using "SHOULD") that the server offer up a (single) 1xx reply at some point in the combine process, but even so this (timeouts) may not be a problem if the server is "smart" about how it combines the file upon receipt.

Thanks,

Robert Oslin

-----Original Message-----
From: Richard Koenning [mailto:[email-masked]] 
Sent: Friday, June 10, 2011 8:07 AM
To: [email-masked]
Cc: Robert Oslin; Lothar Kimmeringer
Subject: Re: [ftpext] COMB command IETF draft proposal

As Van Glass already pointed out in an answer to his own similar proposal only one 1yz message is allowed per command (see RFC 959, p.37). Even when RFC 959 is not engraved into stone one should think twice before violating/changing rules layed down in this RFC.
Best regards,
Richard

Robert Oslin wrote:

> So if the server is working sequntially it might give out some kind of 
> information that helps seeing some progress, but it might be something 
> completely different like a poem ;-)
>

---

From: Lothar Kimmeringer <[email-masked]>
Date: Thu, 16 Jun 2011 18:39:47 +0200

Am 10.06.2011 15:07, schrieb Richard Koenning:
 > p.37). Even when RFC 959 is not engraved into stone one should
 > think twice before violating/changing rules layed down in this RFC.

There is a different between Van Glass' proposal and mine:

Van Flass:

Mine:

In my example there is only one 150-response seperated into
more than one lines.

Best regards,

Lothar Kimmeringer

---

From: Richard Koenning <[email-masked]>
Date: Thu, 16 Jun 2011 19:05:00 +0200

Well, at the first look it seems a little bit like rape of RFC 959, but 
currently i don't know anything which would speak against this solution.
Best regards,
Richard Könning

-- 
Dr. Richard W. Könning
Fujitsu Technology Solutions GmbH, TSP ES&S SWE OS 5
Phone/Fax: +49-89-3222-2927 / -3222-329-2927
E-Mail: [email-masked]

---

From: John C Klensin <[email-masked]>
Date: Fri, 17 Jun 2011 22:31:23 -0400

--On Wednesday, June 08, 2011 11:14 -0500 Robert Oslin
<[email-masked]> wrote:

>...
> Essentially an early draft to ratify the commonly used COMB
> command for support of multi-part (a.k.a accelerated) uploads.
> I welcome comments/questions from the community. Ideally
> redlining the document (or via comments in the doc). I will
> resubmit using formal IETF draft (in ASCII) once I've
> addressed all comments/questions in the MS Word version (hope
> that's ok and doesn't break and IETF rules).

As at least one other person has said, it doesn't break any
rules, but ASCII is a lot easier for many of us to deal with and
comment on, if only out of habit.

To put what I'm about to say in context, I usually take the
position of an FTP purist whose relationship with the protocol
goes all the way back to some of the original design meetings.
That makes me inclined to respond to a lot of proposals with
"that isn't FTP but an attempt to graft another protocol onto
it; why don't you use something else".   That doesn't quite
apply in this case, but...

A few comments about your spec that I don't think others have
made:

(1) It is seriously unclear and needs work to describe what you
intend.  For example, if you say "recombine a file from its
constituent parts" or "instruct the server to recombine the
previously uploaded file parts", it could be read as requiring

  STOR /f.p1
  STOR /f.p2
  STOR /f.p3
  STOR /f.pr
  COMB "/file.dat" "/f.p1" "/f.p2" "/f.p3" "/f.p4"

I think you intend that COMB cause the transfer and then the
combining function, but that is not what the document appears to
say.  Or, given

	"It is up to the user-FTP process to determine when, if,
	and how to split up a file into two or more parts,
	upload each part as a unique file over multiple parallel
	connections, retransmit one or more..."

maybe the above is exactly what you do intend (one could
substitute STOU, which would have some nice advantages, but, if
that were your intention, it is wholly missing from the spec.

(2) You need to be extremely careful with concepts like "file
path" and "directory path".  They are not defined in FTP the way
your description of COMB seems to assume.

(3) Your use of quotes is un-FTP-ish and your command makes
assumptions about the syntax of a name that is not local to the
working directory on the server-FTP.  I'd rather you didn't do
it at all, but, if you must, the specification has to identify
how you would express a file name on the server-FTP system that
was required to actually be, e.g., 
   file/".dat
Your U**x-ish assumptions would probably call for 
  COMB "/file\/\".dat" ...
or
  COMB "/file\/"".dat" ...
but the spec isn't clear and both are error-prone.

Note that the second is seriously pathological given your
"spaces not needed" rule.

(4) Remembering that we originally intended FTP to be
asynchronous between the command and data streams, it may be too
late now, but the FTP-ish way to do this probably would have
been something like:

  CWD wherever-you-want-this-to-end-up
  BOMPT   (Begin-Odd-Multipart-Transfer)
  PASV  (or PORT, here and below)
  STRU local-name
  PASV
  STRU local-name
  PASV
  STRU local-name
  ...
  COMBine remote-name

The STRU commands would presumably all get 2yz replies
specifying the name used or failure codes permitting something
else to be done.  The server would be required to return those
STRU replies in order even if the transfers finished out of
order.

Note that not only specifies completely separate and parallel
data connections (rather than maybe trying to share a single TCP
data stream), but, by using initiation and competition commands
and letting the server keep track of its own part names,  it
completely eliminates the quotes, hoping the path syntax on the
local or remote systems matches whatever you've assumed, and, by
using STRU, eliminates both the need to be sure that the part
names chosen on the sender machine are available on the receiver
one and a small potential security problem.  It also permits use
of REST, etc., on the individual streams if needed.

The use of STRU in that context would also permit an intelligent
FTP-server to use some type of pool (or scratch or reserved
temporary) space for the intermediate files, copying only the
final, reassembled, file into the target directory.  That would
eliminate several of the operational or security risks you
discuss.  Arranging to disassemble the file into pool space on
the client machine (or using the interleaved block approach
discussed below) would eliminate a number of others.

Of course, in a fully asynchronous environment, a sequence of
APPE commands would give a very close approximation of the
above.  And that is how what you are looking for now was
actually done for a while, long ago.

FWIW, the above model could work rather well in the download
direction, elegantly eliminating the need for odd tricks with
the checkpoint/restart mechanism.

All of that said, experience with a number of
distributed/parallel storage systems, including some distributed
database update models and RAID striping, would suggest that the
most transfer-efficient way to do what you are trying to
accomplish would be to use a completely different model:  treat
the file to be transferred as a sequence of blocks of specified
size, open a bunch of connections, push block 1 onto connection
0, block 2 onto connection 1, and continue with each block, M,
going out onto connection (M modulo number of connections).
That would not only permit more optimization but would permit
reassembly to start while the transfer was still in progress.

(5) Some of the material in 3.3 don't make any sense.  If the
server "SHOULD" or "MAY" send the specified responses, it seems
to me that is obligatory on you to specify what happens if the
server chooses to do something else.  Presumably it is not open
season for anything you don't specify -- I'd predict severe
interoperability problems if a server responded to your 550 case
by returning, e.g., 987.

(6) The first two paragraphs of your Security Considerations
section are a little bogus.  Providing COMB support only to
authorized users, etc., may protect against certain classes of
malicious attacks, but they provide absolutely no protection
against ignorance, stupidity, or carelessness, any of which can
be easily exploited by an attacker as well as creating risks of
equally-damaging accidents.

best,
   john

---

From: Robert Oslin <[email-masked]>
Date: Mon, 20 Jun 2011 15:33:22 -0500

>>ASCII is a lot easier ...

Noted.

>>I usually take the position of an FTP purist whose relationship with the protocol goes all the way back to some of the original design meetings.

I greatly appreciate the academic "FTP purist" viewpoint. I am coming at this from a purely tactical and practical angle, i.e. we have millions of users and businesses worldwide using our FTP client and server products that are regularly performing multi-part uploads. My goal is to formalize (and potentially improve) the process so that other vendors can extend this existing benefit to their users, ideally in a non-proprietary fashion, much like our proprietary XCRC command which has been used for performing integrity checking for years and is now finally being formalized as the HASH command (albeit with many improvements) by Mr. Anthony Bryan, for the benefit of the entire FTP client/server vendor and user community.

>>I think you intend that COMB cause the transfer and then the combining function

Originally that was not my intent. Presently COMB is nothing more than an operation for combining a unique set of files. My original intent was for the client process to determine whether to perform a multi-part transfer and for the client to use its own programmatic logic to determine byte offsets and what not, and then issue multiple parallel STOR commands for each part over discreet data channels. As far as the server was concerned these were unique (whole) file uploads. Only the client knew that those uploads were actually parts of larger file or not. See notes on (4) for more information.

>>You need to be extremely careful with concepts like "file path" and "directory path".

Agreed.

>>(3) Your use of quotes is un-FTP-ish and your command makes assumptions about the syntax of a name that is not local to the working directory on the server-FTP.

Thanks. This may be a moot point though. See notes on (4)

>>Note that the second is seriously pathological given your "spaces not needed" rule.

Agreed.

>>(4) Remembering that we originally intended FTP to be asynchronous between the command and data streams, it may be too late now, but the FTP-ish way to do this probably would have been something like:

I don't think it is too late, given the early stages of this draft, but yes from a practical standpoint it would require that vendors who are using COMB in its current incarnation to update their products; however it may be possible to both improve upon the design and grandfather the current design.

>>The STRU commands would presumably all get 2yz replies specifying the name used or failure codes permitting something else to be done.  The server would be required to return those STRU replies in order even if the transfers finished out of order.

Unfortunately my knowledge of the STRU command is quite limited. I don't see (in any RFC) where a local-name value can be provided as an argument. Perhaps you mean STOU (store unique), or is there a way in which STRU is used that I'm totally missing?

>>Note that not only specifies completely separate and parallel data connections (rather than maybe trying to share a single TCP data stream), but, by using initiation and competition commands and letting the server keep track of its own part names,  it completely eliminates the quotes, hoping the path syntax on the local or remote systems matches whatever you've assumed, and, by using STRU, eliminates both the need to be sure that the part names chosen on the sender machine are available on the receiver one and a small potential security problem.  It also permits use of REST, etc., on the individual streams if needed.

I agree with the concept of initiation and completion commands. That was what I was referring to when I wrote in a prior message "I missed something very important in the draft that that will require a bit of draft re-work". I believe a preparatory command is needed in order to alert the server of the client's intent, although my motives are slightly different from yours: I wanted to provide the server with all necessary information needed to (optionally) employ advanced combine techniques, such as streaming the combine process vs. post transfer combine. Here is what I was going to propose, except that I failed to consider STOU, which appears superior to STOR in that STOU eliminates the need for temporary filenames managed by the client (instead, as you indicated, allowing the 2zy reply could include the temporary unique name managed by the server.

Example:
  CWD wherever-you-want-this-to-end-up
  BMPT   remote-name, number-of-parts, original-size-in-bytes
  PASV  (or PORT, here and below)
 STOU  local-name (chunk1)
  PASV
  STOU local-name (chunk2)
  PASV
  STOU local-name (chunk3)
  ...
  COMB remote-name

Here is how my proposed logic differs (but only slightly) from yours:

1. Use of four letter command name for the prepare portion, call it BMPT, PREP, or whatever.

2. The prepare command needs to provide sufficient information to the remote server so that advanced file combine techniques can be used, whether use of scratch or reserved temporary space, or sequence of blocks, or whatever. In our case (since we are thinking sparse files), the server needs to know the name of the destination file, the number of bytes, and total parts that will be transferred. With this information the server can deduce each of the part sizes that will subsequently be sent by the client, and therefore where the server should begin writing each byte offset within the sparse file. For other techniques there may be more or less information required; however I cannot think of any additional parameters aside from size and number of expected parts at the moment.

3. STOU instead of STRU used (which is what I believe you meant).

4. The one concern I have with NOT providing the client-managed individual part names in the preparatory command or in each STOR command is that it is possible for the order of the parts to be received out of sequence. Remember that the client will likely open up multiple data streams in parallel fashion (or nearly so), and the server might not handle them in the order received, hence the server would not know at which byte offset to begin writing each part received (i.e. is this part the beginning of the sparse file, somewhere in the middle, end of file, etc.). I believe the way to overcome this problem is for the client to not begin sending each subsequent part until it has received a 1xx reply from the server indicating that the server has accepted the STOU command, opened a data channel, and recognized it as part X of Y (Y being the number of parts communicated in the BMPT command). Once the 1xx reply is received the client would spawn a new thread and issue the next STOU command. Ideally (depending on latency and such), each part would be initiated within a second or two from each prior part and only after the initiation of the transfer has been recognized by the server. What I have NOT addressed in this email is problems such as a started MP transfer but not all parts are sent, aborted transfers, and inter-session transfers (started MP but lost connection and some parts still remain or were partially transferred).

5. The other problem is when the client has multiple multi-part and standard (single-part) transfers initiated in parallel. I.e. what happens if I send two files using 8 parts and another smaller file using no parts (standard transfer) at the same time. This is a likely scenario, given current client's multi-threaded capabilities.  What we don't want is the server confusing transfers and assigning wrong parts to wrong files. Yes, keeping the STOU <filename> commands identical for each unique source filename makes a lot of sense, but how does the server know which BMPT command to associate with the subsequent STOU commands? Assuming parallelism but showing both logs combined, in chronological order:

BMPT  blah.dat, 4, 2235478
BMPT  blah2.dat, 4, 2431234
STOU A.dat
STOU A.dat
STOU B.dat
STOU A.dat
STOU B.dat
STOU A.dat
STOU B.dat
STOU B.dat
COMB blah.dat
COMB blah2.dat

Notice here that the server will have received sequential (back to back) BMPT (preparatory) commands, followed by back-to-back STOU commands. There needs to be a way for the server to identify which batch of STOU commands belong to which BMTP command.

I think the way to solve this problem is to require that the <Filename> value provided in the preparatory command (BMPT in this example) match the filename provided in the STOU commands (and vice versa). The user could always rename the file later if desired, e.g.

BMPT A.dat, 4, 2235478
BMPT B.dat, 4, 2431234
STOU A.dat
STOU A.dat
STOU B.dat
STOU A.dat
STOU B.dat
STOU A.dat
STOU B.dat
STOU B.dat
COMB A.dat
COMB B.dat
RNFR A.dat
RNTO blah.dat
RNFR B.dat
RNTO blah2.dat

Let me make one more point here: Depending on the combine techniques used bye server, the COMB command may not be of any use other than to indicate to the server that it (the client) is done transferring all parts, kind of like a "hey, I'm done now, did you get everything and combine ok?", to which the server would reply in the affirmative if for example the sparse file had filled out and all bytes were accounted for. If the server received the command but something was missing or there were others errors somewhere along the combine process, the server would reply with a transitive negative completion reply (hey, send me the last part again), or permanent negative completion reply (the bytes received don't match what the preparatory command said I was going to receive, therefore COMB failed"). Likewise the COMB command could retain it's original syntax for backwards compatibility, i.e. if the server receives a COMB command that wasn't preceded by a preparatory BMPT command, then it will expect the COMB command to supply all necessary parameters (dst name, part1, part2, etc.) which means I still need to solve the naming/syntax issues.

Here is an example of a complete sequence (less CWD and PASV/PORT mode operations) that helps summarize what I've stated so far:

On Command Channel
u> BMPT A.dat, 4, 2235478
s> 200 OK, ready for MP transfer
...(wait for 2yz on the various data channels)
u> COMB A.dat
s> 200 Combine successful

On Data Channel 1
u> STOU A.dat
s> 150 Data connection opened for A.dat as part 1 of 4, ready for next part
s> 226 Transfer complete

On Data Channel 2
u> STOU A.dat
s> 150 Data connection opened for A.dat as part 2 of 4, ready for next part
s> 226 Transfer complete

On Data Channel 3
u> STOU A.dat
s> 150 Data connection opened for A.dat as part 3 of 4, ready for next part
s> 226 Transfer complete

On Data Channel 4
u> STOU A.dat
s> 150 Data connection opened for A.dat as part 4 of 4
s> 226 Transfer complete  <--client waits for last part to get 226 reply then command channel sends COMB to confirm all is well and recombined.

>>The use of STRU in that context would also permit an intelligent FTP-server to use some type of pool (or scratch or reserved temporary) space for the intermediate files, copying only the final, reassembled, file into the target directory.  That would eliminate several of the operational or security risks you discuss.  Arranging to disassemble the file into pool space on the client machine (or using the interleaved block approach discussed below) would eliminate a number of others.

Agreed, which is why sending a preparatory command is so important. We just need to make sure that all necessary information is provided so that intelligent FTP servers can use different techniques for combining the files, while making sure we avoid potential problemd associated with parallel multi-part transfers and out-of-sequence sending of parts.

>>FWIW, the above model could work rather well in the download direction, elegantly eliminating the need for odd tricks with the checkpoint/restart mechanism.

Yes it could. Currently we use multiple REST commands at various offsets to accomplish this, but from an FTP purists perspective, that was ever the intended manner for REST to be used. I just don't know if I want to tackle MP downloads in this draft, given that this problem is already solved, albeit in a non-elegant manner.

>>All of that said, experience with a number of distributed/parallel storage systems, including some distributed database update models and RAID striping, would suggest that the most transfer-efficient way to do what you are trying to accomplish would be to use a completely different model:  treat the file to be transferred as a sequence of blocks of specified size, open a bunch of connections, push block 1 onto connection 0, block 2 onto connection 1, and continue with each block, M, going out onto connection (M modulo number of connections).
That would not only permit more optimization but would permit reassembly to start while the transfer was still in progress.

This is how sparse files work and why it is necessary for the client tell the server how many parts and total size of bytes (so that it can perform the M modulo number of connections calculation on the server side). http://en.wikipedia.org/wiki/Sparse_file

>>(5) Some of the material in 3.3 don't make any sense.  If the server "SHOULD" or "MAY" send the specified responses, it seems to me that is obligatory on you to specify what happens if the server chooses to do something else.  Presumably it is not open season for anything you don't specify -- I'd predict severe interoperability problems if a server responded to your 550 case by returning, e.g., 987.

Noted.

>>(6) The first two paragraphs of your Security Considerations section are a little bogus.  Providing COMB support only to authorized users, etc., may protect against certain classes of malicious attacks, but they provide absolutely no protection against ignorance, stupidity, or carelessness, any of which can be easily exploited by an attacker as well as creating risks of equally-damaging accidents.

I think it goes without saying that we can never protect fully against user ignorance and stupidity. Heck, the DELE command is pretty efficient at destroying data as well. My notes in #6 are simply the warning labels that remind you not to use your hairdryer in the bathtub. I.e. misuse (or abuse) of the COMB command can lead to file corruption, which is the same (or worse in some cases) than the destruction of data using DELE. I do think these warnings are necessary and appropriate, but may change them a bit to match the proposed logic changes (which are quite substantial).

Thanks John for your feedback. I wish I had thought of the necessary preparatory commands BEFORE submitting my first draft, but I suppose that is what the working group is for - to point out flaws in the design and ways to improve, even if it does mean wholesale changes are required.

I'll wait until I hear from others and if everyone is on board I'll float a revamped draft that includes an initiation (preparatory) and completion command sequence for MP uploads.

Robert