Subject: [ftpext] End byte of transfer in FTP

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Mon, 29 Nov 2010 01:21:08 +0900

Hi,

In FTP, user can set start byte of transfer by REST command.
But it seems it lacks the feature to set end byte of transfer, like
HTTP's Range request.
Does it really lack this feature? Are there any ftp server/client support this?

Issuing ABOR command is workaround but sometimes it does not stop immediately
and unused bytes are transfered.
Another possible workaround is that just closing data connection and
wait for reply
from control connection(I've not tested this yet).

A example of utilization this feature is Metalink download.
In Metalink, it contains several URIs for one file. So we can download it from
several sources. From each server, client downloads non-overlapping segment.
For example, when downloading 1 file using 2 FTP servers.
We can use REST command for second half of the download and it downloads given
 point to the end of file. This is fine. The control connection can be reused.
For the first half of the download, since we cannot set end byte, we
have to close connection
entirely after first half of file is downloaded. The control
connection cannot be reused this time.
We can use ABOR, but it has drawback as I mentioned earlier.

Best regards,

Tatsuhiro Tsujikawa

---

From: John C Klensin <[email-masked]>
Date: Sun, 28 Nov 2010 18:40:49 -0500

> In FTP, user can set start byte of transfer by REST command.

This is a misuse of the intent of the command and some
historical FTP implementations won't permit it.  In particular,
REST is defined only for Block and Compressed modes (neither of
which is widely used these days) and starts only from a
server-specified checkpoint marker.   While I can believe that
some implementations might construe REST followed by an integer
in stream mode as a byte offset to be used in a subsequent STOR
or RETR command, that usage is certainly not guaranteed by
anything in RFC 959.

Note that design is quite intentional and necessary to reflect a
wide variety of different file system organizations, including
files that are not organized sequentially on the remote or local
host.

As the most trivial example, FTP doesn't even guarantee that the
bits of a sequential file will be transferred in the order in
which they appear in their original form.

> But it seems it lacks the feature to set end byte of transfer,
> like HTTP's Range request.

It seems to me that you are looking for a remote file management
protocol that supports a much more specific file abstraction
than is the case with FTP, which is purely and simply a file
transfer protocol for entire files, not fragments of them.  HTTP
does assume a particular type of file organization model, which
is one reason byte-oriented fragment identifiers work.  If that
model serves your needs, why not just use HTTP?   If you need
the more general organizational model of FTP, you will need to
either transfer the entire file and then extract whatever you
need (using whatever local access method is appropriate to that
file) or find/ use/ design a file management protocol, rather
than a file transfer one.

> In Metalink, it contains several URIs for one file. So we can
> download it from several sources. From each server, client
> downloads non-overlapping segment. For example, when
> downloading 1 file using 2 FTP servers. We can use REST
> command for second half of the download and it downloads given
>  point to the end of file. This is fine. The control
> connection can be reused. For the first half of the download,
> since we cannot set end byte, we have to close connection

And, again, why not use HTTP?

    john

---

From: Richard Koenning <[email-masked]>
Date: Mon, 29 Nov 2010 14:49:35 +0100

> or RETR command, that usage is certainly not guaranteed by

REST is defined for STREAM mode, see RFC 3659, p.13.

Best regards,
Richard Könning
-- 
Dr. Richard W. Könning
Fujitsu Technology Solutions GmbH

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Tue, 30 Nov 2010 23:15:35 +0900

Hi,

Thank you for replying!

> which they appear in their original form.

As Richard suggests, REST was extended to STREAM mode in RFC3695.

> And, again, why not use HTTP?

If we can choose protocol, then, yes, we choose HTTP definitely.
But there are existing FTP servers which we cannot make any changes and
we want to include them in Metalink. Especially if the file is only
served in particular
FTP site.

Tatsuhiro Tsujikawa