Subject: Re: [ftpext] End byte of transfer in FTP

From: Daniel Stenberg <[email-masked]>
Date: Wed, 1 Dec 2010 20:02:50 +0100 (CET)

On Mon, 29 Nov 2010, Tatsuhiro Tsujikawa wrote:

> Issuing ABOR command is workaround but sometimes it does not stop 
> immediately and unused bytes are transfered. Another possible workaround is 
> that just closing data connection and wait for reply from control 
> connection(I've not tested this yet).

curl does FTP range requests that way. It used to just close the transfer at 
the desired end point, and it even works fine with "most" servers while the 
remaining servers we've found requires an ABOR for the server to notice that 
the transfer was indeed closed. And after an ABOR like that, it really is hard 
to re-use the control connection safely, so this kind of "range" request 
certainly has its limitations.

-- 

  / daniel.haxx.se

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Sat, 4 Dec 2010 23:55:43 +0900

2010/11/30 Tatsuhiro Tsujikawa <[email-masked]>:
>
> As Richard suggests, REST was extended to STREAM mode in RFC3695.
>

Sorry for typo:
s/RFC3695/RFC3659/

---

From: Anthony Bryan <[email-masked]>
Date: Sat, 4 Dec 2010 17:15:41 -0500

On Wed, Dec 1, 2010 at 2:02 PM, Daniel Stenberg <[email-masked]> wrote:
> the transfer was indeed closed. And after an ABOR like that, it really is
> hard to re-use the control connection safely, so this kind of "range"
> request certainly has its limitations.

would it be possible to modify REST so that if the FEAT reply included RANGE:

      C> feat
      S> 211- <any descriptive text>
      S>  ...
      S>  REST STREAM RANGE
      S>  ...
      S> 211 end

REST could take a starting point and ending point?

if so, I wonder if servers & clients would want to implement that or
if it's too late in the game.

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Sun, 5 Dec 2010 18:26:46 +0900

> REST could take a starting point and ending point?

I don't know it is possible to extend existing command to get another argument.
But if it can, it is reasonable.

If it cannot, then we can make another command, say, RANG(or RNGE?)
which takes starting
and ending point.

> if it's too late in the game.

At least for client, I'd love to implement this extension in aria2;)
I think it is not too late, because there is big difference with or
without standard.
And still FTP is widely used and FTP server code is maintained, so
there is a plenty
of room for this extension.

Best regards,

Tatsuhiro Tsujikawa

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 6 Dec 2010 08:55:50 +0100 (CET)

> would it be possible to modify REST so that if the FEAT reply included 

I don't see much point in that. I'd rather see a new range function or 
possibly one that sets the end point in the same style REST sets the start 
point.

-- 

  / daniel.haxx.se

---

From: John C Klensin <[email-masked]>
Date: Mon, 06 Dec 2010 09:57:51 -0500

--On Monday, December 06, 2010 08:55 +0100 Daniel Stenberg
<[email-masked]> wrote:

> I don't see much point in that. I'd rather see a new range
> function or possibly one that sets the end point in the same
> style REST sets the start point.

Yes.  I'm still not convinced trying to specify fragments of
stream files is a good idea at all but, if it is to be done,
let's do it with a new command and very clearly-specified
semantics.  Remember that, if transfers are done in TYPE I, 

	-- not only can one have any of the usual
	line-separation characters, but one can have other sorts
	of record markers or length-delimited lines or blocks.
	
	-- if Unicode is being sent consistent with the base
	spec), one gets internal storage forms (often UTF-16LE
	or UTF-16BE) not only UTF-8, and certainly does not get
	one octet per character.

So specifying fragments clearly is not a trivial piece of work
unless the new command also requires a new file STRUcture
variation.

      john

---

From: Richard Koenning <[email-masked]>
Date: Mon, 06 Dec 2010 16:34:36 +0100

> unless the new command also requires a new file STRUcture

I don't see a problem on FTP protocol level, with TYPE I the transfered 
data is an unstructured stream of bytes, in which REST currently defines 
a starting point and in which a RANGE specification would additionally 
define an endpoint.
FTP clients and servers operating on files which are not simply a stream 
of bytes but e.g. have some length-delimited record structure may have 
hard work to do, but this problem exists already with the current REST 
command.

-- 
Dr. Richard W. Könning
Fujitsu Technology Solutions GmbH

---

From: Tatsuhiro Tsujikawa <[email-masked]>
Date: Wed, 8 Dec 2010 21:34:32 +0900

2010/12/7 Richard Koenning <[email-masked]>:
> I don't see a problem on FTP protocol level, with TYPE I the transfered data
> is an unstructured stream of bytes, in which REST currently defines a
> starting point and in which a RANGE specification would additionally define
> an endpoint.

Agree.

> FTP clients and servers operating on files which are not simply a stream of
> bytes but e.g. have some length-delimited record structure may have hard
> work to do, but this problem exists already with the current REST command.
>

Agree.
RFC3659 5.1 states about this problem:

In other representation types and structures
 more effort will be required, but it remains always possible to
 determine the offset with finite, but perhaps non-negligible, effort.

The transfer modes other than STREAM mode use marker to specify restart point
and the marker is sent from sending host.
Then we have no way to know the end point marker because it is not sent.
Because of this, I think it is very hard or impossible to define this
RANGE specification
for other than STREAM mode.
Is it possible to define RANGE specification only in STREAM mode?

Best regards,

Tatsuhiro Tsujikawa