Subject: Re: Followup on IETF72 discussion of FTP protocol extensions andupdates

From: Nicolas Williams <[email-masked]>
Date: Mon, 3 Nov 2008 01:36:00 -0600

On Fri, Oct 31, 2008 at 09:10:09PM -0500, Douglas J. Papenthien wrote:
> >>Personally, I'd rather see SFTP (or FTP over SSH) improved enough to
> >>make FTP go away.  But I don't think I mind this proposed WG.
> >
> >Yep.
> 
> I don't know that I particularly agree with this sentiment.  Much like the
> "transition" to IPv6, I think you're going to be hard pressed to have much
> luck stamping out FTP in favor of anything else regardless of the pros and
> cons.

SSHv2 and SFTP have pretty widespread deployment.  The situation is not
remotely comparable to IPv6.

Also, good old protocols never die.  TELNET is still around, and so is
rsh.  That does not justify continuing to standardize extensions to
them.

That said, there may well be good reasons for wanting to extend FTP in
spite of the existence of SSHv2/SFTP.  I suspect that some will argue
that SSHv2/SFTP have performance issues, and though I believe those are
the result of implementation issues, not protocol design, perception
sometimes carries the day.  If there are enough people who want to do
this work, and the IESG does not object, then a WG may well get
chartered to do the work.

I take no stand yet on the issue of whether a WG should be chartered to
work on FTP, but I do wish those who want to work on it worked on SFTP
instead.
_______________________________________________
Apps-Discuss mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: John C Klensin <[email-masked]>
Date: Mon, 03 Nov 2008 08:31:59 -0500

--On Monday, 03 November, 2008 01:36 -0600 Nicolas Williams
<[email-masked]> wrote:
 
>... 
> That said, there may well be good reasons for wanting to
> extend FTP in spite of the existence of SSHv2/SFTP.  I suspect
> that some will argue that SSHv2/SFTP have performance issues,
> and though I believe those are the result of implementation
> issues, not protocol design, perception sometimes carries the
> day.  If there are enough people who want to do this work, and
> the IESG does not object, then a WG may well get chartered to
> do the work.
> 
> I take no stand yet on the issue of whether a WG should be
> chartered to work on FTP, but I do wish those who want to work
> on it worked on SFTP instead.

Let me suggest a different approach to thinking about this.
Extensions happen because they serve a perceived need and people
care enough about them to create multiple implementations,
standardized or not.  Conversely, if the IETF standardizes
something that no one cares about, we won't see more
implementations than if we had not touched it.  The IETF can
contribute only three things:
  (i) community review, which sometimes improves the technical
quality of a specification and sometimes filters out bad ideas, 
  (ii) a clear and standardized specification that facilitates
having implementations of the extensions be interoperable,
  (iii) standardization of verb and parameter strings to
increase the odds that they are used the same way in different
implementations.

My personal criteria for a WG follow closely on that observation
about implementations and those three possible contributions.
There need to be people, groups, or organizations who are
interested in implementing any given extension, so it is not
just a single-company/ single-product idea.  There need to be
people who are willing to do serious and significant work on the
specification in the IETF, in review, discussion, and
document-editing.  And, even if we decide to not standardize
individual extensions, it seems to me that registries of command
forms and their variations worthwhile (but registry creation
alone may not justify a WG).

With regard to comparison with other protocols, FTP has two key,
closely-related, characteristics that differentiates it from
SFTP, TFTP, file transfer over HTTP, and so on.   Those
characteristics are the use of separate command and data
streams/connections and the asynchronous command design that
facilitates.  That arrangement has distinct advantages in some
situations, such as when files are large or transfers slow.  It
is pathological for deep-NAT environments, but that is at least
one of the reasons we have other protocols for file transfer.
But, e.g., SFTP seems to be to be a possible good argument
against an extension that makes FTP single-stream, but not
against other extensions to FTP.

    john

---

From: Barry Leiba <[email-masked]>
Date: Mon, 03 Nov 2008 09:03:27 -0400

> document-editing.

As co-instigator here, let me note that I absolutely agree with John on this.  If 
there's a reasonable group that thinks these (and/or other) FTP extensions should 
be standardized -- even if there are also some who don't -- I'd like to go ahead 
with this.  And/but I need to see that there is such consensus on proceeding.

> situations, such as when files are large or transfers slow.

I've never understood this argument, really.
Why is it that much more useful to have this model:

  Channel 1: login, get F1, get F2, get F3
  Channel 2: transfer F1
  Channel 3: transfer F2
  Channel 4: transfer F3

...than this model (à la HTTP):

  Channel 1: login, get F1, transfer F1
  Channel 2: login, get F2, transfer F2
  Channel 3: login, get F3, transfer F3

?  The former avoids authenticating separately in each parallel channel, but 
that's really a minor point (and there are ways to make it lighter-weight). 
Otherwise, I don't see why it makes any real difference.

Barry

---

From: Tony Finch <[email-masked]>
Date: Mon, 3 Nov 2008 14:12:47 +0000

On Mon, 3 Nov 2008, Barry Leiba wrote:

> > closely-related, characteristics that differentiates it from SFTP,
> > TFTP, file transfer over HTTP, and so on.  Those characteristics are
> > the use of separate command and data streams/connections and the
> > asynchronous command design that facilitates.  That arrangement has
> > distinct advantages in some situations, such as when files are large
> > or transfers slow.
>

HTTP also has the advantage for small files that you can pipeline multiple
requests down the same connection so you only need to crank open the TCP
window once.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
VIKING NORTH UTSIRE: SOUTHERLY OR SOUTHWESTERLY 4 OR 5, BUT 6 OR 7 IN NORTH AT
FIRST, VEERING NORTHERLY 4 LATER. MODERATE OR ROUGH. OCCASIONAL RAIN. MODERATE
OR GOOD.

---

From: John C Klensin <[email-masked]>
Date: Mon, 03 Nov 2008 09:59:37 -0500

--On Monday, 03 November, 2008 09:03 -0400 Barry Leiba
<[email-masked]> wrote:

>> With regard to comparison with other protocols, FTP has two
>> key, closely-related, characteristics that differentiates it
>> from SFTP, TFTP, file transfer over HTTP, and so on.   Those
> ?  The former avoids authenticating separately in each
> parallel channel, but that's really a minor point (and there
> are ways to make it lighter-weight). Otherwise, I don't see
> why it makes any real difference.

For the example you have given, consider the variation:

  Channel 1: login, get F1, get F2, get F3, go-to-lunch
  Channel 2: transfer F1
  Channel 3: transfer F2
  Channel 4: transfer F3

(a variation that the second option does not permit absent
scripting or an asynchronous HTTP client).  But, more important,
in a command-asynchronous environment and a good implementation,
the STAT command works to give actual, current, status of a
transfer connection (much more useful in my experience than the
"hash" kludge), ABOR (which has never been a synonym for either
REIN or QUIT) can be used to stop a file transfer that was
mistakenly started without tearing everything down, and so on.

While rare in practice, 
    Channel 1: get F1, put F2
    Channel 2: transfer (F1<-, F2-> )
is also clearly permitted by the protocol and is efficient for
some types of channels.

Also, if block transfer (MODE B, or the largely-obsolete
COMPRESSED) modes are used, the response to 

  Channel 1: login, get F1, get F2, get F3, go-to-lunch
may reasonably be
  Channel 2: transfer F1, transfer F2, transfer F3

The fact that there has never been a requirement in FTP to issue
a  command and await a reply also gave it, from the beginning,
what we got into SMTP with pipelining.  That, in turn, has
facilitated implementation of some common FTP "commands",
notably MGET and MPUT, as user interface macros.  While there
have been some truly stinky implementations of those commands,
the protocol design should permit the user to issue the command
macro and then put the FTP session into a background or
ancillary process, let the client and server send multiple files
over as many connections as appropriate, and periodically come
back to the process and issue a STAT to determine what is going
on.

One of the reasons I'd sort of like to see a WG is that I
believe that some of the FTP extensions that have been proposed
over the years are attempts to make poor and incomplete
implementations of the protocol work a little better.   I'd
rather see the need for that class of extension eliminated by
more widespread and better implementations and hope that a WG
can push in that direction, but maybe that is too naive and
utopian.

    john

---

From: Barry Leiba <[email-masked]>
Date: Mon, 03 Nov 2008 10:30:00 -0400

> scripting or an asynchronous HTTP client).

Well, but it *is* all the client, not the model.  The second protocol model 
certainly *does* permit this scenario, if the client supports it.  And the first 
model doesn't permit it either if the client is synchronous.

It gets down to what you say later, which I'll characterize as a need for 
implementors to "think outside the protocol"... that is, not to tie the client 
behaviour so closely to the protocol that it restricts what the user can do.

In this case, if the client allows the user to type "get F1", then "get F2", then 
"get F3" -- or to do the same with drag-and-drop, or whatever -- and then starts 
up the transfers while the user has dim sum, either protocol model works for that.

I'm not saying that FTP doesn't have things going for it -- only that I don't 
think it has a monopoly on asynchonicity because of its separation of control and 
data channels.

> mistakenly started without tearing everything down, and so on.

Agreed; those are clear advantages of the first model.

> can push in that direction, but maybe that is too naive and

If a WG can drive better FTP implementations, that'd be Very Cool.  I'm 
skeptical, but I think it's a nice goal.

Barry

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 3 Nov 2008 10:27:55 -0600

On Mon, Nov 03, 2008 at 08:31:59AM -0500, John C Klensin wrote:
> [...]

So far so good.

> against other extensions to FTP.

SFTP is most certainly asynchronous.  You may want to revise your
position.  (Commands and data may be interleaved, and multiple transfers
may proceed concurrently.)

Nico

---

From: John C Klensin <[email-masked]>
Date: Mon, 03 Nov 2008 12:16:46 -0500

--On Monday, 03 November, 2008 10:27 -0600 Nicolas Williams
<[email-masked]> wrote:

> SFTP is most certainly asynchronous.  You may want to revise
> your position.  (Commands and data may be interleaved, and
> multiple transfers may proceed concurrently.)

Nico, 

I have probably just been using weak implementations of SFTP,
but have yet to see one that permits me to get the status or, or
cancel, a transfer that is in progress without killing off the
connection, that are able to initiate multiple transfers
concurrently, etc.  I confess not having studied the protocol
itself but, if SFTP has those facilities, we may be dealing with
insufficient implementations of both SFTP and FTP rather than a
need for at least some extensions to either (other extensions,
and the probable need for extension registries, may be another
matter).

    john

p.s. if SFTP is also in need of an extension registry, I'd be
happy to sign you up as co-author, send you the XML for
modifications, and try to push both extension registries through
at the same time and with the same document.   No need to
duplicate _that_ particular part of that work.

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 3 Nov 2008 11:32:10 -0600

> and the probable need for extension registries, may be another

The SFTP protocol is more like a remote filesystem protocol than a file
transfer protocol.  This is where SFTP used to fall flat, incidentally,
since remote filesystem protocols typically don't include support for
end-of-line conversion, for example.  Newer versions of SFTP have added
support for such conversions, but most implementations don't have it.

You open a file, read from/write to it, close it.  If you wish to
interrupt a file transfer then you just stop initiating I/O to that
file.  Multiple outstanding READs and WRITEs are allowed concurrently,
over a single channel, to the same file and to many files.

Most SFTP clients are rather simplistic however, and simply transfer a
single file at a time, and keep a small number of concurrent I/Os in
flight (older clients kept a single I/O in flight).

I'd rather see implementors improve SFTP performance.  (And SSHv2
performance, by making SSHv2 channel window sizes adjust better.)

> duplicate _that_ particular part of that work.

SFTP is not yet published as an RFC...

And no, there's no chance that I'll work on any SFTP-related drafts.

I understand perfectly well that if no one stands up to offer to do that
work and plenty of others are interested in the proposed FTP work, then
we may well end up with a WG chartered to work on FTP.  I do believe
there are people who are interested in working on SFTP, and SFTP does
need a home (SECSH WG closed a while back), so I suppose we could even
see a WG chartered to work on both, SFTP and FTP.

Nico

---

From: John C Klensin <[email-masked]>
Date: Mon, 03 Nov 2008 13:39:19 -0500

--On Monday, 03 November, 2008 11:32 -0600 Nicolas Williams
<[email-masked]> wrote:

> I do believe
> there are people who are interested in working on SFTP, and
> SFTP does need a home (SECSH WG closed a while back), so I
> suppose we could even see a WG chartered to work on both, SFTP
> and FTP.

It is up to the ADs, but my personal belief is that chartering a
WG to work on two somewhat-similar protocols in parallel,
especially when one is at a very different level of maturity
than the other (the base FTP spec has been unchanged since the
1970s; there is no published standards-track protocol for SFTP)
would be a mistake.  Either two groups proceeding in parallel or
one that does FTP and then moves on to SFTP would be fine with
me.

Of course, if an FTP WG were chartered, asking it to look at an
individual submission for a standards-track SFTP spec would also
be an option available to the ADs.

But, as you suggest, if there aren't people who are willing to
stand up and volunteer to do serious work on SFTP --presumably
starting with getting an initial standards-track specification
together-- then, by default, there isn't going to be a WG for it.

   john