Subject: [ftpext] FWD: ftp/959 reboot

From: "William F. Maton" <[email-masked]>
Date: Thu, 5 Aug 2010 16:45:41 -0400 (EDT)

All,
 	I'm forwarding an email from Gregory Lundberg who is the author of 
a couple of expired drafts from the old WG and the former maintainer of 
wu-ftpd.  I have his permission to send the relevant extract of an offline 
conversation he and I have been having about FTP in general.  He might 
join us for further observations and discussions when he is able to, but 
in the meantime I can summarize responses to him offline.

Thanks,

wfms

---------- Forwarded message ----------
Date: Wed, 28 Jul 2010 08:44:36 -0500
Subject: Re: [wuftpd-dev] Sample help output from ftpd

My opinion on the restart of the FTPEXT WG is that it's time (well past it, 
actually) to do a reboot on the FTP specification.  Anyone today who attempts 
to determine what the protocol *is* has to look at far to many RFCs, some 
offering contradictory specifications.  This is unacceptable for one of the 
core protocols of the Internet.

Over the years, some have said the FTP is a dying protocol.  Mainly this comes 
from the HTTP camp.  The problem is very few people step back and look at 
first-principles.  Another problem is what I call "nominal realism", which is a 
bit of a misuse of the term.  What I mean is they see the word "file" in "file 
transfer protocol" and presume the FTP is only meant to transfer files in the 
traditional meaning as a sequential dataset on computer mass storage.

To understand the FTP's place in the scheme of things we first need to ask 
ourselves: what are the various ways we can use IP, TPC and UPD for the 
transmission of large streams of information.  A very efficient scheme is to 
use a TCP channel for the control of data transmissions and separate channels 
for each transmisison.  This is the scheme implemented by the FTP. Another 
scheme is to use a single channel for both control and data as in the HTTP.  A 
little analysis shows that the scheme used by HTTP works best when a large 
number of relatively short transmissions are involved.  But when the size (or 
duration) of the transmission increases, the FTP quickly becomes more 
efficient.

A re-boot of the FTP specification should include some discussion of this. It 
servers both as a rationale as to why the protocol is still important, and as a 
guide for implementors.

Some years ago, the Apache Group was quite interested in taking a look at what 
it would take to merge wuFTPD into the Apache code base.  This turned out to 
not be possible because the Apache code base assumed a single channel for both 
control and data and could not be bent far enough to support the two-channel 
approach of the FTP.  This, however, was a software limitation. There is no 
particular reason a daemon could not be designed which supports both the FTP 
and the HTTP.  Care should be taken when specifying the operation of the FTP so 
as to not specify or encourage the use of a particular implementation method. 
If we look at the current FTP specification we see some discussion which does 
promote a particular design.

Another thing I'd suggest looking at is avoiding any semantic meaning for the 
data channel.  The term "file" is often used.  But the data channel is just a 
data channel.  Once established, the FTP really does not care what is sent, 
which direction it goes, or if data is travelling both wats at the same time. 
The job of the FTP is to reliably and securely establish the data channel 
between two resources; nothing more.  Recent additions to the protocol have 
provded addition information about the resources.  This recognized that there 
is more to a resource than simply it's name and establishing a reliable 
communication can depend upon that information.

One of the great features of the FTP is the actual *EASE* with which the 
protocol can be realized.  It is virtually impossible to have a meaningful HTTP 
session using Telnet.  While this is little used, part of the power of the FTP 
is it remains possible.

The great failing of the FTP also stems from its strength.  The use of a second 
connumication channel opens the door to abuse or misdirection of that channel. 
When I was last active I was in the process of designing a means where the 
controlling client and server could exchange information about the actual 
channel and ensure it was not intercepted or misdirected.  I imagine all work 
in that direction ceased.  If we re-visit the protocol, however, this should be 
one of the goals.

OK .. enough for now ..

ttyl
-- Greg

---

From: Daniel Stenberg <[email-masked]>
Date: Thu, 5 Aug 2010 23:19:21 +0200 (CEST)

On Thu, 5 Aug 2010, William F. Maton wrote (quoting Gregory Lundberg):

> Anyone today who attempts to determine what the protocol *is* has to look at 
> far to many RFCs, some offering contradictory specifications.  This is 
> unacceptable for one of the core protocols of the Internet.

I can only agree that the situation is muddy and far from good.

> A little analysis shows that the scheme used by HTTP works best when a large 
> number of relatively short transmissions are involved.  But when the size 
> (or duration) of the transmission increases, the FTP quickly becomes more 
> efficient.

> A re-boot of the FTP specification should include some discussion of this. 
> It servers both as a rationale as to why the protocol is still important, 
> and as a guide for implementors.

I read this opinion from time to time, and I'm very curious: what facts or 
what particular details of TCP are you backing up this statement on (realizing 
this statement was quoted from a secondary author)? Why would a single HTTP 
stream be any worse than a single FTP stream? Given a large enough file, the 
tiny tiny difference that the HTTP headers add vanish, and the FTP transfer 
typically suffers from MANY more initial RTT delays etc.

So yes, I would be very interested in such a rationale. In the past I've 
confronted others who have said similar things, but I've not yet heard anyone 
provide a believable explanation.

> Care should be taken when specifying the operation of the FTP so as to not 
> specify or encourage the use of a particular implementation method. If we 
> look at the current FTP specification we see some discussion which does 

Really? As a client implementor I've never really seen that. Perhaps I'm just 
too indoctrinated by it by now. Any particular wordings or phrases that stand 
out and explain what you mean?

> Another thing I'd suggest looking at is avoiding any semantic meaning for 
> the data channel.  The term "file" is often used.  But the data channel is 
> just a data channel.  Once established, the FTP really does not care what is 
> sent, which direction it goes, or if data is travelling both wats at the 
> same time. The job of the FTP is to reliably and securely establish the data 
> channel between two resources; nothing more.

I agree. I supposed we could look for inspirations for this in the httpbis 
work.

> protocol can be realized.  It is virtually impossible to have a meaningful 
> HTTP session using Telnet.  While this is little used, part of the power of 
> the FTP is it remains possible.

I find that statement rather funny and something I wouldn't agree with. A 
HTTP-minded person would say it exactly the same way but replace HTTP and FTP 
in that sentense. The fact that HTTP is a single connection still wins I'd 
say, and makes TELNET possible while telnetting a FTP transfer with telnet 
requires two telnet invokes, or with PORT something like nc or what not.

> The great failing of the FTP also stems from its strength.  The use of a 
> second connumication channel opens the door to abuse or misdirection of that 
> channel.

My experience with the data connection: it was possibly a good idea back in 
the early days. It is now a bad idea due to NATs, stateful firewalls, FTP over 
TLS and possibly other stuff.

* NATs close idling connections, and there's no way to really portably keep
   traffic going on the control channel while a really long and slow data
   transfer is in process so implementations have to deal with the control
   channel being killed by the NAT before the data is done.

* Firewalls need to understand FTP in order to support it fully without
   opening up ranges for it "just in case".

* When FTP traffic is sent encrypted, said firewalls don't see the FTP traffic
   and thus cannot act stateful so the data connection cannot get setup.

-- 

  / daniel.haxx.se

---

From: Anthony Bryan <[email-masked]>
Date: Mon, 9 Aug 2010 15:17:50 -0400

On Thu, Aug 5, 2010 at 5:19 PM, Daniel Stenberg <[email-masked]> wrote:
>> Anyone today who attempts to determine what the protocol *is* has to look
>> at far to many RFCs, some offering contradictory specifications.  This is

Daniel, any thoughts/insight into the RFC 1123 discussion, which
appears to update RFC 959?

> I agree. I supposed we could look for inspirations for this in the httpbis

with inspiration from httpbis, I've been working on a collection of
FTP RFCs: 959, 2389, 2428, 2577 (for now).

the idea would be to have an updated, coherent, harmonious document of
RFCs concerning FTP along
with implementation and real world info like httpbis.

http://www.metalinker.org/test/ftp/draft-bryan-ftpbis-00.txt
http://www.metalinker.org/test/ftp/draft-bryan-ftpbis-00.html

so far, besides collecting the RFCs and errata together, there are no
real changes to text. here's the changelog:

    Incorporate [RFC0959] Errata.
    Incorporate [RFC2389] FEAT at Section 4.1.4.
    Incorporate [RFC2428] "FTP Extensions for IPv6 and NATs" at Section 4.1.2.1.
    Incorporate [RFC2577] in Section 10.
    Include abstract of [RFC4217] in Section 10.2.
    BCP 14 wording.

TODO if there is interest:

    Determine if changes in RFC 1123 should be applied.
    Triage and include or reference other FTP RFCs - RFC 2228, 2640,
2773?, 3659, 4217.
    Update BNF to ABNF.
    Standardize command organization (subsections?) like EPRT/EPSV,
with separated ABNF?
    Move the whole sentence that is Section 8 into Section 3.2?

-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 9 Aug 2010 23:11:45 +0200 (CEST)

On Mon, 9 Aug 2010, Anthony Bryan wrote:

> Daniel, any thoughts/insight into the RFC 1123 discussion, which appears to 
> update RFC 959?

Yes, it seems to update or at least clarify a lot of RFC959 details but I must 
admit I've never really used 1123 much as a reference. I really can't tell at 
this point how valuable those sections are or aren't.

> FTP RFCs: 959, 2389, 2428, 2577 (for now).

I would suggest 3659 to be one of the main RFCs for any modern FTP 
implementation, thanks mostly to SIZE and MTDM in my own case.

> http://www.metalinker.org/test/ftp/draft-bryan-ftpbis-00.txt

Very cool.

I noticed how it mentions RFC3659 describes REST as an extension, but that's 
slightly inaccurate. RFC959 includes REST already, it was just clarified and 
more detailed in RFC3659.

-- 

  / daniel.haxx.se

---

From: "William F. Maton" <[email-masked]>
Date: Mon, 9 Aug 2010 20:03:00 -0400 (EDT)

On Mon, 9 Aug 2010, Daniel Stenberg wrote:

> Yes, it seems to update or at least clarify a lot of RFC959 details but I 
> must admit I've never really used 1123 much as a reference. I really can't 
> tell at this point how valuable those sections are or aren't.

I'm wondering how big an exercise it would be to determine the 
applicability of these types of RFCs relating to FTP?  But anyways...

> implementation, thanks mostly to SIZE and MTDM in my own case.

+1

>> http://www.metalinker.org/test/ftp/draft-bryan-ftpbis-00.txt

Yes well done!  If 1123 has been applied in practice, then perhaps 
obsolete that component of 1123 as well?

wfms

---

From: Anthony Bryan <[email-masked]>
Date: Mon, 9 Aug 2010 23:54:28 -0400

On Mon, Aug 9, 2010 at 5:11 PM, Daniel Stenberg <[email-masked]> wrote:
>> Daniel, any thoughts/insight into the RFC 1123 discussion, which appears
>> to update RFC 959?
>

it would be nice to know how authoritative they are. it sounds like at
least some people are not familiar w/ 1123. I only came across it
after digging deep into stuff.

now reading RFC 3659 more in depth, it at least treats it as more up
to date than 959 in Section 5:

   Note that there
   are some flaws in the description of the restart mechanism in STD 9,
   RFC 959 [3].  See section 4.1.3.4 of RFC 1123 [9] for the
   corrections.

> implementation, thanks mostly to SIZE and MTDM in my own case.

3659's absence is only due to my laziness. :) getting these old RFCs
into xml2rfc shape (even with rfc2xml) takes me quite a long time. the
sheer length of 3659 & the fact that no one might be interested kept
me from including it. it adds about 50 pages and is now included.

>> http://www.metalinker.org/test/ftp/draft-bryan-ftpbis-00.txt

thanks!

> more detailed in RFC3659.

thanks. I removed the original description from the 959 text so it is
not duplicated. if I kept working on this, I would introduce some
uniformity to the commands section.

---

From: Anthony Bryan <[email-masked]>
Date: Tue, 10 Aug 2010 00:03:00 -0400

On Mon, Aug 9, 2010 at 8:03 PM, William F. Maton <[email-masked]> wrote:
> I'm wondering how big an exercise it would be to determine the applicability
> of these types of RFCs relating to FTP?  But anyways...

which RFCs besides 1123?

1123 only has about 15 pages dealing w/ FTP, I hope there should be
enough of us here to determine what's still valid?

> Yes well done!  If 1123 has been applied in practice, then perhaps obsolete
> that component of 1123 as well?

thanks!

I haven't applied 1123 to what I'm working on yet.

do you mean "Obsoletes: 1123"? if so, I don't think that's possible.
"Updates:" might be more correct, as 1123 contains much more than
those few pages on FTP.

---

From: "William F. Maton" <[email-masked]>
Date: Tue, 10 Aug 2010 06:09:09 -0400 (EDT)

>> of these types of RFCs relating to FTP?  But anyways...

Speaking generally of course.

> "Updates:" might be more correct, as 1123 contains much more than

I'm grasping at appropriate IETF terminology, but yes, something along the 
line of either obsoletes just that part of RFC 1123, or updates 1123, 
whichever conveys the idea best.

wfms

---

From: "William F. Maton" <[email-masked]>
Date: Tue, 10 Aug 2010 13:11:59 -0400 (EDT)

> after digging deep into stuff.

So I've had another read of RFC 1123 4.1 coming from a different light. 
All it seems to do is clarify what implementors of RFC 959 should be 
supporting.  "RFC 959 was SO barebones as to leave interpretation wide 
open, so he's a clarifcation on what to do to make FTP servers and clients 
right so they work."

But I'm speculating because despite my efforts at looking for context 
(mailing lists, emails on the Internet), the RFC itself is all we have to 
go on.

If we look at 4.1.5 then we have an idea of what RFC 1123 is trying to 
say and perhaps compare that to the post Oct 1989 outcome:  Have 
implementations of the succeeding decade adopted any of that.

Didn't someone here once say that a working group on FTP would require a 
lot of work? :-)

>   RFC 959 [3].  See section 4.1.3.4 of RFC 1123 [9] for the

OK, so FTPbis 'updates' RFC 1123, not obsoletes it.

> me from including it. it adds about 50 pages and is now included.

I'll have another look.

wfms

---

From: Anthony Bryan <[email-masked]>
Date: Tue, 10 Aug 2010 21:31:43 -0400

On Tue, Aug 10, 2010 at 1:11 PM, William F. Maton <[email-masked]> wrote:
> So I've had another read of RFC 1123 4.1 coming from a different light. All
> it seems to do is clarify what implementors of RFC 959 should be supporting.
>  "RFC 959 was SO barebones as to leave interpretation wide open, so he's a
> clarifcation on what to do to make FTP servers and clients right so they
> work."
>
> (mailing lists, emails on the Internet), the RFC itself is all we have to go
> on.

I contacted the author, perhaps we'll hear from him.

reading RFC 3659 more, it refers to "STD 9, RFC 959 and STD 3, RFC
1123" as if they're to be taken together.

> If we look at 4.1.5 then we have an idea of what RFC 1123 is trying to say
> and perhaps compare that to the post Oct 1989 outcome:  Have implementations
> of the succeeding decade adopted any of that.

can some implementors chime in?

> Didn't someone here once say that a working group on FTP would require a lot
> of work? :-)

nahhh! :)

>> me from including it. it adds about 50 pages and is now included.

I merged in the changes from RFC 1123 now:

---

From: "William F. Maton" <[email-masked]>
Date: Tue, 10 Aug 2010 22:05:40 -0400 (EDT)

> 1123" as if they're to be taken together.

And then there's RFC 1579 which makes a brief mention of RFC 1123.  So I 
think I'm starting to gain a historical context.

> can some implementors chime in?

(switching hats)

I'm searching through the wu-ftpd mail archives (such as I have them) and 
see a few mentions about RFC 1123 and I see mentions of v 2.5.0 supporting 
that...I'm just wondering if we're not arguing the obvious in that RFC 
1123 found its way into implementations in one way or another 15-20 years 
ago and all this is a moot discussion now?

> http://www.metalinker.org/test/ftp/draft-bryan-ftpbis-00.html

Cool, thanks!

wfms

---

From: Anthony Bryan <[email-masked]>
Date: Sun, 15 Aug 2010 15:50:27 -0400

On Tue, Aug 10, 2010 at 10:05 PM, William F. Maton <[email-masked]> wrote:
>>> go
>>> say
> that...I'm just wondering if we're not arguing the obvious in that RFC 1123
> found its way into implementations in one way or another 15-20 years ago and
> all this is a moot discussion now?

Bob Braden, RFC 1123 author, says that 1123 indeed "Updates: 959" but
updates wasn't applied as religiously in 1988 as it is now.

I think this is worth correcting, at least in errata or something
similar (if not a new document like the one I've been working on) so
people can find the updates & know about the changes.

current implementations are probably mostly fine. the way things are,
someone working on a new FTP app wouldn't even know 1123 existed.

>> http://www.metalinker.org/test/ftp/draft-bryan-ftpbis-00.html

my pleasure!

---

From: Tim Kosse <[email-masked]>
Date: Sun, 15 Aug 2010 23:29:07 +0200

On 2010-08-11 03:31, Anthony Bryan wrote:
>> If we look at 4.1.5 then we have an idea of what RFC 1123 is trying
>> to say and perhaps compare that to the post Oct 1989 outcome:  Have

I have been aware of RFC 1123 but haven't taken it much into
consideration when implementing FileZilla (Server).

Most of the points from RFC 1123 either were irrelevant for the
implementation (e.g. 4.1.2.2, 4.1.3.4. ...) or were already a
well-established best-practice most if not all popular implementations
implicitly adhere to (e.g. 4.1.2.5, 4.1.2.6)

Even though rare, there are still clients out there that use commands
like XCWD (4.1.3.1) and TYPE L 8 (4.1.2.1). It appears that mostly
(only?) legacy mainframe systems are using these commands, I haven't
actually seen these commands being used by any modern client.

Personally I would gladly see some parts of FTP obsoleted with other
parts updated.

Without going into the details too much, I would like to obsolete the
following:
- All TYPEs other than I and A
- All MODES other than S (while the proposed MODE Z is a nice idea, I
don't think it is relevant anymore, internet bandwidth has increased
substantially since the first MODE Z draft was introduced)
- All STRUs other than F and the related SNMT command
- Telnet IP and Synch signals
- XCWD and related commands

On the updated part I would like to see
- Mandatory UTF-8 support
- Mandatory MLSD
- Mandatory TVFS would be nice

While it would definitely break compatibility with old implementations,
it would greatly simplify the protocol. I wonder if it is even possible
or desired to change FTP in such a drastic way or if that is a clear no-go.

Tim

---

From: Anthony Bryan <[email-masked]>
Date: Sun, 15 Aug 2010 18:55:12 -0400

> or desired to change FTP in such a drastic way or if that is a clear no-go.

thanks, Tim.
I think drastic changes to FTP are a no-go. drastic changes belong in
new protocols (SPDY etc).

what I am proposing is deprecating things that are no longer in use
and recommending best practices. the nearly 40 year history of FTP
makes obsoleting things that may not really be obsolete troublesome.
but I think we can do a lot to help possible new implementors.

for instance, regarding XCWD and related that you mention, we can keep
the original text from RFC 1123:

            All FTP implementations SHOULD recognize both forms of these
            commands, by simply equating them with extra entries in the
            command lookup table.

change the above to "FTP server" and add something like:
            All FTP client implementations SHOULD NOT send these
deprecated commands.

that way we preserve compatibility.

similarly, we may not be able to add new things that are mandatory
(MUST/REQUIRED). but we can certainly use SHOULD/RECOMMENDED for those
things that all modern implementations SHOULD support.
-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Paul Ford-Hutchinson <[email-masked]>
Date: Mon, 16 Aug 2010 10:48:16 +0100

> - XCWD and related commands

I really don't think this is a good idea.  I certainly use "TYPE E" in my 
FTP server.  IBM Mainframe and OS/400 systems use FTP extensively and need 
alot of these weird and wacky (from a WIN/*IX viewpoint) options.

It is vitally important to remember when tinkering with the protocol that:

A) a 'file' isn't necessarily a chunk of bits sitting sequentially on a 
disk somewhere
B) a 'directory' isn't necessarily a hierarchical tree of containers

For example, in the FTP server I wrote, CDUP makes no sense at all.

> or desired to change FTP in such a drastic way or if that is a clear 
no-go.

My 2c - big, big no-go.

Cheers,
Paul

---

From: Paul Ford-Hutchinson <[email-masked]>
Date: Mon, 16 Aug 2010 10:48:16 +0100

> someone working on a new FTP app wouldn't even know 1123 existed.

My experience of creating an FTP server from scratch in 1994 was that:

RFC959 was obviously useful
RFC1123 was just a list of things that I had to implement even if it made 
no sense in my environment (this is a _good_ thing from a standards and 
interoperability point of view)
There were some other RFCs that were also useful, I don't remember exactly 
how though.

To actually get it to work, I had to do an awful lot of dumping of tcp 
traffic between existing clients and servers because the spec (RFC959) is 
nowhere near defined enough to write an implementation.  (for example, the 
relative timing of the opening/closing of the data connection and the 
related FTP control connection commands is critical but undocumented, 
which is why I put them into RFC4217.)

I guess that the point I'm trying to make is:
- to write an implementation, you need to do more than just read the 
obvious RFCs, whilst we can and should help people to find the required 
information, in order for them to get to end of job coding such a beast 
will require either a huge documentation effort on our behalf, or a bit of 
gumption on theirs.

Paul

---

From: "William F. Maton" <[email-masked]>
Date: Fri, 20 Aug 2010 18:53:26 -0400 (EDT)

+1

wfms

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sat, 21 Aug 2010 11:00:40 +0200

> with implementation and real world info like httpbis.

I don't think simple reorganization and clarification will do it for FTP. There is a ton of stuff in the current specs that is no longer used at all (all the fancy file/record organization stuff), stuff that is technically legal but doesn't make sense (PASV+PORT), stuff that is technically legal but isn't used in practice and using it in practice would cause big problems (PASV with a different address in the 227 than the control channel address, EPSV with an argument), stuff that should work but has issues in practice (EPSV), stuff that should have been retired a decade ago but is still in wide use (PASV+PORT) and so on.

I don't think revisiting the FTP spec without tightening the spec and deprecating some currently legal options makes much sense. Yes, it's annoying to have to read so many documents to create an FTP server or client, but the real problem isn't the number of documents, but that what's in those documents and what FTP servers and clients do in practice is so different from what's in those documents. In that sense an effort like RFC 1771 -> RFC 4271 where the focus was to document what's really out there seems to make sense.

---

From: Anthony Bryan <[email-masked]>
Date: Sat, 21 Aug 2010 17:44:38 -0400

> I don't think revisiting the FTP spec without tightening the spec and deprecating some currently legal options makes much sense. Yes, it's annoying to have to read so many documents to create an FT…

yep that's what I'm proposing, but there doesn't seem to be enough
interested people.

I posted the httpbis charter here earlier, but I think it's worth reading
http://datatracker.ietf.org/wg/httpbis/charter/