Subject: [ftpext] draft-yevstifeyev-ftp-uri-scheme-02 posted

From: Mykyta Yevstifeyev <[email-masked]>
Date: Sun, 19 Jun 2011 08:05:30 +0300

Hello,

After working on the document a bit, I've submitted a new version of 
draft-yevstifeyev-ftp-uri-scheme-02:

http://tools.ietf.org/html/draft-yevstifeyev-ftp-uri-scheme-02

The differences from -01 are at 
http://tools.ietf.org/rfcdiff?url2=draft-yevstifeyev-ftp-uri-scheme-02.txt

I personally believe the document is almost ready, yet, I'd like to seek 
more community comments (if any) on it.

Thanks,
Mykyta Yevstifeyev

---

From: John C Klensin <[email-masked]>
Date: Sun, 19 Jun 2011 10:38:34 -0400

--On Sunday, June 19, 2011 08:05 +0300 Mykyta Yevstifeyev
<[email-masked]> wrote:

> Hello,
> 
> After working on the document a bit, I've submitted a new
> version of draft-yevstifeyev-ftp-uri-scheme-02:
> 
> I personally believe the document is almost ready, yet, I'd
> like to seek more community comments (if any) on it.

First of all, I appreciate the many improvements in this
document.  Most of my difficulties with the earlier version
remain and, as a result, I don't think we are at "almost ready".
For example:

(1) Regardless of what was done in RFC 1738, if something is
going to be called an "FTP" URI, it should reflect the FTP
protocol is a serious and comprehensive way.   Failure to do so
is, IMO, a "known technical omission" and, using logic you have
used in other contexts, would require deprecating the existing
definition and moving it to Historic if it were in a stand-alone
document.   Perhaps, if you want to go down this path, you
should think about a new "ftpanon" URI that would incorporate
these capabilities (and maybe one or two more, see below), drop
<user-pass> from the syntax entirely (always send "anonymous"
and then respond to any password prompt based on what it says),
and so on.   If you did that, it would make sense to have the
document also update 1738 to deprecate its definition of an FTP
URI (or to have a discussion with the IESG about whether the
"obsoleted by" status of 1738 already accomplished that), which
I assume is part of your objective.

(2) In a world in which a lot of servers (maybe a majority) run
Linux or flavors of Unix and many client machines run something
else and get confused when trying to interpret files with
LF-only line terminations as text), I still believe that support
for TYPE is critical, even in a the stripped-down "ftpanon" URI
concept but certainly with a full FTP URI.

(3) To the extent to which the FTPEXT2 WG is actually doing
anything, a full FTP URI needs to consider the changes that are
being made there, or at least to have a mechanism for being
extended to recognize additional commands and keywords.   Again,
reducing your expectations to what I describe above as an
"ftpanon" URI would eliminate at least most of that problem.  If
you want a general FTP URI, I think that WG should carefully
review the spec once it has drained its queue of
charter-specified pending work.

(4) If your goal is not really to define a URI but to describe
the way in which the "ftp" URI is being used today, that would
be ok as an Informational document.  But, to the extent to which
what the description did or did not include constituted a known
technical omission or defect, that document would be
inappropriate for standards track.

(5) It in just a nit compared to the above, but I can find
absolutely nothing in this specification that updates RFC 959 in
any way.

All just my opinion, of course.

    john

---

From: Mykyta Yevstifeyev <[email-masked]>
Date: Mon, 20 Jun 2011 12:10:58 +0300

John,

> I assume is part of your objective.
I really consider a number if technical omissions in the document.  
However, spontaneous deprecation of the 'ftp' scheme which is currently 
used and introducing the new 'ftp' URI scheme which no one has heard 
about isn't a good option (see Section 2.1 of RFC 4395).  My proposal 
for experimentation is probably a solution.  Let's proceed with this 
document first, and then undertake an effort to produce a completely new 
specification, first experimental, and then if success, deprecate this 
scheme entirely and move that experimental to Standard Track (see 
http://www.ietf.org/mail-archive/web/ftpext/current/msg00302.html).

I really don't think 'ftpanon' scheme may be useful.  It is 
inappropriate to create several schemes for one protocol (unless it is a 
secured variant, which is the "current practice").  Having the 
"userinfo" in the URI scheme for FTP is better that deprecating it since 
might decrease the possibilities (and, thus, usefulness) of such scheme.
> concept but certainly with a full FTP URI.
Leaving the type-code part will probably create a precedent - why do we 
provide the possibility to denote use of the only one option in the 
URI.  If type-code is retained, we should provide the extension 
mechanism and make type-code a part of it.  But see below.
> charter-specified pending work.
Extension mechanism is really necessary.  Yet, if we try to describe the 
scheme as it is currently used there is no place for it here.  Of 
course, if we agree to undertake the effort to produce a completely new 
scheme specification, extension mechanism will be a must there.  But not 
now.
> inappropriate for standards track.
I agree with Martin here.  It's better to have what is currently used on 
Standards Track rather something invented for the first time.  RFC 2026, 
Section 4.1.1:

>     A Proposed Standard specification is generally stable, has resolved
>     known design choices, is believed to be well-understood,
>
>     [ . . . ]
>
>
>     Usually, neither implementation nor operational experience is
>     required for the designation of a specification as a Proposed
>     Standard.  However, such experience is highly desirable, and will
>     usually represent a strong argument in favor of a Proposed Standard
>     designation.

> absolutely nothing in this specification that updates RFC 959 in
I think updating RFC 959 is to reflect that we introduce (in fact, 
document) one of the protocol elements for FTP.

Mykyta

---

From: Mykyta Yevstifeyev <[email-masked]>
Date: Mon, 20 Jun 2011 12:56:27 +0300

20.06.2011 11:39, "Martin J. Dürst" wrote:
> Hello John,
>
> [ . . . ]
>
> With that, I don't want to necessarily say that I would disagree with 
> what you wrote e.g. specifically about the TYPE parameter. 
I may argue whether type-code part is really useful in the scheme.  I've 
checked the behavior of the browsers with regard to this part.  The results:

    * Firefox 4.0.1 (with no FTP-related addons): seems to ignore. 
      ftp://ftp.rfc-editor.org/in-notes/rfc-ref.txt;type=aft has to
      return 500 (checked; see below), but it doesn't.  The same is with
      ftp://ftp.rfc-editor.org/in-notes/rfc-ref.txt;type=f (F type-code
      isn't specified yet).
    * Google Chrome 12.0: the same.
    * Safari 3.2.1: the same.
    * IE 8: handles ";type=a" or ";type=i"; URIs with other one-letter
      type-codes are not resolved at all; if type-code has more than one
      letter - ignores.

I've also checked this with 2 FTP clients.

    *   FireZilla 3.3.2.  Type-code part is simply handled as a part of
      a path resulting in URI ftp://ftp.rfc-editor.org/in-notes/;type=a
      being interpreted as:

> Status:    Resolving address of ftp.rfc-editor.org
> Status:    Connecting to 64.170.98.47:21...
> Status:    Connection established, waiting for welcome message...
> Response:   220 "FTP Server Ready"
> Command:    USER anonymous
> Response:   331 Please specify the password.
> Command:    PASS **************
> Response:   230 Login successful.
> Command:    OPTS UTF8 ON
> Response:   200 Always in UTF8 mode.
> Status:    Connected
> Status:    Retrieving directory listing...
> Command:    CWD /in-notes/;type=a
> Response:   550 Failed to change directory.
> Command:    PWD
> Response:   257 "/"
> Status:    Directory listing successful

    * SmartFTP 4.0: the same.

FYI: bare Telnet connection to the server for checking TYPE command: 
http://i51.tinypic.com/21b10jt.jpg

So defining the scheme as currently used requires the deprecating the 
type-code part, I'm convinced.  It is mostly useless.

> But I very strongly think that this and other details have to be 
> argued on their merits, rather than on a on the base of a non-existing 
> principle of "if it's in the protocol, it has to be in the URI scheme".
Agreed here.

Mykyta
>
> Regards,   Martin.
>
>
>
> On 2011/06/19 23:38, John C Klensin wrote:
>>
>>
>> _______________________________________________
>> Uri-review mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/uri-review
>>
>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 20 Jun 2011 17:39:53 +0900

Hello John,

I'm not at all an expert in FTP, nor in the details of the current ftp: 
URI implementations. But the overall direction of your mail seems to be
"if it's in the FTP protocol, then it has to be in the ftp URI scheme".

In my eyes, that's just a non sequitur. If you look at HTTP, there are a 
lot of bits and pieces that are in the protocol but are not in the URI 
scheme, for good reasons. If you look at the mailto: URI scheme, then 
there are also a lot of features in SMTP and in the message format that 
are not available in the scheme. In fact for the mailto: scheme, in my 
understanding, the header part of the message format is now covered, but 
it was the spec that caught up with implementations, not the other way 
round.

Also, to claim that a spec that describes what's currently implemented 
(let's assume it does) can only go to informational, and that for 
standards track, it's necessary to cover all the features in the base 
protocol is totally new to me. As far as I understand it, the IETF is 
first and foremost about "rough consensus and running code" and only 
later about "known technical omissions".

With that, I don't want to necessarily say that I would disagree with 
what you wrote e.g. specifically about the TYPE parameter. But I very 
strongly think that this and other details have to be argued on their 
merits, rather than on a on the base of a non-existing principle of "if 
it's in the protocol, it has to be in the URI scheme".

Regards,   Martin.

---

From: John C Klensin <[email-masked]>
Date: Mon, 20 Jun 2011 08:13:41 -0400

--On Monday, June 20, 2011 17:39 +0900 "\"Martin J. Dürst\""
<[email-masked]> wrote:

> Hello John,
> 
> I'm not at all an expert in FTP, nor in the details of the
> current ftp: URI implementations. But the overall direction of
> your mail seems to be

Not really what I'm saying.  I just think that an FTP URI has to
be sensitive to FTP usage and the way the protocol actually
works.  At least IMO, a poor job was done of that in 1738; if
one proposes a standalone, standards-track FTP URI today, I'm
going to do parts of that analysis and then keep repeating
"known technical omission".  As far as the extensions are
concerned, I imagine that some of them are important and some
are not.  Since both the URI review and FTPEXT2 mailing lists
are being copied on this, I suggest that the latter ought to be
considering whether, if a given extension is not important
enough to be included in the URI, it is really worth the trouble
as an extension.  I can well imagine the answer for some
extensions being "not important enough for the URI but still
worth doing", but I think that is a discussion the WG needs to
have".  

Three examples of the "known technical omission" problem: 

(i) At least historically, the most-used command in the FTP
protocol, other than the basic authentication ones (USER, PASS),
CWD and PWD and the basic ones actually involved in transferring
files (RETR, STOR, and probably PASV) has been TYPE.  It is not
like, e.g., STRU or the file structuring commands, which many
people and servers can go for many years without ever seeing.
TYPE was really important in the early days of the net when we
not only had EBCDIC floating around, but had at least three
different ways to encode ASCII.  A decade and a half ago, only
the CRLF issue remained and, if examined from a narrow web
perspective (see below), it doesn't make much difference (the
number of problems it causes every year, with lusers not
understanding the symptoms and a distinct shortage of standard,
user-accessible, conversion utilities on most systems, remains
non-trivial).   It gets more important again as we see more and
more files in non-ASCII character sets, both Unicode and
non-Unicode.  Even with the former alone, an IMAGE transfer may
yield any of more than a half-dozen forms (see
draft-ietf-appsawg-3536bis for a list that still does not
include the obscure ones).  So, to me, saying "TYPE" is not
needed or should be treated like any other extension indicates a
lack of understanding of the underlying protocol.

(2) There is currently a proposal in the FTPEXT2 WG to add a
HOST command to FTP (draft-ietf-ftpext2-hosts) for exactly the
same reason we needed to add one to HTTP.  I'm personally not a
fan of that particular way of doing the job, but one of the
strong arguments for it is that it could potentially be
automated in a shim or API that connects an FTP URI processor to
the FTP protocol as "always send HOST, if it is rejected as not
recognized, ignore and pretend it wasn't sent".  But, if we
think HOST is important, that needs to be considered and
explicitly discussed in any FTP URI document.  If it isn't
important enough to be considered and discussed, then the
extension probably isn't important enough to adopt and use.
Since the WG hasn't acted on the proposal, the issue is somewhat
forward-looking, but that puts the current FTP URI proposal in
the same boat as MAILTO -- it is hard to define a URI for a
protocol to which it is not integral and has to be retrofitted
when the protocol itself is being changed/extended in ways that
should rationally affect the URI.

(3) For obvious security reasons, username-password pairs ceased
being popular around the IETF many years ago.  Username-password
pairs embedded in URIs in clear text are much worse.  Some of
the extensions you (and Mykyta) dismiss provide ways around that
problem.  In addition, for the very popular 'anonymous' use of
FTP, we know what the password patterns are: a reserved/known
keyword, such as "guest"; an email address; or "server simply
doesn't care what is sent".   That could be built into the
URI-interpreting protocol interface mechanism as well, but the
current document circles around it with a lot of handwaving.

Note that neither (2) nor (3) would require any changes to the
URI or its syntax, only a specification that understood the FTP
protocol and its use much better.

> In my eyes, that's just a non sequitur. If you look at HTTP,
> there are a lot of bits and pieces that are in the protocol
> but are not in the URI scheme, for good reasons. 

Sure.  I trust I don't need to point out to you either that URLs
and HTTP were designed together for use with each other and that
most of the decisions as to what was left out were made after
careful consideration, not by accident and in haste (and, for
the record, I'm criticizing 1739, not Mykyta's proposal).

> If you look
> at the mailto: URI scheme, then there are also a lot of
> features in SMTP and in the message format that are not
> available in the scheme. In fact for the mailto: scheme, in my
> understanding, the header part of the message format is now
> covered, but it was the spec that caught up with
> implementations, not the other way round.

And the mailto: spec is also an example of the difficulties one
can get into when one naively retrofits a URI on top of an
established and widely-deployed protocol.

> Also, to claim that a spec that describes what's currently
> implemented (let's assume it does) can only go to
> informational, and that for standards track, it's necessary to
> cover all the features in the base protocol is totally new to
> me. As far as I understand it, the IETF is first and foremost
> about "rough consensus and running code" and only later about
> "known technical omissions".

Informational documents that describe something as already
deployed and unlikely to change are actually common and are
written with the understanding that the IETF has little to offer
in terms of adding value to the protocol other than to document
it.  For standards track, I invite you to read 2026, where
"known technical omissions" is one of the few clear bars to
adopting a Proposed Standard.

> With that, I don't want to necessarily say that I would
> disagree with what you wrote e.g. specifically about the TYPE
> parameter. But I very strongly think that this and other
> details have to be argued on their merits, rather than on a on
> the base of a non-existing principle of "if it's in the
> protocol, it has to be in the URI scheme".

See above.  I'm not arguing  "if it's in the protocol, it has to
be in the URI scheme".  I'm arguing precisely for some careful
consideration on a case-by-case basis and against a different
non-existent principle of "if it was omitted from RFF 1738, we
don't need it".

regards,
    john

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 20 Jun 2011 14:30:28 +0200 (CEST)

> I've also checked this with 2 FTP clients.

As a counter-example: curl has supported type= in FTP URLs since many years. 
Even sent over HTTP proxies when a FTP URL using type= was used and a HTTP 
proxy is specified, and I know there are HTTP proxies that deal with it 
properly.

-- 

  / daniel.haxx.se

---

From: Mykyta Yevstifeyev <[email-masked]>
Date: Wed, 22 Jun 2011 07:10:55 +0300

Hello all,

> worth doing", but I think that is a discussion the WG needs to
I agree here, so let's have a discussion.
> lack of understanding of the underlying protocol.
Probably.  However, if we agree on leaving this part, we will need to 
have a discussion on the handling of this part in the URI, since RFC 
1738 isn't clear with this regard.
> should rationally affect the URI.
I've carefully read the FTPEXT2's draft on the proposed HOST command.  
HOST is surely a good proposal to use at all and I currently think 
including it in the URI specification for FTP is probably a good idea.  
So, with this respect I can agree with you and, therefore, agree to 
include this proposal in the 'ftp' URI specification.
> current document circles around it with a lot of handwaving.
In the current draft we have the following situation: if there are no 
user credentials in the URI at all, anonymous FTP is used; however, if 
there are a user name and password (or just a user name), they are first 
used.  IMO, denoting user credentials in the URI isn't bad enough to 
abandon such capability.  Those people who care about security much will 
be free not to include the user information in the URIs.  Yet, if, for 
instance, some person wants to share their FTP account with the other 
one, including the user-info in the URI is the best option among all 
possibilities to do this.  So I don't think removing the user-info part 
from the URI and making use of anonymous FTP mandatory is a good idea.  
I'd better care about denoting the account information in the URI, which 
is one of the "technical omissions" too.
> the record, I'm criticizing 1739
Maybe you meant 1738 here.
> , not Mykyta's proposal).
>
>> If you look
This is mostly a question to discuss.  My personal opinion is a bit 
double.  From the one side, Standards Track document should describe 
what is widely-accepted.  From the other side, it must not have 
different technical issues which aren't clear enough.  I think something 
like this is appropriate for our case: taking the existing URI 
specification and appending any necessary features to it.
> non-existent principle of "if it was omitted from RFF 1738, we

All the best,
Mykyta Yevstifeyev

---

From: Mykyta Yevstifeyev <[email-masked]>
Date: Thu, 23 Jun 2011 07:16:36 +0300

Hello,

I've made several changes to the draft-yevstifeyev-ftp-uri-scheme based 
on comments from John and I'd like to request community feedback on 
them.  Please find my changes descriptions in-line.
Currently the syntax for <ftp-path> is:

>      ftp-path       = cwd-part [ "/" name ] [ typecode-part ]
>      cwd-part       = *( "/" cwd )
>      cwd            = segment-nsc
>      name           = segment-nsc
>      segment-nsc    = *pchar-nsc
>      pchar-nsc      = unreserved / pct-encoded / sub-delims-nsc / ":"
>                     / "@"
>      sub-delims-nsc = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" /
>                     / "," / "="
>                     ; RFC 3986 <sub-delims> excluding semicolon (";")
>                     ; character (ASCII [ASCII, RFC0020] character 0x3B)
>      typecode-part  = ";type=" typecode
>      typecode       = "a" / "e" / "i" / "u" / typecode-ext
>      typecode-ext   = ALPHA
The appropriate step was added in the algorithm in Section 2.2.3:

>    (2)  if the <typecode-part> is present, perform the TYPE command with
>         the <typecode> as an argument;
>
>      Note:  There are currently four options of the <typecode> part.
>      They include "a", "e", "i", which are specified in RFC 959
>      [RFC0959] and stand for ASCII, EBCDIC text and binary transmission,
>      respectively, and "u", which is specified in RFC mmmm [I-D ietf-
>      ftpext2-typeu] and stands for Unicode data.  Additional data types
>      may be accommodated in the URI via the <typecode-ext> production.
This incorporates the ongoing work of FTPEXT2 WG regarding U data type 
for Unicode data and provides a possibility to add new data types in the 
URI.
> including it in the URI specification for FTP is probably a good 
> idea.  So, with this respect I can agree with you and, therefore, 
> agree to include this proposal in the 'ftp' URI specification.
Now 2 new paragraphs were added in Section 2.2.1:

>    Upon establishing a successful TCP connection, the client SHALL first
>    try to identify the host it is trying to access using the HOST
>    command [I-D ietf-ftpext2-hosts].  It is performed by sending this
>    command with the <host> part of the URI as an argument.
>
>    If either 500 or 502 reply is received in response (which identify
>    that the HOST command is unrecognized or unimplemented,
>    respectively), the client SHALL act as if a HOST command had not been
>    sent and continue processing the URI.  If either 501 or 504 reply is
>    received (which identify that the supplied hostname is syntactically
>    invalid or it is unavailable, respectively), the client's behavior
>    depends on how does the server react.  If, in accordance with Section
>    3.3 of RFC nnnn [I-D ietf-ftpext2-hosts], the server chooses to
>    terminate the connection, the client SHALL notify the user and take
>    no further actions.  Otherwise, if the server does not terminate the
>    connection, the client SHALL act as if a HOST command had not been
>    sent and continue processing the URI.
Best,
Mykyta Yevstifeyev