Subject: Ftp-WG: FEAT UTF8 does NOT mean \r and \n ok in pathnames?

From: Pat LaVarre <[email-masked]>
Date: Wed, 16 Jan 2002 19:33:53 -0500

> > > P.S. As yet, munnari.oz.au:21
> > > replies 500 to FEAT.
...
> > [email-masked] 1/16/02 11:39am
> > Yes, it does.
> > Try port 9000.
...
> aka 128.250.22.2:9000

Good fun, thank you.

Is munnari.oz.au:9000 intended as a reference
implementation of an FtpServer?  i.e. made available
to be complimented by imitation?

I'm pleased to see this FtpServer includes the UTF8
option among the lines of its 211 reply to FEAT ...
but I don't yet understand how it complies with
rfc2640 [International Ftp]: "On receipt of a pathname
containing a <CR><NUL> sequence the <NUL> character
MUST be stripped away."

Here, when my FtpClient writes the thirteen bytes "CWD
/\r\0NOOP\r\n", I see the munnari.oz.au:9000 FtpServer
replies only to the "CWD /" part, as if it saw CR LF
EOP where I wrote CR NUL:

        C> CWD /%0DNOOP
        s> 250 CWD command successful.

(By "EOP" I mean end-of-packet.  To see EOP anywhere
but at the end of a packet is to discard all the data
that follows within the packet.)

Here, when my FtpClient writes the the twelve bytes
"CWD /\nNOOP\r\n", I see the munnari.oz.au:9000
FtpServer replies _twice_ rather than once, as if it
saw CR LF where I wrote LF:

        C> CWD /%0ANOOP
        s> 250 CWD command successful.
        s> 200 NOOP command successful.

Can anyone here easily reproduce my observations?

[Anybody even interested?]

Have I grossly misunderstood how to escape & delimit
Ftp commands sent across the control connection?

Maybe "everyone knows" that rfc2640 [International
Ftp] is tongue-in-cheek where it says it has abandoned
the rfc959 restriction forbidding \r and \n in
pathnames?

Just as rfc959 [Ftp] was tongue-in-cheek where it said
\0 may appear in pathnames?

Thanks in advance.    Pat LaVarre

P.S.

The port number "9000" appears as a "whole word"
nowhere outside this thread in my local cache of the
<http://w3.hethmon.com/ftpext/zips/> archives.

P.P.S.

Most of the FtpServer's I've met respond as
munnari.oz.au:9000 does i.e. ftp.apple.com,
ftp.sun.com, members.aol.com, ....

But ftp.microsoft.com:21 differs usefully:

        C> CWD /%0ANOOP
        s> 250 CWD command successful.
        C> CWD /%0DNOOP
        C> QUIT
        s> 550 /QUIT: ... cannot find the file ...

Seemingly this ftp.microsoft.com:21 FtpServer is
seeing CR LF EOP where I wrote LF and seeing EOP where
I wrote CR NUL.

Therefore I can't believe my FtpClient is writing
other than what I tell it to write without first
dreaming up a theory to explain both the
munnari.oz.au:9000 and the ftp.microsoft.com:21 kinds
of responses.

To say the misbehaviour is on my end, I'd also have to
explain why my own FtpServer running on my local
notwork (not just localhost) says that my FtpClient is
writing what I tell it.

__________________________________________________
Do You Yahoo!?
Send FREE video emails in Yahoo! Mail!
http://promo.yahoo.com/videomail/

---

From: Alun Jones <[email-masked]>
Date: Thu, 17 Jan 2002 09:03:32 -0500

>\0 may appear in pathnames?

That's really quite insulting of you to suggest that either RFC was 
tongue-in-cheek.  The RFCs are technical documents of the way that 
implementations _should_ work.  As I've told you several times before, 
there are many reasons - perhaps even good reasons - why implementations 
don't work the way that the RFCs document.  This does not mean that either 
one is poking fun at the Internet community in general.

While it's somewhat confusing (and galling) to see RFC after RFC describe 
FTP URLs in a manner completely contradictory to the way they are 
implemented by most browsers, the case you're discussing isn't nearly as 
bad as that.  You're talking about a very exceptional circumstance as if 
it's a common occurrence, and then asserting (without any supporting 
information) that these exceptions are handled incorrectly in order to 
annoy people.

Alun.
~~~~

--
Texas Imperial Software   | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place   | http://www.wftpd.com or email [email-masked]
Cedar Park TX 78613-1419  | VISA/MC accepted.  NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.

---

From: Robert Elz <[email-masked]>
Date: Thu, 17 Jan 2002 23:25:51 -0500

Date:        Wed, 16 Jan 2002 19:33:53 -0500
    From:        Pat LaVarre <[email-masked]>
    Message-ID:  <[email-masked]>

  | > > [email-masked] 1/16/02 11:39am
  | > > Yes, it does.
  | > > Try port 9000.
  | ...
  | > aka 128.250.22.2:9000

Oh well, once that wasn't public information.  Never mind...

  | Is munnari.oz.au:9000 intended as a reference
  | implementation of an FtpServer?

No.  Only of the MLST/MLSD commands (and even then not really
a reference implementation, most just a sample implementation,
were it a reference the code would be available, and it isn't,
it is way too ugly).

  | I'm pleased to see this FtpServer includes the UTF8
  | option among the lines of its 211 reply to FEAT ...

Probably it shouldn't.   I used it to generate a lot of the examples
in the mlst draft, and as the draft says, *every* example in it comes
from an actual exchange with one ftp server or another (some of the
output sometimes deleted, absolutely nothing invented).

To get an example with UTF8 in the Feat, I needed a server to say that...

  | but I don't yet understand how it complies with
  | rfc2640 [International Ftp]: "On receipt of a pathname
  | containing a <CR><NUL> sequence the <NUL> character
  | MUST be stripped away."

It most likely doesn't.   I will look at that sometime.

  | Here, when my FtpClient writes the thirteen bytes "CWD
  | /\r\0NOOP\r\n", I see the munnari.oz.au:9000 FtpServer
  | replies only to the "CWD /" part, as if it saw CR LF
  | EOP where I wrote CR NUL:
  | 
  |         C> CWD /%0DNOOP
  |         s> 250 CWD command successful.
  | 
  | (By "EOP" I mean end-of-packet.  To see EOP anywhere
  | but at the end of a packet is to discard all the data
  | that follows within the packet.)

No, there is certainly no knowledge of packets in there, with TCP
only a very broken implementation can possibly see that.

What actually happens is that the command is read (the whole thing,
up to the \r\n) then it is all treated as a C string (terminated by
a null) and then the \r before the end of string (which was previously
assumed to be a \n) is removed.    Packet boundaries have nothing at
all to do with it (send more in the same packet, you'll verify that
easily enough).

And yes, treating this stuff as C strings is broken - but it is also easy,
and doesn't actually affect anything (on this VFS, there are no legal
pathnames that contain a \0 so none will ever be sent, and there's no
rational reason for a client to ever send one).

  | 
  | Here, when my FtpClient writes the the twelve bytes
  | "CWD /\nNOOP\r\n", I see the munnari.oz.au:9000
  | FtpServer replies _twice_ rather than once, as if it
  | saw CR LF where I wrote LF:

Yes,  it will do that too.

  | Can anyone here easily reproduce my observations?

There is no need.   I'm quite sure they're all correct.

  | [Anybody even interested?]

Not very.

  | Have I grossly misunderstood how to escape & delimit
  | Ftp commands sent across the control connection?

No.

  | Maybe "everyone knows" that rfc2640 [International
  | Ftp] is tongue-in-cheek where it says it has abandoned
  | the rfc959 restriction forbidding \r and \n in
  | pathnames?

No.   But that they aren't forbidden doesn't mean that any particular
server is required to support names like that.

  | Just as rfc959 [Ftp] was tongue-in-cheek where it said
  | \0 may appear in pathnames?

And the same there.

  | The port number "9000" appears as a "whole word"
  | nowhere outside this thread in my local cache of the
  | <http://w3.hethmon.com/ftpext/zips/> archives.

No, until now, only a comparatively small set of people knew of that
server ...  No guarantees it will remain forever (the implementation
is not very effecient).   And I guess now I'm going to have to disable
the "anyone can write" directories that that server provided.

  | Most of the FtpServer's I've met respond as
  | munnari.oz.au:9000 does i.e. ftp.apple.com,
  | ftp.sun.com, members.aol.com, ....

They're most probably hosted on unix systems, and all derived from
the same code base.

  | But ftp.microsoft.com:21 differs usefully:

But that one almost certainly isn't.

  | To say the misbehaviour is on my end,

No-one is saying that, are they?

kre

---

From: "Mark Symons" <[email-masked]>
Date: Fri, 18 Jan 2002 00:02:41 -0500

| > > Try port xxxxx

> Oh well, once that wasn't public information.

Aghhh!  Sorry,  I was not thinking.

> Never mind...

You can say that.  I still feel embarrassed. =)

> To get an example with UTF8 in the Feat, I needed

Try anon login here:

ftp://ftp.ipswitch.com

FEAT
211-Extensions supported
 SIZE
 MDTM
 MLST size*;type*;perm*;create*;modify*;
 LANG EN*
 REST STREAM
 TVFS
 UTF8
 AUTH SSL;TLS-P;
 PBSZ
 PROT C;P;
211 end

This is WS_FTP Server v3.00.  The current version is actually v3.01.  I have
not personally tested WS_FTP server myself in the last few months - I am
totally tied up with another product.  But I will always take time out to
chase things if anyone reckons that there is odd behavior vis-a-vis
compliance with the MLST or similar drafts.

Mark Symons
Ipswitch, Inc
Augusta GA