Subject: Re: [BEHAVE] one week WGLC, draft-ietf-behave-ftp64-05

From: Dave Thaler <[email-masked]>
Date: Tue, 21 Dec 2010 18:11:29 +0000

Iljitsch van Beijnum wrote:
> On 30 sep 2010, at 23:43, Dave Thaler wrote:
> > The technical comment is that the text currently breaks
> > internationalization as specified in RFC 2640 section 4, whereby the
> > text part of responses can be
> > UTF-8 non-English characters.  Currently several places in draft -05
> > either appear to specify the literal response (in English) or constrain the syntax
> to be ALPHA.
> > In contrast RFC 2640 section 4 specifies %x01-%xFF instead of ALPHA.
[...]
> I didn't change the freetext specification for RFC 2640 compliance because I'm
> afraid this may break certain clients and the ALG is not in a good position to
> negotiate language or UTF8 support. I added this text:
> 
> <xref target="RFC2640" /> specifies the ability for clients and servers to
> negotiate the language used text messages between the two of them. This will
> typically involve the use of UTF-8 encoded characters outside the 7-bit ASCII
> range. Because client compatibility with RFC 2640 is unknown, ALGs need adopt
> a conservative position in this regard. As such, an ALG MUST NOT use any
> characters outside of the 7-bit ASCII range in the text that it generates.
> However, an ALG MUST transparently forward commands and responses
> containing UTF-8 encoded text when those occur.
[...]

That's not sufficient.

RFC 2640 section 4.1 states:
    Once negotiated, server-PI
    MUST return server messages and textual part of command responses in
    the negotiated language and encoded in UTF-8.

RFC 2640 section 4.3 states:
    A server-FTP process that supports the LANG command, and language
    support for messages and command responses, MUST include in the
    response to the FEAT command [RFC2389], a feature line indicating
    that the LANG command is supported and a fact list of the supported
    language tags.

So if the FTP ALG allows the LANG command to succeed, and the FTP64
then generates text in a language (like English) other than the negotiated
one, *that's* what "may break certain clients" as you say, because that
would appear to the client that the server is violating the MUST in
RFC 2640 section 4.1 above.  Some languages might not even be representable
in VCHAR's.   If you believe "the ALG is not in a good position to negotiate
language or UTF8 support" then you would want to prevent this from 
occurring.  I think the options for an FTP64 are

a) always reject LANG command with an error-response (RFC 2640 section 4.2)
    In this solution, I think it should process the FEAT response and remove
    the fact that the LANG command is supported, or it may break certain
    clients that may expect it to succeed when the FEAT says it will.

b) reject the LANG command with an error-response whenever the FTP64
    cannot generate text in the specified language.   In this solution,
    I think it should process the FEAT response and remove any language
    tags that it doesn't support, and remove the fact that the LANG command
    is supported if the list becomes empty.

c) appear to violate the section 4.1 MUST and generate text in a language
    other than what the client expects.   This is what the draft currently implies
    and I think this is broken.

I think (a) would be the simplest solution, though it prevents internationalization.
(b) is the more powerful solution, but is probably overkill for a transition
mechanism that we don't want to be the recommended solution.

-Dave

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 22 Dec 2010 00:00:51 +0100

On 21 dec 2010, at 19:11, Dave Thaler wrote:

>> As such, an ALG MUST NOT use any

> I think the options for an FTP64 are

>    clients that may expect it to succeed when the FEAT says it will.

I don't like this, because it's too tricky. I don't know of any implementations that support RFC 2640, and if those are not tested against this solution can easily create subtle bugs.

>    is supported if the list becomes empty.

See above.

>    and I think this is broken.

I don't think there is any solution that conforms to the spirit of RFC 2640: that would be to provide text in the negotiated language. Rejecting language negotiation may be correct protocol-wise, but it really doens't help the user to get the text from the server in a non-desired language just because the ALG can't generate responses in that language.

I'm going to ask on the ftpext list to see if anyone knows about RFC 2640 implementations. If not, then the most prudent course of action would be to move RFC 2640 to historic, IMO.

---

From: Dave Thaler <[email-masked]>
Date: Wed, 22 Dec 2010 00:32:03 +0000

To your later question, Microsoft's IIS does support RFC 2640
in its FTP server.  Microsoft's current FTP clients, on the other hand, do not
support RFC 2640.

-Dave

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 22 Dec 2010 12:40:38 +0100

[CC to ftpext, please prune as desired when following up]

> in its FTP server.  Microsoft's current FTP clients, on the other hand, do not

And so does ProFTPD.

I quickly checked maybe a dozen FTP sites. A few of them supported LANG, although none any languages other than English, one UTF8 but not LANG, one LANG but not FEAT (which is in non-conformance to RFC 2640). So I guess a quick deprecation is off the table.

After reading RFC 2640 a bit more closely, it turns out that it actually allows CR/LFs in filenames, but those are escaped using a null character. This probably warrants a warning in the FTP64 document, as ALG implementers may not be aware of this and of course a null character has special meaning in languages such as C.

Back to the original problem.

Ideally, the ALG would observe the LANG negotiation and then send text in the desired language. However, there is no way to make sure that the ALG supports all the languages that clients and servers may support, so even if the ALG makes this attempt, there will be cases when it can't provide text in the desired language.

When that happens, there are two choices:

1. block the LANG negotiation and use the default language

2. ignore the LANG negotiation and use the default language

The advantage of 1 is consistency. However, this is more complex to implement and despite some evidence of LANG support I fear that actual use of this feature is too low to see good real-world testing, so there is a significant risk of bugs that won't come to light until the ALG is widely deployed. Another downside is that the user is now deprived of text in the desired language for text sent by the server that isn't touched by the ALG. Most notably, this would be the greetings. With failed LANG negotiation, users probably also wouldn't be able to use UTF-8 user names, passwords or path names.

The advantage of 2 is implementation simplicity and that the majority of the text is still in the desired language. However, some server responses will be in the ALG's (default) language rather than in the negotiated language. (But informed users can still see what's going on from the numeric response, the text is merely additional information.)

I propose the following:

- provide some explanation of the issue in the document
- ALGs MUST support UTF-8 and the CR0LF thing
- ALGs MAY monitor the LANG negotiation and adjust their language accordingly (with no opinion about what happens when the ALG supports the user desired language but the server doesn't)
- if the ALG doesn't monitor the LANG negotiation or doesn't support the negotiated language, it uses its default language

Thoughts?

---

From: Dave Thaler <[email-masked]>
Date: Wed, 22 Dec 2010 16:40:06 +0000

> -----Original Message-----
> From: Iljitsch van Beijnum [mailto:[email-masked]]
> Sent: Wednesday, December 22, 2010 3:41 AM
> To: Dave Thaler
> Cc: [email-masked]; Behave WG; [email-masked]
> Subject: Re: [BEHAVE] one week WGLC, draft-ietf-behave-ftp64-05
> 
> I quickly checked maybe a dozen FTP sites. A few of them supported LANG,
> although none any languages other than English, one UTF8 but not LANG, one
> LANG but not FEAT (which is in non-conformance to RFC 2640). So I guess a
> quick deprecation is off the table.
> 
> After reading RFC 2640 a bit more closely, it turns out that it actually allows
> CR/LFs in filenames, but those are escaped using a null character. This probably
> warrants a warning in the FTP64 document, as ALG implementers may not be
> aware of this and of course a null character has special meaning in languages
> such as C.
> 
> Ideally, the ALG would observe the LANG negotiation and then send text in the
> desired language. However, there is no way to make sure that the ALG supports
> all the languages that clients and servers may support, so even if the ALG makes
> this attempt, there will be cases when it can't provide text in the desired
> language.
> 

Yes for LANG (as opposed to the FEAT exchange), those are the same as I 
described in my email.

> The advantage of 1 is consistency. However, this is more complex to implement
> and despite some evidence of LANG support I fear that actual use of this feature
> is too low to see good real-world testing, so there is a significant risk of bugs
> that won't come to light until the ALG is widely deployed. Another downside is
> that the user is now deprived of text in the desired language for text sent by the
> server that isn't touched by the ALG. Most notably, this would be the greetings.
> With failed LANG negotiation, users probably also wouldn't be able to use UTF-8
> user names, passwords or path names.

The last sentence is not correct.   The LANG command only controls the language
of the server's text responses.  It has no effect on the ability for the client
(or server) to use UTF-8 user names, passwords, or path names.

But yes, the advantage is consistency, and the disadvantage is that you're
limited to languages supported by both the server and the ALG.

> 
> The advantage of 2 is implementation simplicity and that the majority of the text
> is still in the desired language. However, some server responses will be in the
> ALG's (default) language rather than in the negotiated language. (But informed
> users can still see what's going on from the numeric response, the text is merely
> additional information.)

To repeat my earlier point, the disadvantages are
1) Text from the ALG may render as garbage (even overwriting the numeric response,
   making it impossible for the user to see what's going on... remember some languages
   are bidirectional so this isn't just about the backspace character)
2) Some clients might even crash or have other unexpected behavior, since
    2 appears to violate RFC 2640 and hence could easily have untested behavior.

To me, the above disadvantages are strong arguments against 2.

> - ALGs MAY monitor the LANG negotiation and adjust their language accordingly
> (with no opinion about what happens when the ALG supports the user desired
> language but the server doesn't)

Personally, I think the above MAY needs to be a MUST.

> - if the ALG doesn't monitor the LANG negotiation or doesn't support the
> negotiated language, it uses its default language
> 
> Thoughts?

-Dave

---

From: Dave Thaler <[email-masked]>
Date: Wed, 22 Dec 2010 17:19:08 +0000

One more comment...

[...]
> > Ideally, the ALG would observe the LANG negotiation and then send text
> > in the desired language. However, there is no way to make sure that
> > the ALG supports all the languages that clients and servers may
> > support, so even if the ALG makes this attempt, there will be cases
> > when it can't provide text in the desired language.
> >
> > The advantage of 1 is consistency. However, this is more complex to
> > implement and despite some evidence of LANG support I fear that actual
> > use of this feature is too low to see good real-world testing, so
> > there is a significant risk of bugs that won't come to light until the
> > ALG is widely deployed. Another downside is that the user is now
> > deprived of text in the desired language for text sent by the server that isn't
> touched by the ALG. Most notably, this would be the greetings.
> > With failed LANG negotiation, users probably also wouldn't be able to
> > use UTF-8 user names, passwords or path names.
> 
> of the server's text responses.  It has no effect on the ability for the client (or
> server) to use UTF-8 user names, passwords, or path names.
> 
> > The advantage of 2 is implementation simplicity and that the majority
> > of the text is still in the desired language. However, some server
> > responses will be in the ALG's (default) language rather than in the
> > negotiated language. (But informed users can still see what's going on
> > from the numeric response, the text is merely additional information.)
> 

Of course, in theory, neither of the above *should* be a problem 
if implementations deal with text correctly.  I guess the main point
is about RFC compliance and levels of testing.

I wouldn't want to apparently violate RFC 2640 without the 
FTPEXT group's consent, so that would be the question to the FTPEXT list.

-Dave

---

From: "Dan Wing" <[email-masked]>
Date: Thu, 23 Dec 2010 10:06:36 -0800

<soapbox>
This thread underscores why standardizing an ALGs is terribly 
difficult.  In this case, the addition of a new feature (LANG
command) on the client and server requires the ALG to either
break that new feature or to fully support the new feature.
For the ALG to fully support LANG, that means an ALG has 
to support all possible languages that can be expressed by 
LANG.

... and, remember, FTP is a "simple" protocol.
</soapbox>

The best solution I see is the FTP64 ALG has to (MUST) 
remove LANG from the server's FEAT response, and translate
the LANG command to NOOP and return a 5xy response to the
client.

If an FTP client, running on an IPv6-only machine, does not
like LANG being disabled, there are two solutions:

  1. the FTP client has to implement the "ALGS DISABLE64" 
     command, and ensure the IPv4-only FTP server it is 
     using properly supports the EPSV command.  This
     is, effectively, draft-ietf-ftpext2-ftp64-00.
OR
  2. have the FTP server add an AAAA record, and properly
     support the EPSV command.  This causes the FTP traffic
     to not use the NAT64 or FTP64 ALG.

-d

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Dave Thaler
> Sent: Wednesday, December 22, 2010 9:19 AM
> To: Dave Thaler; Iljitsch van Beijnum
> Cc: [email-masked]; [email-masked]; Behave WG
> > The last sentence is not correct.   The LANG command only controls
> the
> > language
> > 1) Text from the ALG may render as garbage (even overwriting the
> numeric
> > response,
> >    making it impossible for the user to see what's going on...
> remember some
> > languages
> _______________________________________________
> Behave mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/behave

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 23 Dec 2010 19:12:58 +0100

> the LANG command to NOOP and return a 5xy response to the

But how does making the server use the default language make the user's experience any better?

---

From: "Dan Wing" <[email-masked]>
Date: Thu, 23 Dec 2010 10:27:01 -0800

> Sent: Thursday, December 23, 2010 10:13 AM
> To: Dan Wing
> Cc: 'Dave Thaler'; [email-masked];
> [email-masked]; 'Behave WG'

The user might have to read something in the FTP server's default
language, but it will be ASCII.  That's better than the FTP client
potentially throwing an error.

-d

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 23 Dec 2010 22:42:48 +0100

> potentially throwing an error.

Dave mentioned the client potentially crashing. I don't buy that. An FTP client can obviously handle ASCII if the desired LANG is not supported, so then it's going to crash when it sees the same characters when another language is negotiated but it otherwise sees the same text. It's not like 7-bit ASCII characters never occur in languages that use other scripts.

Maybe at this point in the discussion it would be good to hear from implementers. There are already tons of NAT44 FTP ALGs out there, I wonder how those solve this.