Subject: Re: Two comments on the filexfer draft

From: Jeffrey Altman <[email-masked]>
Date: Thu, 18 Jan 2001 9:31:48 EST

> > if we want to preserve character set information in file names.  At
> > the moment most file systems do not have a well defined character
> > set.
> 
> There are a few different cases to consider:
> 
> 1. Communication between a unicode-aware system and a system that uses
>    a smaller character set like iso-8859-1.
> 
> 2. Communication between two systems using small character sets.
> 
> 3. Communication between two unicode aware systems.
> 
> To get (1) to work, using utf8 for all names seems like a good idea.
> 
> For (2), if we ignore character sets issues completely, it will always
> "work", but with confusing results unless the character sets on both
> sides are the same. The problems should be familiar to anyone using
> characters beyong usascii.

How can you say that "it will always work"?  It never works unless you
have to be lucky enough to be using the same character set on both
systems.  Kermit has dealt with this issue of character set issues for
almost 20 years.  Kermit explicitly requires the user to specify the
character sets used on the local and remote systems.  And then the
sender translates from the local set to a standard ISO-8869-X (or now
Unicode) character set and the receiver translates back to the 
character set on its system.

If you don't do this you are playing a game of chance.

> For (3) names on the wire will be utf-8. It doesn't really matter if
> they are in utf-8 because we require utf-8, or because it happens to
> be the natural octet-string representation of the system's native
> character set. Either way, differences in canonicalization will cause
> subtle problems.
> 
> The most interesting cases are (1) and (2). If we want to facilitate
> communication between unicode and non-unicode systems, we should use
> utf-8 (this seems to be the ietf recommended choice). But then
> communication in case (2) will fail completely (not just produce
> confusing names) if the systems use distinct character sets. We either
> have to accept that, or not mandate utf-8, which in turn leads to
> mechanisms for explicitly declare what character set is being used.
> 
> > We can't perform a translation from a random series of bytes to
> > Unicode; nor can we translate from Unicode to an unknown character
> > set.
> 
> We could say that if an implementation have no idea of what character
> set is being used, it should interpret each byte as a latin1
> character, and then convert the string to utf-8. This conversion at
> least doesn't not destroy any information, and it's very simple, if
> normalization form C is applied (draft-duerst-i18n-norm-04.txt).

This will fail on Windows because the Microsoft Code Pages used for
storing file names violate the rules of ISO-8859 character sets since
they contain printable characters in the C1 control range.

> I think the first question we have to ask is if it is important to be
> able to use filenames that are random byte strings, with no sensible
> character set? If I understand Tatu right, at least he thinks so (I
> haven't made up my mind yet). If so, I think it is a mistake to
> mandate utf-8, it's better to leave the interpretation of filenames 
> mostly undefined like the current (-00) draft, and add a separate
> message like
> 
>   uint8  SSH_FXP_UOPEN
>   string filename [ in utf-8, normalization form C ]
>   uint32 id
>   uint32 pflags
>   ATTRS  attrs
> 
> and similarly for SSH_FXP_UOPENDIR. The rest of the requests can be
> used as they are, with the understanding SSH_FXP_NAME uses the same
> conventions as the request it is a response to.

Character sets must be translated from the local into some agreed upon
intermediary form.  Otherwise, you run the risk of taking a file name
from Windows (CP1252) and placing it on Unix where all the
applications use ISO-8859-1 and now suddenly you have file names that 
can't be displayed to the user and worse (when using ECMA-48
compatible terminals: VT...) the use of the C1 space for printable
characters can cause the terminal to be shifted into an undesireable
state.

> > NTFS uses Unicode as does Plan 9 and BFS (BeOS)
> 
> How do these filesystems deal with normalization?

Both of these filesystems are Unicode 1.x based.  Unicode 1.x precedes
most canonicalization rules and certainly all of the discussions
regarding security problems with Unicode.  I doubt there is any
normalization performed in these file systems.

 Jeffrey Altman * Sr.Software Designer      C-Kermit 7.1 Alpha available
 The Kermit Project @ Columbia University   includes Secure Telnet and FTP
 http://www.kermit-project.org/             using Kerberos, SRP, and 
 [email-masked]          OpenSSL.  SSH soon to follow.

---

From: [email-masked] (Niels M�ller)
Date: 22 Jan 2001 01:29:03 +0100

Jeffrey Altman <[email-masked]> writes:

> systems.

What I meant with "work" was that you will get the file across,
although perhaps with a file name that is improperly translated. This
is different (and possibly better) than the fatal "File could not be
transferred because of incompatibel file name character sets" error
which you would get if you *do* convert filenames to and from utf-8
(or use some other means to negotiate character sets), and the two
systems don't support the same characters locally.

I'm not saying this is a good idea, I just want to point out that the
problems it causes are well known and people have been able to live
with them before. To avoiding problems, one either avoids using
"strange" characters (which one would *have* to do anyway if character
set translation were done more strictly), or rename the files later. 

> they contain printable characters in the C1 control range.

Huh? Either the character set is considered known, and it is mapped
properly to utf-8, or it is assumed to be bogus, and the characters
are interpreted as latin-1 (with the control characters mapped to
corresponding unicode reserved area in the latin-1 range) for purposes
of mapping it to and from utf-8. This idea may be stupid, but I don't
see how your example breaks it.

I would expect a well-behaved ssh implementation on windows to know
the character set that is used, and that an ssh implementation on unix
will either know what character set is to be used for file names on
the system, or be happy to use a microsoft codepage for some
filenames.

> characters can cause the terminal to be shifted into an undesireable

To me, that would be annoying, but not a terrible problem. And the
alternative to getting files with garbled names is failure to transfer
those files at all.

In another mail you wrote:

> Please read the ISO standards for character-sets.  An application is
> not broken if it cannot display a file name because the file name is
> illegal.  

I'm not sure what "illegal" means here. From my point of view, each
conceivable file name is either legal or illegal (on a given operating
system or file system).

1. If a filename is legal, than applications should be prepared to
   handle it some reasonable way. Applications that don't do that are
   broken.

2. And if a name is illegal, then the system must not allow any
   application to create a file with that name. If the system allows
   that, the system is broken. An example is the NFS deamon in some
   unix kernel that that allowed clients (in particular Macs) to
   create directory entries containing the '/' character.

We're drifting a little off-topic here. One important questions for
the working group ought to be

* Do we need to support file names that are random byte strings?

The only objection to normalization I've noticed so far (from Tatu)
seems to be based on a desire to support file names that are random
byte strings.

Regards,
/Niels