Subject: Re: Two comments on the filexfer draft

From: Jeffrey Altman <[email-masked]>
Date: Wed, 17 Jan 2001 17:16:41 EST

Using UTF-8 (or UCS2) for filenames raises all sorts of issues.

 . normalization is required to prevent files with duplicate names
   from being created on the receiving system; or the sending system
   transmitting two files with equivalent names but different 
   representations to the receiver

 . normalization is required because it is very likely that given
   two files with equivalent names it is only possible to access
   one and not the other.  This could be a security hole in that I
   can cause a file to be uploaded which will be accessed instead
   of the file that is really desired.

 . normalization as defined by the Unicode committee must be done
   by the sender and the receiver must reject any string that is 
   not normalized

 . normalization is a nightmare.  Performing normalization requires
   the every character in the string be looked up in a database to 
   determine its shortest form.  There are also compatiblity issues.
   When determining equivalence do you expand every character to base
   character and sorted list of modifiers and then perform the
   comparison (extensible) OR do you reduce every multi-byte
   representation to a single character (when possible) and then 
   perform the comparison (not backward compatible).

But before we even make this decision to use UTF-8, we need to decide
if we want to preserve character set information in file names.  At
the moment most file systems do not have a well defined character
set.  NTFS uses Unicode as does Plan 9 and BFS (BeOS); but most other
file systems are ignorant of character sets.  The file names are just
a series of 8-bit bytes.  It is up to the applications to determine
what the bytes mean.  

If we do not think that we can determine the character-set associated
with a file name what should we do?

We can't perform a translation from a random series of bytes to
Unicode; nor can we translate from Unicode to an unknown character
set.

 Jeffrey Altman * Sr.Software Designer      C-Kermit 7.1 Alpha available
 The Kermit Project @ Columbia University   includes Secure Telnet and FTP
 http://www.kermit-project.org/             using Kerberos, SRP, and 
 [email-masked]          OpenSSL.  SSH soon to follow.

---

From: [email-masked] (Niels M�ller)
Date: 18 Jan 2001 13:57:09 +0100

>  . normalization is a nightmare. Performing normalization requires

From draft-duerst-i18n-norm-04.txt, at least checking that a string is
in normalized form seems simpler than normalizing it. I've written
code to normalize unicode strings (into expanded form), and it's
not terribly difficult, but I'd still prefer not having to do it in
ssh implementation, in particular not in the server.

I'd also like to quote the main principles from that draft:

: For protocol elements used as identifiers, this document recommends
: Internet protocols to specify the following:
: 
: - Comparison SHOULD be carried out purely binary (after it has been
:    made sure, where necessary, that the texts to be compared are in
:    the same character encoding).
: - Any kind of text, and in particular identifier-like protocol
:    elements, SHOULD be sent normalized to Normalization Form C.
: - In case comparison fails due to a difference in text normalization,
:    the originator of the non-normalized text is responsible for the
:    failure.
: - In case implementors are aware of the fact, or suspect, that their
:    underlying infrastructure produces non-normalized text, they SHOULD
:    take care to do the necessary tests, and if necessary the actual
:    normalization, by themselves.
: - In the case of creation of identifiers, and in particular if this
:    creation is comparatively infrequent (e.g. newsgroup names, domain
:    names), and happens in a rather centralized manner, explicit checks
:    for normalization SHOULD be required by the protocol specification.

> the moment most file systems do not have a well defined character

There are a few different cases to consider:

1. Communication between a unicode-aware system and a system that uses
   a smaller character set like iso-8859-1.

2. Communication between two systems using small character sets.

3. Communication between two unicode aware systems.

To get (1) to work, using utf8 for all names seems like a good idea.

For (2), if we ignore character sets issues completely, it will always
"work", but with confusing results unless the character sets on both
sides are the same. The problems should be familiar to anyone using
characters beyong usascii.

For (3) names on the wire will be utf-8. It doesn't really matter if
they are in utf-8 because we require utf-8, or because it happens to
be the natural octet-string representation of the system's native
character set. Either way, differences in canonicalization will cause
subtle problems.

The most interesting cases are (1) and (2). If we want to facilitate
communication between unicode and non-unicode systems, we should use
utf-8 (this seems to be the ietf recommended choice). But then
communication in case (2) will fail completely (not just produce
confusing names) if the systems use distinct character sets. We either
have to accept that, or not mandate utf-8, which in turn leads to
mechanisms for explicitly declare what character set is being used.

> Unicode; nor can we translate from Unicode to an unknown character

We could say that if an implementation have no idea of what character
set is being used, it should interpret each byte as a latin1
character, and then convert the string to utf-8. This conversion at
least doesn't not destroy any information, and it's very simple, if
normalization form C is applied (draft-duerst-i18n-norm-04.txt).

I think the first question we have to ask is if it is important to be
able to use filenames that are random byte strings, with no sensible
character set? If I understand Tatu right, at least he thinks so (I
haven't made up my mind yet). If so, I think it is a mistake to
mandate utf-8, it's better to leave the interpretation of filenames 
mostly undefined like the current (-00) draft, and add a separate
message like

  uint8  SSH_FXP_UOPEN
  string filename [ in utf-8, normalization form C ]
  uint32 id
  uint32 pflags
  ATTRS  attrs

and similarly for SSH_FXP_UOPENDIR. The rest of the requests can be
used as they are, with the understanding SSH_FXP_NAME uses the same
conventions as the request it is a response to.

> NTFS uses Unicode as does Plan 9 and BFS (BeOS)

How do these filesystems deal with normalization?

/Niels