Subject: Internationaliztion: UTF-8 for file names

From: "Joseph Galbraith" <[email-masked]>
Date: Wed, 20 Mar 2002 15:23:11 -0700

As currently specified, I do not believe SFTP meets
the requirements of "IETF Policy on Character Sets
and Languages" [RFC2277].

Last time this issue was raised, it stalled out on
issues of normalization.

I've now read the NFS specification on the matter
[RFC3010, section 11.]  NFS elected to not deal
with the normalization.  I would say that this
would be a great deal better than what we have
now (which is non-interoperability for filenames
that are not 7-bit US-ASCII).  I would also
say that if it was good enough for NFS, it is
probably good enough for SFTP.

I would propose that we reference the 
RFC3010, section 11 and specify UTF-8
for filenames.

On particularly compelling argument 
(at least to me) specified in RFC3010
is that a universal character set is
required because various path components
may be specified in different character-sets.

So, for example, trying to negotiate the
char-set in use during protocol startup or
even on a per path basis, is insufficient.

- Joseph

---

From: [email-masked] (Niels Möller)
Date: 21 Mar 2002 13:41:37 +0100

"Joseph Galbraith" <[email-masked]> writes:

> with the normalization.

I don't think that's correct. NFSv4 doesn't require normalization on
the wire, but it *does* say that implementations must deal properly
with normalization. Quote of the end of section 11 (my enphasis):

   The NFS version 4 protocol does not mandate the use of a particular
   normalization form at this time.  A later revision of this
   specification may specify a particular normalization form.
   Therefore, the server and client can expect that they may receive
   unnormalized characters within protocol requests and responses.  If
   the operating environment requires normalization, then the
   implementation *MUST NORMALIZE* the various UTF-8 encoded strings
   within the protocol before presenting the information to an
   application (at the client) or local file system (at the server).

I admit the text is a little vague, but my interpretation is that if
the operating system's open() function deals properly with unicode
equivalence and do the right thing (unlikely on a unix system, don't
know about windows), then the nfs implementation can pass
filenames directly to open without normalizing then. However, if the
open function distinguishes between various equivalent forms of
filenames (i.e. not implementing unicode equivalence in the way that
is *required* for unicode conformance), then the NFS code must perform
normalization to the form the rest of the system uses.

I'd prefer always sending normalized data on the wire, but it is
acceptable with unnormalized data as long as implementations are
*required* to do the right thing.

> even on a per path basis, is insufficient.

I don't understand why you can't negotiate characterset using some
extension at the initial sftp handshake. I understand that there are
some problems with what rfc 3010 calls "local character sets", but I
see no problem with an extension that you use at startup that says
"let's use the universal utf8 characterset/encoding".

I also find the discussion in RFC3010 a little strange. If one ever
uses different character sets for different components of a path name,
one should expect some problems. I agree totally with that. But that
isn't a problem with sftp or nfs: you'll get about the same problems
if everything is on a single local filesystem. So I'll put those
problems in in the "broken local configuration" category.

I think the best way forward with utf8 support is that someone who
wants it writes up draft specifying an extension for enabling utf8
file names (or an extension for negotiating arbitrary character sets),
and then we can iron out the details in the wg.

/Niels

---

From: "Joseph Galbraith" <[email-masked]>
Date: Thu, 21 Mar 2002 08:54:56 -0700

> problems in in the "broken local configuration" category.

That depends on the operating system.

NT is perfectly capable of supporting this,
and having it work localy.

I name a directory using cyrillic, and put files named
in Japenese inside of it.  Not a problem.  That's because
NT uses a unicode to store file names.

(Now, I admit, this won't work if you are using the FAT file
system, but that's a different story, but it is possible for
this to work perfectly using NTFS.)

My guess is that there are other filesystems
out there that use unicode for file names.
BeOS's filesystem did, if I remember correctly.

> and then we can iron out the details in the wg.

Well, we could go that route, however, in that case, 
when UTF-8 is not in use, we must specify what charset
is in use, according to my reading of RFC2277 3.1.

Also, my reading of RFC2277 3.1 leads me to believe that
not using UTF-8 is at least discouraged for new protocols.

I really think just specifying filenames as being encoded
in UTF-8 is the best solution.  UTF-8 is already used
throughout the other pieces of SSH; it complies with RFC2277,
and it allows systems that can support multiple char-sets to
work.

It also doesn't require us to register with IANA charset
specifications, or try to use some existing specification
(which unless it's Code Page information is hard for me --
and I really don't want anyone in the world to even think
about code pages unless they MUST :-)  Not to mention that
code pages are poorly specificied and non-standard.

We might be able to go with some other solution and still be
in compliance with RFC2277 as long as UTF-8 could be negotiated,
but that just complicates things because then we have to deal
with a client and server that can't come to an agreement on
what charset to use.

I really think my users would be happiest if this stuff just
worked, which is possible with UTF-8 required, but seems
unlikely otherwise.

- Joseph

---

From: [email-masked] (Niels Möller)
Date: 22 Mar 2002 14:55:08 +0100

> NT uses a unicode to store file names.

Then I'd say you're not using different character sets, all parts of
the filename is stored as unicode in the filesystem. There's a system
wide policy that filenames are always stored in unicode.

In this case (now I'm talking about a generic OS with unicode
filenames and locales; I don't know if and how windows locales work in
detail), if I use a locale that says that I only understand filenames
in latin-x, filenames I create will be converted from latin x to
unicode before they reach the real filesystem. When I access the
filesystem, names that are representable in latin-x get converted
before I see them, and if I want to access files using japanese
filenames, I'll simply lose in one way or the other.

I claim this problem is *irrelevant* for NFS and sftp: It's good
enough if things that *work* using a local filesystem works over the
network. For configurations that cause trouble even on a local
filesystem, it's fine if we don't get it right either (although we should
fail as gracefully as possible).

> BeOS's filesystem did, if I remember correctly.

Unix filesystems (or any eight-bit clean filesystem) support unicode,
the only thing you need to do is to create a system wide policy that
says that all filenames should use utf-8 using some particular
normalization form, in particular normalization is needed for the "/"
character.

> is in use, according to my reading of RFC2277 3.1.

I'll not argue about IETF policies, as I'm not familiar enough with
them.

> and it allows systems that can support multiple char-sets to

That is acceptable to me IF and ONLY IF the spec says which party is
responsible for implementing the unicode equivalence requirements.
(I don't recall off hand what the other ssh specs says about
normalization of passwords and usernames, but they are also *badly*
broken if they neglect the normalization issues).

> worked, which is possible with UTF-8 required, but seems

As I said above, I can accept utf-8 on the wire, iff the normalization
stuff is addressed properly. But you should be aware that it does
*not* magically solve all character set problems problems.

For instance, consider a unix system where one user uses latin-1 for
his filenames, and another user uses utf-8. The server can't know
this, perhaps there's a system wide policy that utf-8 should be used,
and the latin-1 user ignores it. Then he will not be able to access
his files over sftp (and he actually has a *better* chance with the
current sftp protocol: sftp will work perfectly as long as he uses it
between similar systems).

Don't get me wrong, I'm not saying that handling this situation right
should be a requirement for sftp (it's similar to the situation
described in my first paragraphs that I argue we need not solve), I'm
just giving an example of a real problem that isn't solved by adopting
utf8 on the wire. utf-8 is a more or less universal character set. It
is not a magic wand.

Regards,
/Niels