Subject: Re: IUTF8 pseudo-terminal mode

From: Vincent Lefevre <[email-masked]>
Date: Mon, 12 May 2008 02:00:56 +0200

On 2006-01-02 18:20:50 -0500, Jeffrey Hutzelman wrote:
> One could argue that an SSH server running on such a system should look 
> at the configured locale and configure the PTY appropriately, and that's  
> probably even a good idea.

What "configured locale"? The user may use a locale which is not the
default one at the system level. Perhaps you mean that the SSH client
should propagate the locale (more precisely, the charmap) to the
remote side, so that the SSH server can use it (in which case the
server should make this information available at the user level too,
as it can be useful to set up the LC_CTYPE environment variable).

-- 
Vincent Lefèvre <[email-masked]> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

---

From: Nicolas Williams <[email-masked]>
Date: Sun, 11 May 2008 21:00:55 -0500

> What "configured locale"? The user may use a locale which is not the

Whatever the locale for that channel/session/whatever.

> should propagate the locale (more precisely, the charmap) to the

SunSSH 1.1 does (by having the client set per-channel LANG/LC_*
environment variables).

It's less than perfect: the client has no idea what a client-side locale
maps to on the server side.

Also, it's not just character sets that matter, but language for
localization of messages, date formats, etc...

Nico
--

---

From: William Ahern <[email-masked]>
Date: Sun, 11 May 2008 19:17:46 -0700

> should propagate the locale (more precisely, the charmap) to the

Not more precisely. That's less precise on so many levels ;) Locale is
merely incomplete.

Locale and encoding are distinct, _particularly_ as regards Unicode, and
perhaps even for some ISO 2022 "character sets".

Strictly speaking, a character map has nothing to do with formatting, etc,
nor is it synonymous with encoding. UTF-8 is, I suppose, assumed over an
8-bit channel. But UTF-7 or some other scheme is equally possible; UTF-7 is
plausible, even. UTF-16 likely. Many systems will employ both UTF-8 and
UTF-16 together; UTF-16 (for historical misunderstandings which pertain to
this very discussion, but also practically because for Asian languages it's
slightly more efficient) is the internal encoding for most heavy Unicode
APIs--like Java, C#, and IBM's ICU. It doesn't make sense to assume
applications will always translate UTF-8 on both sides to/from UTF-16 over
the opaque data channel.

In practice, of course, the "locale" environment variable often specifies
both locale and encoding. But the distinction should be made, because I18N
continues to get botched by standards and implementation groups.

---

From: Colin Watson <[email-masked]>
Date: Mon, 12 May 2008 10:18:50 +0100

On Mon, May 12, 2008 at 10:07:47AM +0200, Vincent Lefevre wrote:
> On 2008-05-11 21:00:55 -0500, Nicolas Williams wrote:

OpenSSH, as configured in Debian, does this too.

> I meant in a way that always works in practice.
> 
> > It's less than perfect: the client has no idea what a client-side
> > locale maps to on the server side.
> 
> Yes, that's the problem: it's not always possible to rebuild a correct
> LC_CTYPE on the remote side, e.g. if LC_CTYPE is "en_US", one doesn't
> have information about the charset on the remote side.

Locale names are indeed opaque as far as POSIX is concerned, so there's
no portable way to pick them apart. But even if locale names are in
principle identical (i.e. client and server running the same release of
the same operating system), there's a further problem. With glibc,
locale definitions are quite large (thus inconvenient to distribute in
pre-generated form) and take some time to generate, so it's fairly
common for distributions to set things up so that you only generate the
locale definitions you need. The locale you're using on the client may
simply not exist on the server.

Now, in some ways this does end up invoking undefined behaviour; you're
asking for a locale that doesn't exist. Messages will of course end up
being output in the C locale, and so on. But it is *terribly* useful to
at least get the character encoding right, as otherwise you get hopeless
garbage on the screen and it may well not be very obvious what the
problem is.

This is compounded by the fact that there is no equivalent of "C" for
UTF-8 in glibc: that is, there's no way to say "I just want a basically
unlocalised system that happens to use the UTF-8 encoding for
everything". Thus even people who don't care about localisation have to
select something like en_US.UTF-8 or en_GB.UTF-8, and any time they ssh
to a server that doesn't have those locales generated they end up with
screwed-up output from full-screen terminal applications.

> Well, localization of messages and date formats are just a user choice.
> If they are different on the remote side, this isn't really a problem.
> Concerning the character set, the remote one must be compatible with
> the local one, at least when a terminal is used (ditto for the IUTF8
> pseudo-terminal mode). Otherwise the user can't view or edit non-ASCII
> characters correctly.

Absolutely.

-- 
Colin Watson                                       [[email-masked]]

---

From: Vincent Lefevre <[email-masked]>
Date: Mon, 12 May 2008 10:07:47 +0200

---

From: der Mouse <[email-masked]>
Date: Mon, 12 May 2008 09:03:57 -0400 (EDT)

> UTF-8 in glibc: that is, there's no way to say "I just want a
> basically unlocalised system that happens to use the UTF-8 encoding
> for everything".

I suspect part of the reason for this is the difference between octet
streams and character streams.

Older Unices, and even some current ones, don't use character strings
for things like usernames and filenames.  They use octet strings, with
conversion to characters happening very late on display and conversion
from characters happening very early on input, getting whatever
octet<->character mapping the display and input devices happen to use.

And that doesn't work at all well when you have an octet<->character
mapping that doesn't map each to exactly one of the other - such as
UTF-8.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 12 May 2008 09:35:46 -0500

> I meant in a way that always works in practice.

It tends to, though only by dint of luck.

> have information about the charset on the remote side.

There should be a way.  SSHv2 already negotiates languages.  A
negotiation of charsets per-channel wouldn't be too difficult to add.

> Well, localization of messages and date formats are just a user choice.

Yes, but the user doesn't want to be making that choice all the time.

What if the user reads three languages and doesn't know which the remote
server has localizations for?  What if the user doesn't have "dot files"
on the server?  It'd be nice if the client and server could just settle
on a locale that meets the user's needs with no additional input from
the user.

> If they are different on the remote side, this isn't really a problem.

Yes it is.

> pseudo-terminal mode). Otherwise the user can't view or edit non-ASCII

I agree with the first point, but not necessarily with the second (the
client can always put the local terminal into raw mode and let the
server-side IUTF8 mode take over).

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 12 May 2008 10:10:05 -0500

> OpenSSH, as configured in Debian, does this too.

Good!

> Locale names are indeed opaque as far as POSIX is concerned, so there's

Well, no, there's a bit of rhyme and reason to [some] POSIX locale names
-- it just isn't used for all locale names.  But point taken: to
implement a decent locale negotiation protocol one would need an API by
which to ask questions about a locale name (what charset, what language,
what collation, ...).

> simply not exist on the server.

Right.  Add in the fact that users can, and often do speak/read multiple
languages, and there's a real need to negotiate locales.  SSHv2 already
has a language negotiation facility for localization of server messages
intended for the user; it's a start.

So, given a decent locale API one could take the charset of the locale
requested by the user and the languages negotiated for the session, and
search for a best match locale on the server-side.

But still, remotely setting a channel's LC_*/LANG is a pretty poor way
of dealing with this problem.  For one, it isn't documented in any RFC,
so you can't expect implementors to know what to do with it.

> UTF-8 in glibc: that is, there's no way to say "I just want a basically

Nor in Solaris.

> > If they are different on the remote side, this isn't really a problem.

If I can't read the messages then they aren't helpful, but yeah, at the
very least they should display properly.

Nico
--

---

From: Vincent Lefevre <[email-masked]>
Date: Mon, 12 May 2008 18:25:33 +0200

On 2008-05-12 09:35:46 -0500, Nicolas Williams wrote:
> on the server?

There's also the problem that if the user shell is bash, the .bashrc
(or .bash_profile) isn't always read (fortunately, Debian's bash doesn't
have this problem).

> It'd be nice if the client and server could just settle on a locale
> that meets the user's needs with no additional input from the user.

If full locale negociation could be added to SSH, this would be great.

> > If they are different on the remote side, this isn't really a problem.

Well, less than the charset problem. BTW, I generally want some of my
locales to be defined locally on each machine. For instance, I use
LC_TIME=en_DK (to have ISO-8601 dates) under Linux, but this locale
doesn't exist under Mac OS X; worse than that, if I try to use it
under Mac OS X, it makes a terrible mess.

> server-side IUTF8 mode take over).

I don't understand, or is it just that this isn't implemented in
OpenSSH yet?