Subject: I18N & L10N proposal for GSS-API (v2u2/v3)

From: Nicolas Williams <[email-masked]>
Date: Mon, 8 May 2006 15:30:42 -0500

I propose the following solution to the I18N issue:

1) Add constants for identifying character encoding schemes with the
   prefix 'GSS_C_CES_' (e.g., GSS_C_CES_UTF8).

   The type could be INTEGER or OID, I don't care much either way.

2) Add new versions of:

    - GSS_Import_name()
    - GSS_Display_name()
    - GSS_Display_status()

   that take an additional input parameter indicating the character
   encoding scheme used or to be used.

   (And don't forget the display_as_name_type input parameter of
   GSS_Display_name_ext() in the naming extensions I-D.)

3) Specify two GSS_C_CES_* constants to begin with:

   a) GSS_C_CES_UTF8

     Meaning that input human readable strings are encoded in UTF-8, and
     output human readable strings are to be encoded in UTF-8.

   b) GSS_C_CES_DEFAULT

     Meaning that the CES used/to be used is determined in a
     platform-specific manner, and typically [on POSIX/POSIX-like
     systems] being that which is associated with the current locale.

     Note that GSS_C_CES_DEFAULT, by itself, does not create special
     portability concerns because applications that wish to use
     GSS_C_CES_UTF8 on POSIX/POSIX-like platforms in non-UTF-8 locales
     will have to know how to do codeset conversions that they wouldn't
     have to know about in platforms with a single codeset/CES.  That
     is, I18N portability concerns derive from the platforms themselves,
     not the GSS-API.

4) Create an IANA registry of GSS_C_CES_ values/semantics.

5) Deprecate the Latin-1-only functions and note that on some platforms
   they expect/use Latin-1 while on others they expect/use the CES of
   the current locale or equivalent local concept.

6) L10N

   We might as well tackle L10N and give GSS_Display_status() an input
   parameter indicating an RFC3066 language tag or tags to be used for
   localization of error messages.

Nico
-- 

_______________________________________________
Kitten mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/kitten

---

From: Martin Rex <[email-masked]>
Date: Mon, 8 May 2006 22:59:00 +0200 (MET DST)

I would propose to add the following three versions instead:

gss_import_name_utf8()
gss_display_name_utf8()
gss_display_status_utf8()

because I consider it too much of a burden for gssapi mechanisms
to be able to cope with every possible codepage under the sun.

With the limitation to utf8, a gssapi mechanism must only know
how to transform between UTF8 and the (canonical) network encoding
of printables (travelling within gssapi context level tokens or
being communicated out-of-band, as with the Kerberos KDC
communication).  The application designers should have to
figure out (or worry) how many or which codepages (and conversions)
are necessary to support specific environments or scenarios.

I'm just trying to add support for arbitrary codepages at the
gssapi at some point of our application, and it grows the
code by 8-12 MBytes ...

About L10N: Although I don't want to prevent anyone from going
down that rathole, I will continue to use exclusively the
english variant of the error message (if possible) within
our application.  I have been requiring that for interoperability
certification with our application that the gssapi library
produces meaningful error messages in english and manually review
them (at least for those error-situation that gsstest tries).

-Martin

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 8 May 2006 16:02:42 -0500

> to be able to cope with every possible codepage under the sun.

Someone has to: the application or the mechanism.

Perhaps we should settle that first.

I vote for the burden to be on the mechanism -- there are more
applications than mechanism implementations [in my world].

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 08 May 2006 17:02:57 -0400

>    The type could be INTEGER or OID, I don't care much either way.

How about an RFC2978 charset name?  Or, if you really want something 
numberic, the MIBenum values from the same registry.

> 4) Create an IANA registry of GSS_C_CES_ values/semantics.

See above - we should reuse the registry that already exists for this 
purpose, rather than inventing our own.  If we decide the functionality 
represented by your GSS_C_CES_DEFAULT is important, we can provide a way to 
indicate that is the desired behavior (for example, in C, one could pass 
NULL instead of a charset name)

Aside from this, I agree with this proposal as it applies to GSSAPIv3.
For GSSAPIv2 we are only empowered to "clarify", not to define new API 
elements.  So I think all we can/should do is apply the same note you 
describe in your point 5, indicating that there are many implementations 
which use the charset of the current locale rather than Latin-1 as 
specified by GSSAPIv2.

-- Jeff

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 8 May 2006 16:29:51 -0500

> applications than mechanism implementations [in my world].

Allow me to expand as to why (and then offer a concession):

 - Mechanisms MUST deal with stringprep and/or other conversion issues
   and this *cannot* be left to the application.

   I.e., mechanisms already MUST have some of the I18N burden placed on
   them.

   Particularly since mechanisms may have different stringprep
   requirements.

 - There are (or, darn it, should tend to be) more applications than
   mechanisms.

   I.e., placing more of the I18N burden on mechanisms means less code
   than placing more of the I18N burden on applications.  And less code
   is better.

The concession I may be willing to make:

 - REQUIRE support only for GSS_C_CES_UTF8, and REQUIRE that
   GSS_C_CES_DEFAULT be an alias for GSS_C_CES_UTF8 if it isn't
   something else.

   This would mean that portable GSS-APIv3 applications would have the
   burden of doing any charset conversions or requiring that they be run
   in UTF-8 locales only.

   However, applications intended to be portable within the POSIX world
   would typically be able to rely on  which, for my purposes would be
   sufficient.

Eventually most OS vendors will deprecate and stop supporting and
shipping non-Unicode locales, so technically GSS_C_CES_DEFAULT would
ultimately become an alias for GSS_C_CES_UTF8, and which makes
GSS_C_CES_DEFAULT ultimately nothing more than a transition crutch...

...But a very good crutch.

Nico
--

---

From: Martin Rex <[email-masked]>
Date: Tue, 9 May 2006 17:27:38 +0200 (MET DST)

> Perhaps we should settle that first.

the two largest camps of existing mechanisms are:

 - Kerberos and Kerberos-like  either completely ignoring the issue
   or using UTF8 (on the network) like Microsoft's implementation

 - SPKM or SPKM-like using X.509 certificates -- and they do have
   a canonical encoding (inside the certs) of mostly ASCII,
   ISO-8859-1 (=t61string abuse) and UTF8STRING

I don't know how many gssapi mechanisms there are today that
offer to convert a well-defined canonical network encoding
into an arbitrary local codepage.  There are likely VERY few.

Claiming that an implementation supports the locale, while in
fact it completely ignores the problem is not going to help us.
MIT Kerberos and closely related implementations definitely
ignore the issue so far, so the only interoperable codepage
which they support is 7-bit US-ASCII -- not "locale"!

Microsoft's Kerberos might be among those few that really
support the locale on the ASCII-Variants of the SSPI-calls.

> applications than mechanism implementations [in my world].
 
I'm strongly opposed to putting any burden besides
iso-8859-1 and UTF8 onto the GSS-API (v3) mechanisms!

Requiring a gssapi mechanism to grow by 8 MBytes in codesize
in order to support a deprecated-upon-birth functionality sounds
quite unreasonable to me.

-Martin

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 9 May 2006 10:37:45 -0500

> fact it completely ignores the problem is not going to help us.

I didn't make that claim.

> iso-8859-1 and UTF8 onto the GSS-API (v3) mechanisms!

Did you see my follow-up to myself?  Wherein I offer a concession in
this regard that should make you happy?

> quite unreasonable to me.

Huh?  Dynamic linking and all that.

But anyways, in my follow-up to the mail you were replying to I offer to
make only GSS_C_CES_UTF8 REQUIRED, which means that you could have a
compliant mechanism implementation that caters only for UTF-8 and does
no conversions.

Nico
--

---

From: Jeffrey Altman <[email-masked]>
Date: Tue, 09 May 2006 13:16:26 -0400

>From a practical perspective I don't see this proposal as being much
different than adding the "_utf8" versions that Martin is suggesting
and using the existing versions as they are used today.  Applications
will eventually migrate to the _utf8 versions and eventually we will
deprecate the old versions.

I do believe that applications should speak to portable APIs in a
portable character set.  I would like to think that mechanism
implementations are more cross-platform than applications.  By that I
mean that a mechanism should not have to rely on platform specific
functions in order to accomplish their job.  That a GSS-KRB5 mechanism
written for Solaris, would work just as well on Linux and MacOS X.
The set of locales that are implemented on an operating system is
specific to the OS.  I do not believe that GSS mechanism authors
should be required to support the superset of all locales and write
platform specific code to enable character-set translation.

Mechanisms will have to utilize character-set manipulation routines
in order to implement stringprep.  However, those routines are going
to be portable and are not functions that would be provided by the OS
in any case.

Applications on the other hand already have to be locale aware and
the authors need to perform character set manipulation already when
dealing with a mixture of UCS2, UTF8, and non-Unicode representations.
I do not believe that strongly encouraging the use of UTF8 for GSS is a
significant additional burden and by doing so we will improve security
by reducing the potential for character-set confusion to be used to
attack applications which are unaware of the input character-set of
the peer.

Jeffrey Altman (speaking as an individual)

burden.  However, it would
_______________________________________________
Kitten mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/kitten

---

From: Jeffrey Altman <[email-masked]>
Date: Tue, 09 May 2006 13:27:29 -0400

> compliant mechanism implementation that caters only for UTF-8 and does

Nico:

All I see this resulting in is a new negotiation problem.  The
application and the mechanism now have to negotiate over the common
set of character-sets that both support.

If we don't support negotiation and a mechanism is passed
GSS_C_CES_KLINGON what is it supposed to do?  All it can do is
return an error indicating unsupported character-set.  At that point
the application will have to perform translation to UTF8.  While the
concept of selectable character sets has the potential to save a few
hours of application developer programming time, I don't see it
providing any real benefits.  All of the major OSs already provide
application developers with the functions to convert from locale to
UTF8 and back.  Requiring that application developers make such calls
or be UTF8 based prior to using GSSv3 is truly not a big deal in my
mind.

---

From: Michael B Allen <[email-masked]>
Date: Tue, 9 May 2006 13:34:26 -0400

I'm a newbie here so don't weight my input much but I've done a lot of
i18n conscience programming so here's a data point for your stats.

Text in interprocess messages (e.g. network protocol messages) should
be UTF-8. If it has been preestablished that text is of another encoding
(e.g. Latin1) then that should be changed even if it means compromising
backwards compatibility.

Programming interfaces should use the encoding defined by the current
locale. Virtually all APIs are written this way regardless of what the
standards say.

Mike

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 9 May 2006 13:52:13 -0500

> written for Solaris, would work just as well on Linux and MacOS X.

Stop right there.  The GSS-API is supposed to help develop portable
applications, not portable mechanisms.  I see nothing, in fact, about
portable mechanisms in RFC2743.

Also, clearly GSS-API applications typically have to use OS facilities
that can be and are platform-specific, because, for example, GSS-API
applications are intended to be _networked_ applications, and details of
socket/XTI/TLI/other networking APIs vary.  So though the GSS-API
promotes application portability, there's no way that it can guarantee
this.

For example, how do you write a portable server application that
properly localizes messages for its remote users?  Whether the
application uses the GSS-API, SASL or no security framework at all it
still has to deal with this problem and nothing in the GSS-API can help
it as the GSS-API is NOT a general operating system interface -- it's an
interface to security mechanisms.

I'd really rather not rathole on this though.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 9 May 2006 14:00:14 -0500

On Tue, May 09, 2006 at 01:27:29PM -0400, Jeffrey Altman wrote:
> return an error indicating unsupported character-set.

Correct.

And when using GSS_C_CES_DEFAULT, if the default wouldn't be UTF-8 and
it isn't supported, then an error would have to be returned as well.

>                                                        At that point
> the application will have to perform translation to UTF8.

There will be no such applications, but if there were they would fail.

Applications that use GSS_C_CES_DEFAULT should be prepared to fail and
tell users to use a UTF-8 locale.

Applications that use GSS_C_CES_UTF8 should just work (provided the
mechanisms work too).

Applications that use any other CES should be prepared to either perform
their own codeset conversions or fail and, if desired, offer users an
alternative.

>                                                            While the

On platforms where the locale determines the codeset used in many/most
APIs having to modify apps to know how to do codeset conversions for odd
APIs like the GSS-API would be at the very least obnoxious.

My bottom-line: I want a compliant way to do what our applications do
and I don't want to have to modify our applications to perform codeset
conversions.

Nico
--

---

From: Martin Rex <[email-masked]>
Date: Tue, 9 May 2006 21:05:00 +0200 (MET DST)

> locale. Virtually all APIs are written this way regardless of what the

I believe this to be a major misconception!

Few, if any, current APIs *really* support locale.  Quite a lot of code on
Posix/Unix platforms entirely ignores it -- which is completely different.

The only APIs that seem to support locale today are those in the
base language standard (ctype family in the programming language C).

When I started working with Unix machines on a daily basis in 1993,
I was supplied with a HP PARISC workstation.  When working on
the graphical console under X-Windows, the default "terminal emulator"
provided by the OS vendor was the "hpterm" and the default fonts
were the "hp roman" fonts using some codepage that was definitely
not iso-8859-1 -- but the default tty attributes were 7-bit with
parity enabled, so this wasn't the only problem...

My language (German) contains a few non-ASCII characters, and sometimes
I was reading plain text messages with such characters in iso-8859-1,
so I adjusted my settings to use the standard X11 instead.
(I also fixed the tty settings, shell capabilities and switched
to using standard x11 xterms across all of the Unix platforms).

Anyway -- because none of the software was locale-aware (i.e. did NOT
seperate internal/network/filesystem encoding from visualization and
user i/o encoding) it did matter significantly what font one selected!

When I wrote my first program for Microsoft Windows in 1993, it was
a 3270-terminal emulation for accessing IBM Mainframes from a
Windows NT based PC (in order to replace several hundred green-screens).

I was faced with the codepage conversion issue from the beginning
for this program due to the fact that Mainframes use some un-identified
EBCDIC codepage and (our) Windows machines used an ASCII-based
single-byte codepage from the ISO-Latin series.  When adding the
possibility for the user to select an arbitrary font, I had
to take into account which codepage/encoding was used by the font
in order to visualize the characters correctly.
On a Windows system, Fixedsys is an "ANSI" font, whereas "Terminal"
(also known as Raster font in a CMD.EXE) is encoded with DOS/OEM.

The encoding of the font for visualization is entirely seperate
from the codepage that the application uses internally (here the
IBM mainframe in the backend).  Actually, the Mainframe backend
was equally ignorant about codepages as most applications are
(and were at that times), and it was entirely dependent on the
context (or country) what codepage the people wanted to use.

So in addition to the font selection, I added the capability to
select a particular EBCDIC codepage in order to communicate with
the IBM Mainframe.  From what I remember there are 13 western
european codepages that contain the exact same set of characters
from ISO-8859-1, with about a dozen character posistions garbled
(there was some ancient economical reason behind this garbling,
 but in 1993 it appeared like pure nonsense).

Anyway -- we had Germans in the HR department needing Umlauts,
we had developers that needed brackets and braces for programming
in C and we had some users in spain, scandinavia and eastern
european countries which all wanted support for their specific
codepages for their application.

Why am I telling such a long story:
Well, I don't think one can save the application level (which
implements the presentation layer) from doing locale-specific
conversions (e.g. to match font encodings).

So I think there should be a resonable and easy to implement
canonical encoding at the API level, and all of the localization
and codepage conversion issues should be handled by the
application layers -- because they will have to do it anyway
if the application wants to offer localization.

There are areas/scenarios (set-top-boxes, embedded like WLAN-Routers
or palmtops/mobile phones) with serious restrictions on size/space
of the code and there the application must decided how much
localization is necessary and how it is implemented.

If I start notepad on my current WinXPsp2 machine and type some
Umlauts, then change the font from its default (lucida console 10pt)
to Terminal 9pt then I no longer see umlauts, I see mathematical
symbols instead.  If I continue typing umlauts with Terminal font
selected, then I continue to get mathematical symbols, not umlauts.
This might appear consistent -- but it clearly shows that "notepad.exe"
on Windows XP does *NOT* support locale, instead it uses the
"just-use-8" approach and ignores the codepage (of both, the text
it processes and the encoding of the chosen font).
Btw. Microsoft Visual Studio 2005 text editor behaves just the same,
as does vi on Unix (and probably a lot of other text editors on Unix).

-Martin

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 9 May 2006 14:22:21 -0500

Martin, 1993 was a long time ago, and some Unix vendors (see my e-mail
address for one example) care about backwards compatibility, so that as
we internationalize the OS we do look to leverage a current locale's
codeset assumption where we can (e.g., by deprecating non-UTF-8 locales,
say -- which Sun has not yet done, BTW) and we tend to prefer UTF-8, all
because it leads to less code change, not only for our code bases but
also for third parties'.

Nico
--

---

From: Martin Rex <[email-masked]>
Date: Tue, 9 May 2006 22:03:00 +0200 (MET DST)

> because it leads to less code change, not only for our code bases but

Please take into account that there is a significant lag in what
"new features" vendors ship with their most current release of their
software compared to what other software vendors use to build their
software.

E.g. we use Solaris 5.8 (sparc) for build that release of our
software that is most widely used, and we're likely going to build
that particular release on Solaris 5.8 until at least 2010
(even though customers are allowed to install/use the software
 on solaris 5.9 and soon 5.10).

Similarly, there is code that I ship "release-independent",
which must work on solaris unchanged down to SunOS 5.6 and on
Windows down to Win95/NT4 (everything else would significantly
complicate software logistics).  That includes our OEM PKI toolkit
and the Windows-specific gsskrb5.dll wrapper for Microsoft's
Kerberos SSP (gsskrb5.dll shares most code with its brother
gssntlm.dll -- and that can be used on Win95/NT4 as well).

Contrast this to the DLL-nightmare that Microsoft has started
with their MSVCRT-family of DLLs (the C Runtime library),
i.e. do a "dir %SystemRoot%\msvcrt*.dll" on a windows machine
that has seen all all of the compilers since VStudio98 and
a decent set of apps/helpers installed...

Incompatible changes are evil.
Having software logistics in place to hide frequent incompatible
changes from your customers / end users doesn't make it less evil.

If we add only the three utf8-capable API calls to GSS-API v3,
then (a) more gssapi mechanism implementors are likely to adopt
it because they're "fairly" easy to implement, and (b)
application programmers are likely to use them when they find
them because they will get exactly the same functionality
from every gssapi implementation that has these calls.

-Martin

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 9 May 2006 15:23:38 -0500

> software compared to what other software vendors use to build their

I am aware.  If, as you say, you ship applications built for Solaris and
you use Solaris' libgss then you'll have to accept it's current
expectation that the codeset used is that which is associated with the
current locale.

> gssntlm.dll -- and that can be used on Win95/NT4 as well).

At some point third parties have to drop support for OS releases that
are no longer supported by the OS vendors.  Sun does not support any
release of Solaris earlier than Solaris 8, Microsoft does not support
NT4, and so on.

> a decent set of apps/helpers installed...

Huh?  (That's rhetorical.)

> Incompatible changes are evil.

Yes, at least without sufficient warning.

Note that going from "non-ASCII in char * argument to this function has
undefined results" to "this char * argument to this function expects
strings in the CES of the current local" is not an incompatible change
unless the "undefined results" in the original were reliable and came to
be relied upon.

> changes from your customers / end users doesn't make it less evil.

If you'd like to make a complaint about Solaris in this regard please be
specific, and use an OpenSolaris discussion list, not an IETF list.

> from every gssapi implementation that has these calls.

This is a good argument, but doesn't win me over completely because as I
see it GSS-API app developers on POSIX/POSIX-like systems will likely
write wrappers around these functions that perform any necessary codeset
conversions from/to the current locale's codeset and UTF-8 as required
in your proposed gss_*_utf8 functions.

I.e., let us give such developers what they want already.

Nico
--

---

From: Michael B Allen <[email-masked]>
Date: Tue, 9 May 2006 17:23:17 -0400

On Tue, 9 May 2006 21:05:00 +0200 (MET DST)

> Posix/Unix platforms entirely ignores it -- which is completely different.

True, but those programs are just slightly broken. To fix them to *really*
support locales is a matter of:

  1) fixing any code that needs to understand variable width encoding (e.g.
     caseless comparison of UTF-8 strings) and
  2) calling setlocale(LC_CTYPE, "") in main

Newer, robust libraries use this convention because it supports the
desired full Unicode UTF-8 locale as well as folks still running Latin1
on their PARISC from 1993. As long everyone sticks to the plan things
seamlessly cascade together and conversion only occurs when interprocess
communication is necessary (e.g. network protocols and files) or when
interacting with code that does not follow the convention.

Mike

---

From: Martin Rex <[email-masked]>
Date: Tue, 9 May 2006 23:43:34 +0200 (MET DST)

Michael B Allen wrote:
> Martin Rex wrote:

I mentioned it before: traditional multi-byte locales provide real
pitfalls for code that is not locale-aware.

The problem is if the existing code does some amount of parsing on
the string and looks for particular occurrances of certain ASCII
characters in strings, not knowing whether it is dealing with
a single-byte codepage or a traditional multibyte codepage,
because the trail byte of such a multibyte codepage can be
confused with a regular ASCII character.  UTF-8 is safe because
with UTF-8, every octet of a multi-byte character as the highest bit set.

MIT Kerberos 5 does definitely have such a problem with the realm seperator
"@" character -- and every GSS-API mechanism that implement the hostbased
service nametype and is locale-ignorant will also have this problem with
this nametype.

And we still don't know how and where to handle IDN (internationalized
domain names) into punycode conversion for hostbased service names
and within kerberos service principals...

-Martin

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 9 May 2006 16:55:37 -0500

> and within kerberos service principals...

There is plenty of material in the IETF KRB WG mailing list archive and
proceedings.  I would say that we *do* know what to do about the
intersection of IDN and Kerberos V, and punycode is not involved.  (What
matters is the stringprep profiles, not the particular encoding of
Unicode.)

I don't want us discussing Kerberos V I18N, but I do want us to agree
that that that a mechanism-specific issue, not a matter for Kerberos V
or RFC4121 applications -- if we can't reach consensus on this then we
have a problem.  This isn't to say that GSS-API apps shouldn't use UTF-8
natively, mind you, but that there's no inherent link between what a
mechanism does on the wire and what the API requires of applications as
long as the API doesn't force one to use a non-Unicode CES.

Even if we adopt the gss_*_utf8() proposal we'd still not specify
stringprep profiles, or even just normalization forms for gss_*_utf8()
UTF-8 inputs/outputs, but GSS mechanisms that use Unicode on the wire
MUST specify stringprep profiles.

Nico
--

---

From: Michael B Allen <[email-masked]>
Date: Tue, 9 May 2006 18:14:39 -0400

On Tue, 9 May 2006 23:43:34 +0200 (MET DST)

> with UTF-8, every octet of a multi-byte character as the highest bit set.

Pardon me for hopping along half way through this thread but are
you suggesting that users should be forced to convert all text to
UTF-8? Meaning functions like gss_display_name would require UTF-8 as
input? If so, then I doubt people will actually implement that. Let's
think about the two cases:

  Case 1: Force UTF-8
    Program possibly converts to/from UTF-8 to talk to gssapi AND have
    the mechanism lib possibly convert to/from UTF-8 to talk to the lib
    backing the mech.

  Case 2: Use the Locale Dependent Encoding
    Have the mechanism lib possibly convert to/from the locale dependent
    encoding to talk to the lib backing the mech.

> service nametype and is locale-ignorant will also have this problem with

I don't think GSSAPI should try to make up for deficiencies in
other libraries thereby pushing the burden further up the chain onto
users.

If you do that I'm going to invest in neck braces :->

Mike

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 9 May 2006 17:34:31 -0500

>     encoding to talk to the lib backing the mech.

Exactly.  Case 2 is going to be preferable for application developers on
platforms where there is such a concept as 'locales'.

Martin may complain that this means that such applications are less
portable, but that is a choice to be made by the application
developer...

..And my proposal gives framework/mechanism implementors the choice of
whether to force app developers to use UTF-8 or give app developers
the choice of using UTF-8 explicitly or the CES of the current locale.

For all the POSIX-like platforms and because framework/mechanism
implementors will generally have to deal with codeset/normalization/
stringprep issues directly, I suspect that framework/mechanism
implementors will provide the latter choice, and that application
developers on POSIX-like platforms will take the ces-of-current-locale
option, which will not significantly hinder portability to other
platforms that are more in tune with Unicode already.

> other libraries thereby pushing the burden further up the chain onto

The particular "deficiency" Martin was discussing in the text you quote
is not something applications can fix for the mechanism, period.

Forcing the use of UTF-8 at the API does NOT make all instances of that
sort of problem go away, and the mechanism MUST deal with normalization,
prohibited codepoints, unassigned codepoints, etc...

> If you do that I'm going to invest in neck braces :->

Exactly.

Nico
--

---

From: Martin Rex <[email-masked]>
Date: Wed, 10 May 2006 02:00:00 +0200 (MET DST)

Michael B Allen wrote:
> input?

For backward compatibility with GSS-API v2 based apps the original
calls with (de-facto and spec-ignorant) implementation defined behaviour
will still be present.  Noone will force the application to use
the utf8-variants of the three affected gssapi calls
gss_import_name, gss_display_name and gss_display_status.

btw. gss_display_name only has an opaque gss_name_t input handle,
I assume you refer to gss_import_name with a printable nametype.

>     If so, then I doubt people will actually implement that. Let's

Someone will have to do a conversion between the codepage used
by the user and the codepage used by the gssapi mechanism.
Currently there are VERY few gssapi mechanism that are doing
codepage conversions (and capable of doing it for arbitrary,
user-chosen locales).

>     encoding to talk to the lib backing the mech.

can be done through the legacy API with all the known problems,
i.e. what to do with non-representable chars, what about
unsupported locales, ...

Nico answered:
: Forcing the use of UTF-8 at the API does NOT make all instances of that
: sort of problem go away, and the mechanism MUST deal with normalization,
: prohibited codepoints, unassigned codepoints, etc...

When adding _utf8 support into my gsskrb5.dll I'm going to let
Microsoft's APIs fight that out, and if I was a gssapi mechanism
implementor I would ignore normalization and unassigned codepoints,
just do a basic check whether it appears to be corretly encoded
(i.e. if it only contains ASCII plus appropriately formed
 multi-byte sequences).

-Martin

---

From: Michael B Allen <[email-masked]>
Date: Tue, 9 May 2006 21:16:24 -0400

On Wed, 10 May 2006 02:00:00 +0200 (MET DST)

> by the user and the codepage used by the gssapi mechanism.

Right. That's why Case 2 is superior - no conversion between user and gssapi.

> i.e. what to do with non-representable chars, what about

I understand what you're saying but I don't understand how you came to
the conclusion that requiring conversion to UTF-8 or somehow using UTF-8
internally would be a good idea. There's no end of arguments against it:

1) UTF-8 in itself isn't going to help with non-representable chars
  or unsupported locales or anything like that anyway because you get
  out of gssapi what you put in to gssapi. Forcing UTF-8 is just going
  to push the problem out of libgssapi up into the hands of annoyed
  programmers.
2) If you're not running in the UTF-8 locale you probably have a really
  good reason (e.g. Japanese) in which case the programmer/admin/user
  is in a much better position to setup the environment to yield the
  correct behavior.
3) Think about java.lang.String (UCS-2LE), Win32 UNICODE_STRING
  (UTF-16LE), C++ strings (UTF-16), etc. Even talking about UTF-8 in the
  context of these environments makes no sense. GSSAPI implementations
  are going to use whatever the premier string type is for the target
  environment / language (i.e. no one will implement it).

Mike

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 9 May 2006 20:19:40 -0500

> When adding _utf8 support into my gsskrb5.dll I'm going to let
                                    ^^^^^^^^^^^

Is that an implementation of RFC4121?

If so...

> (i.e. if it only contains ASCII plus appropriately formed

...sorry, it's not that easy.  Kerberos V I18N will involve applications
of stringprep, including normalization, prohibited codepoints, mapped
codepoints, etc...

(Now, if we make Kerberos V I18N something that only works with Kerberos
V extensions then we might get away with not stringprepping principal
names on the client side, but youd' still have to stringprep passwords
for password-based pre-authentication.  BTW, I'm a little rusty on krb5
I18N.)

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 9 May 2006 20:26:11 -0500

> internally would be a good idea. There's no end of arguments against it:

BTW, it can be useful to declare UTF-8 a common medium in the middle of
certain systems.  That's an approach we are going to take for filesystem
I18N (the middle being between the syscall and VFS boundaries).

>   is in a much better position to setup the environment to yield the

Well, *we* (OS vendors/developers) should want you to use UTF-8 locales
always -- it simplifies a lot of problems, and if we could do it in one
fell swoop then it would make this problem go away :)

>   environment / language (i.e. no one will implement it).

Stepping back a bit, the *abstract* interfaces would probably be
specified in terms of Unicode (version 4?), not including any particular
encoding thereof.

The C bindings of the *abstract* interfaces would be specified in terms
of UTF-8.  And so on.

But, yes, I agree that specific language bindings will use whatever the
language prefers.

Nico
--

---

From: Luke Howard <[email-masked]>
Date: Wed, 10 May 2006 12:37:12 +1000

>  (UTF-16LE), C++ strings (UTF-16), etc. Even talking about UTF-8 in the

UNICODE_STRING is UCS-2LE, no?

-- Luke

--

---

From: Michael B Allen <[email-masked]>
Date: Tue, 9 May 2006 23:17:08 -0400

On Tue, 9 May 2006 20:26:11 -0500

> I18N (the middle being between the syscall and VFS boundaries).

Would that be a library used by a single process like GSSAPI? If you
were writing UTF-8 to the filesystem then I might agree with your design
as a filesystem is a form of interprocess messaging. However, it does not
sound like interprocess communication is involved and therefore unless
there is intense string manipulation required, such as case insensitive
path matching, I'm not certain I would agree. Of course I don't know
the details.

> fell swoop then it would make this problem go away :)

CJK in UTF-8 requires 3 bytes per character. Expect resistance.

You guys make it sound like there's something "wrong" with non-UTF-8
locales and that everything else is "legacy". There are many environments
that do not use UTF-8 (or Unicode for that matter) for legitimate
reasons. That might limit their interoperability but these days that's
usually a consciencous decision.

> The C bindings of the *abstract* interfaces would be specified in terms

So you do advocate requiring UTF-8? Mmm, for a minute there I thought we
were in the same camp :-/

First, I want to reiterate that for encoding of network protocol messages
I *am* advocating UTF-8 wherever possible. It's the library interface
that I think should use the C locale encoding. So not all of the code
needs to worry about locale dependent encodings. Ultimately all I'm
really talking about is adding some mbrtowc(3) loops here and there. The
implications for the code are not that bad for GSSAPI. In fact I wouldn't
be surprised if the GSSAPI layer itself works fine as is, and that it's
the mechanisms that need the attention. In that case the library backing
the mechanism should be fixed or wrapped.

In a sentence, I advocate requiring UTF-8 if there's interprocess
messaging or if there's string manipulation that is otherwise difficult
to perform with plain mbrtowc(3) loops. Otherwise, respect my locale!

But then again, I'm a newbie here. Maybe I'm missing something important.

Mike

---

From: Michael B Allen <[email-masked]>
Date: Tue, 9 May 2006 23:23:24 -0400

On Wed, 10 May 2006 12:37:12 +1000

> UNICODE_STRING is UCS-2LE, no?

You could be right about that. I'm honestly not certain. I know it used
to be UCS-2LE but I think MS has started to use UTF-16LE all over so I
refuse to believe that somehow they've managed to keep their peas and
carrots separate.

Mike

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Wed, 10 May 2006 00:54:43 -0400

On Tuesday, May 09, 2006 04:55:37 PM -0500 Nicolas Williams 
<[email-masked]> wrote:

> I don't want us discussing Kerberos V I18N

Nor do I.  Let's save that discussion for the appropriate time and place, 
and for _after_ reading draft-ietf-krb-wg-rfc1510ter (which, unfortunately, 
seems to have expired.  I guess I'll be prodding the editor about that).

-- Jeff, krb-wg chair

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 10 May 2006 00:31:34 -0500

> path matching, I'm not certain I would agree. Of course I don't know

That was an aside, not to be taken as justification for anything in
regards to the GSS-API.

> CJK in UTF-8 requires 3 bytes per character. Expect resistance.

I think there will be no resistance, but I'm not an I18N expert.

> were in the same camp :-/

No, I advocate requiring that the API support the explicit use of UTF-8
and that optionally it support -just-use-current-locale's-ces.

Nico
--

---

From: Martin Rex <[email-masked]>
Date: Wed, 10 May 2006 17:35:05 +0200 (MET DST)

> path matching, I'm not certain I would agree. Of course I don't know

Well, we and a lot of others use GSS-API to protect the communication
between two distinct entities, a communication initiator and a communication
acceptor.  These two might use IPC for communication, but it is going
to be a "network communication" quite often.  The authentication service
provided by the GSS-API mechanism usually conveys the identity of the
initiator (taken from the credentials of the initiator) to the
acceptor somewhere within a context level token that is emitted
by gss_init_sec_context() of the initiators gssapi implementation
and to be consumed by gss_accept_sec_context() of the acceptors
gssapi implementation.  When mutual authentication is requested or
provided, this also works in the other direction (just that
the initiator has to supply the expected identity of the acceptor
to the initial call of gss_init_sec_context() and the gssapi
mechanism performs the comparison/matching (authentication) against
the conveyed identity.

So in the general case, as far as the names of the identies of
initator and acceptor involved, there are two, potentially differing
locales involved--maybe even a third locale of a trusted third party
like a KDC which participates the authentication procedure.

> usually a consciencous decision.

Not really.  But there's a requirement in the IETF to address I18N
in all standards, and also a requirement for a mandatory-to-implement
variant -- and most standards have settled on UTF-8 for that purpose.

As I mentioned before, there are a lot of SPKM- and proprietary SPKM-like
gssapi mechanisms out there using an X.509 PKI for authentication
and X.509 happens to use almost exclusively the encodings from
one of 7-bit ASCII, ISO-Latin1 (abused T61string), and UTF8String.
Microsoft has chosen to use UTF-8 for Kerberos principal names.

There is little chance that we could sneak a GSS-API v3 past the IESG
without addressing I18N and without exactly one mandatory-to-implement
encoding that will allow I18N -- which more or less boils down to
a Unicode encoding.  IMHO, UTF-8 seems to be the most reasonable
choice here.

-Martin

---

From: Michael B Allen <[email-masked]>
Date: Wed, 10 May 2006 12:38:39 -0400

On Wed, 10 May 2006 17:35:05 +0200 (MET DST)

> a Unicode encoding.  IMHO, UTF-8 seems to be the most reasonable

I completely agree that you want to be fully
internationalized. Obviously. It's a great mandate. I'm just trying to
explain as a programmer who has written i18n consciencous code how to
do that politely.

Again, the GSSAPI layer itself is fine right now as is [1]. It's
the *mechs* that need attention. If the library backing the mech is
providing Latin1 strings with umlauts and the user is in the UTF-8
locale and the mech is simply using strcpy, then that's just a bug in
the mechanism. Plain and simple BUG. Those bugs can be fixed by either
"fixing" the library backing the mech (hint: MS has done us a favor
by using UTF-8) or wrapping it's functions to convert to the locale
dependent encoding.

I say, convert at the mech boundy once. If you mandate UTF-8 regardless
of locale then you still have to convert at the mech boundry AND then
the program has to convert again at the GSSAPI user API.

If you want to push the i18n boundry a little, mandate that mech boundries
properly convert to the locale dependent encoding.

For example, let's say you have some SPKM mech that's doing Latin1 but
the user is using a ISO-8859-4 locale [2]. By the new mandate the mech
is required to convert all Latin1 strings to ISO-8859-4 (using iconv(3)
presumably). Done. You might get EILSEQ errors or '?' marks in your
string but that should just indicate to the admin/user that they should
upgrade to the UTF-8 locale or dodge those characters.

Mike

[1] If not all it needs are probably a few mbrtowc loops.
[2] I don't think Latin1 and ISO-8859-4 are fully compatible but play
along and assume that the programmer/admin/user knew what they were
doing or had a really good reason for chosing that locale.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 10 May 2006 11:58:55 -0500

> Microsoft has chosen to use UTF-8 for Kerberos principal names.

Is PKI properly internationalized?

Kerberos V isn't, and what MS does is not correct.  (Again, wrong forum
for that discussion.)

> a Unicode encoding.  IMHO, UTF-8 seems to be the most reasonable

I can imagine a programming language where UTF-8 would not be the right
choice.

This is an API, not a protocol (protocol I18N considerations belong in
the mechs), so mandates to use UTF-8, stringprep, etc. don't apply, only
a mandate to ensure I18N and L10N should apply.

Now, I do agree that UTF-8 is the way of the future, but until we
deprecate non-UTF-8 locales I'll be reluctant to have a UTF-8-only API,
and a Latin-1-only alternative is NOT sufficient either.

It's pretty simple:

 - we agree on requiring support only for Unicode in the API, with
   appropriate language-specific bindings (e.g., UTF-8 for C bindings),

but:

 - I also insist on an optional locale's-current-codeset option because
   on POSIX/POSIX-like platforms that is the most comfortable option for
   application developers.

Nico
--

---

From: Jeffrey Altman <[email-masked]>
Date: Wed, 10 May 2006 13:08:00 -0400

Michael:

GSS-API v2 upd 1 requires that all strings used for input and output
be ISO-Latin-1.  This is the requirement today.  It is a bug for
any mechanism to be returning anything other than ISO-Latin-1.  It is
also a bug for any application to provide as input anything other than
ISO-Latin-1.

As it turns out, most applications ignore this requirement as do most
mechanisms.  That does not change what the requirement as stated by
the RFC is.  These implementations "just-use-8" today and hope for the
best.

For GSS-APIv3 we can choose to do something different.  What I am
wondering is whether we need to define both a GSS-API and a GSS-SPI.
The layer that implements both the GSS-API and the GSS-SPI would be
responsible for handling translation from locale to UTF8 and mechanisms
(which would only implement the GSS-SPI) would be required to handle
UTF8 on their boundary.

In this model I would prefer the GSS-API to have separate locale and
utf8 variants.  The locale variants would be allowed to return an error
if locale to utf8 translation services are not available.

I do not believe it is necessary for the GSS-API to support arbitrary
character-set as input and I would prefer to avoid creating another
character-set registry.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 10 May 2006 12:11:03 -0500

> character-set as input and I would prefer to avoid creating another

No new registry would be required in my proposal.

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Wed, 10 May 2006 13:56:40 -0400

On Wednesday, May 10, 2006 12:38:39 PM -0400 Michael B Allen 
<[email-masked]> wrote:

> the mechanism. Plain and simple BUG.\

First of all, there generall is no "GSSAPI layer".  The GSS-API is 
implemented directly by mechanisms.  This provides for fun when you want to 
hve a multi-mechanism glue layer, but that's all behind the API and thus an 
implementation matter (i.e. out of scope for the GSS _API_ specification, 
which is what we are discussing on this list).

Second, it's best not to think in terms of "the library backing the mech", 
because that introduces assumptions about what's behind the API which may 
not be in line with actual implementations.  Sure, there are mechanisms 
that make use of libraries, but the GSS-API mechanism layer generally does 
a non-trivial amount of work, and there can be mechanisms where there is no 
such library.  A better way to say this would be "If the mechanism uses 
Latin-1 on the wire and just returns wire strings to the app without 
conversion, and the user is in a UTF-8 locale, then that is a bug".

Third, the behavior you describe is actually _not_ a bug, because the 
GSSAPIv2 spec explicitly requires the use of Latin-1.  What would be a bug 
would be to fail to convert strings which are _not_ Latin-1 on the wire.

Unfortunately, many existing implementations ignore this part of the spec, 
and instead either use whatever the current locale is for the API, 
converting between _that_ and whatever is used on the wire, or worse, do 
nothing at all, depending on the application to use the right encoding (and 
with some mechanisms, this works, as long as both ends behave that way and 
happen to use the same charset).

What Martin is proposing, for the upcoming GSSAPIv3, is a new set of API's 
which are defined to use UTF-8 rather than Latin-1.  The hope is that this 
time, both mechanisms and applications would actually stick to the spec, 
converting as needed between UTF-8 and whatever they actually want to use.

What Nico is proposing, also for the upcoming GSSAPIv3, is a different new 
set of API's, which provide a way for the caller to explicitly indicate 
what charset he is using.  It would then be up to the mechanism to convert 
between _that_ and whatever is needed on the wire.  He further proposes 
initially defining two values for the charset parameter, with one meaning 
"utf-8", and the other meaning "whatever the current locale is".

The advantages of either Martin's proposal or Nico's UTF-8 mode are the 
same: mechanisms need only deal with UTF-8, those mechanisms that use UTF-8 
on the wire need not do charset conversion at all (though they will still 
need to do whatever stringprep or other transforms are required to 
implement the mechanism), and applications working with data in a known 
charset can get a consistent behavior from the GSS-API which does not 
depend on the environment in which the application is run.

The advantages of Nico's current-locale mode is that applications which are 
working with data in a locale-dependent charset, including any data which 
is entered by or displayed to the user, can use a locale-dependent API 
which "just works" when presented with such data, and which plays nicely 
with the wide-variety of locale-dependent API's likely to exist on a 
locale-aware system(*).

Now, Martin has objected that Nico's approach would require mechanisms to 
perform converstion (which it would, as would the UTF-8 approach, for 
mechanisms not using UTF-8 on the wire).  This objection mostly seems to be 
based on the size of the code, and more significantly, the tables, required 
to perform such conversions.  That would be a valid concern if every 
mechanism implementation had to contain a copy of that code and those 
tables.  But the practical reality is that both applications and mechanisms 
are going to perform conversions using a system API provided for that 
purpose, so no matter what you do, the code and tables don't appear in 
either the application or mechanism implementation, but _do_ appear in the 
system.  So, I do not find this argument compelling.

Now, there are going to be issues with systems that provide no such 
interface.  On those systems, there's going to be trouble no matter what 
you do, unless both the application and mechanism work exclusively in 
UTF-8.  For well-desined mechanisms, you're still going to have a problem, 
because even a mechanism which never has to see a charset other than UTF-8 
is still going to have to do string preparation, generally including 
Unicode string normalization, which requires its own tables of nontrivial 
size.  And a platform that doesn't provide facilities for converting 
between UTF-8 and the locale-dependent charsets for whatever locales it 
supports is not going to provide facilities for Unicode normalization, 
either.

Note that on a system where only Unicode is supported, one can simply 
declare the "locale-dependent charset" to always be UTF-8, and then both of 
Nico's proposed modes act identically.

Myself, I very much like the idea that, in an environment which has locales 
and provides charset-conversion utilities, an application should be able to 
pass user-provided data expressed in the charset of the current locale 
directly to the API without having to deal with conversion.  As Nico points 
out, there are or should be way more applications than mechanisms, and we 
should make life convenient for application programmers.

I'm not saying that _every_ application should be forced to behave this 
way; each application should be able to choose the interface variant whcih 
works best for it.  And I have to admit, I also like the idea of having an 
interface that is extensible in this dimension.  Someday we're going to get 
conquered by technologically-advanced aliens who don't use Unicode, and 
we're going to have to support whatever charset they do use.

(*) Yes, Martin, we know that in 1993 there weren't so many such API's and 
the world was a pretty sad place for people who wanted to actually use 
locales.  However, the situation has changed, things work well today, and 
when designing a modern API it is reasonable to assume, to a certain 
extent, that the rest of the system is similarly modern.  It is not 
reasonable to constrain the design of a new API based on the limitations of 
13-year-old systems which are beyond their end-of-life and will never 
support the new API.

-- Jeff

---

From: Michael B Allen <[email-masked]>
Date: Wed, 10 May 2006 14:31:09 -0400

On Wed, 10 May 2006 13:08:00 -0400

> (which would only implement the GSS-SPI) would be required to handle

I see. So you're more concerned with mechglue mechanism
interoperability? Is SPI "service provider interface"?

Mike

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Wed, 10 May 2006 14:36:25 -0400

>    on POSIX/POSIX-like platforms that is the most comfortable option for

And after reading this discussion the past several days, I've decided that 
I support using an approach which provides for that additional mode.

The real question is, what do you do when an application tries to use a 
mode that is not supported?  And more importantly, how do you prevent that 
from happening too often?

Particularly, for the situation Nico is talking about, it would be nice if 
applications built in certain environments could assume the current-locale 
mode will work.  Ideally, it would be good if applications written for, 
say, POSIX, could simply assume the mode will work and thus not have to 
_have_ any code to deal with charset conversion.  To this end, I'll suggest 
a few requirements:

For the purposes of this proposal...

The phrase "locale-aware" refers to an environment in which there is some 
concept of "locale" or of a current or default character set which can be 
configured either system-wide or separately for each user, session, 
process, or whatever, _and_ in which _either_ it is possible to tell at 
runtime what charset is in use, _or_ there are standard system-provided 
interfaces which support conversion between the charset in use and a fixed 
or caller-specified charset.

The phrase "fixed-charset" refers to an environment in which only a single 
charset is used, and in which it is reasonable to expect that charset to be 
known at the time code is built for that environment.

It is possible to have environments which are neither locale-aware nor 
fixed-charset.

For example, a system whose interfaces all use UTF-8 exclusively is 
fixed-charset.  But a system which uses exactly one charset based on which 
of several mutually-exclusive packages is installed is _not_ fixed-charset, 
because the same program can end up being expected to use different 
charsets depending on the environment in which it is run.  Such an 
environment may be "locale-aware", if it is possible to tell at runtime 
which charset is in use, or if the system provides conversion routines 
between that cnarset and, say, UTF-8.

"Unicode mode" refers to a mode of operation in which the GSS-API uses 
Unicode strings, in some encoding which may be dependent on the language 
environment (for example, I'd expect C-bindings to use UTF-8, but in some 
other language it might be more natural to use UCS2 or UTF-16).

"Locale mode" refers to a mode of operation in which the strings used by 
GSS-API are expressed in a character set and encoding which depends on the 
environment.

(1) All GSS-API implementations MUST support the Unicode mode.

(2) In a fixed-charset environment which uses UTF-8 as the system charset,
    Locale mode MUST be supported and MUST use UTF-8.

(3) In a fixed-charset environment which uses a system charset other
    than UTF-8, but which provides a mechanism for converting between
    the system charset and UTF-8, Locale mode MUST be supported and
    MUST use the system charset.

(3) In other fixed-charset enviroments, Locale mode SHOULD be supported;
    if Locale mode is supported, it MUST use the system charset.

(4) In a locale-aware environment which provides facilities that can
    be used to convert between the current charset and UTF-8, Locale
    mode MUST be supported and MUST use the current charset.  Note that
    such facilities could consist of conversion functions which use the
    current charset implicitly, or of a generic conversion interface
    such as iconv() coupled, with a way to find the current charset.

So basically, you always have to support Unicode, and you also have to 
support Locale mode if the system provides the facilities needed for 
mechanisms to do so in a reasonable way.  This allows applications written 
for environments where such facilities are guaranteed to be available to 
make the assumption that Locale mode will be supported, without placing 
undue burden on mechanism implementors in environments where such 
facilities are not provided by the system.

It would probably be nice to be able to tell at compile time whether the 
Locale mode will be supported.  In C bindings, this might be done with a 
preprocessor symbol indicating support for Locale mode.  I consider this a 
reasonable approach because we are standardizing an API, not an ABI; 
implementors who want to standardize an ABI and provide multiple mechanisms 
supporting that ABI may have to introduce further constraints.

-- Jeff

---

From: Jeffrey Altman <[email-masked]>
Date: Wed, 10 May 2006 15:22:24 -0400

> interoperability? Is SPI "service provider interface"?

SPI is "service provider interface".

Deciding to standardize a API / SPI model would require that the
IETF agree to standardize an ABI.  This would be a huge departure from
prior efforts and it would certainly go badly.

As Jeffrey Hutzelman reminds us, the GSS-API is an API and the API is
frequently implemented by the mechanisms themselves.  I made the above
comment in response to one of Martin's concerns which was the quantity
of code that mechanisms implementing GSS-API would be required to
implement.  I too am concerned about arbitrary levels of character-set
complexity being required of mechanisms.  Not all operating systems
on which GSS-API is implemented have a concept of locale.  Others as
Martin has pointed out do not have control over the character-set used
by the end user.  Martin's examples of 1993 HP-UX and Microsoft Windows
have been disregarded by this audience but user's entering data into
a web form which is later passed into GSS-API is a modern day example
of the same problem.  The user has a locale which is some Windows ANSI
code page, the web server is running under ISO Latin-1.  Now if the
user's input is provided to gss_foo() and it is expected to be
translated by GSS-API based upon the web server's locale, the
translation is going to be wrong.  The web server must take
responsibility for performing the translation from the user's input
locale to the web server's locale and therefore might as well convert
to UTF8.

I believe Nico's latest proposal is sufficient to reduce the burden
required of mechanism implementers while still providing the tools
necessary for applications that can benefit from a set of locale based
functions.

Jeffrey Altman

---

From: Michael B Allen <[email-masked]>
Date: Wed, 10 May 2006 15:28:42 -0400

Genereally I'm in your camp Jeff. Thanks for the summary.

If you guys want a UTF-8 interface that's fine as long as I (and I think
I speak for most if not all other developers) can ignore it. That's
the only point I feel strongly about. As for using UTF-8 internally,
that sounds like an implementation detail to me.

Mike

---

From: Michael B Allen <[email-masked]>
Date: Wed, 10 May 2006 16:15:47 -0400

On Wed, 10 May 2006 15:22:24 -0400

> on which GSS-API is implemented have a concept of locale.  Others as

First, I regret saying "locale" because it's not like we're talking
about formatting monitary units. Each environment has a premier string
type and provides facilities for converted between that type and other
encodings. If it does not have such a facility then that system isn't
really a candidate for gssapi support in the first place.

Second, it seems to me that using UTF-8 internally is an implementation
detail and therefore I don't feel stongly about it. However, just in
case this somehow percolates up into developer space I just want to say
that I think it is *more* complex because in the non-UTF-8 case (which is
really all we're worried about) you're converting twice instead of once
(e.g. you could end up doing Latin1 <-> UTF-8 <-> ISO-8859-4).

I do acknowledge that there is an advantage to using UTF-8 for complex
string processing. If you're searching for an '@' sign in a string, with
UTF-8 you can use traditional strchr or *ch == '@' style code. Under
my proprosal the C implementation *might* be complicated with code to
extract each character with a routine like mbrtowc(3) and then compare
with something like wch == L'@'. But virtually 99% of that type of
processing occurs within the mechanism implementations which again seems
to me like an implementation specific detail. And thus mandating UTF-8
at mech boundries doesn't make sense to me.

> .... but user's entering data into

This is such an extreme example it's a red herring. Web servers provide
facilities for dealing with the different locales of clients. That is
clearly the responsibility of the web server and not of libraries used
by it.

Mike