Subject: Charter: internationalization is out of scope

From: Sam Hartman <[email-masked]>
Date: Wed, 26 Apr 2006 17:34:20 -0400

Jeff Altman asked me a question about the charter and
internationalization.  Looking at your charter, internationalization
is currently out of scope.

I'd be happy to take a proposal to extend the charter to include
internationalization to the IESG; I do not anticipate approvalbeing
hard.

Note that if the WG does not work on internationalization, you will
likely have trouble getting API drafts approved because they will not
meet IETF internationalization requirements.

_______________________________________________
Kitten mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/kitten

---

From: Ken Raeburn <[email-masked]>
Date: Wed, 26 Apr 2006 17:42:50 -0400

> meet IETF internationalization requirements.

If it's an IETF requirement, doesn't that implicitly put it in scope  
for any working group with a deliverable subject to that requirement,  
unless it's explicitly described as out of scope?

Ken

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 26 Apr 2006 17:18:54 -0500

Simpler:

I18N is out of scope, but GSS-APIv2u1 clarifications are in scope, which
means that...

...we can clarify the locale/codeset issue while leaving
Unicode-specific versions of GSS_Display_name(), GSS_Display_status()
(which would also need localization, i.e., langtags) and
GSS_Import_name() out of scope.

The I18N/L10N clarification is trivial:

 - for name types other than GSS_C_NT_EXPORTED_NAME the
   input_name_string is an octet string conforming to the codeset of the
   current locale;

 - the name_string and status_string_set outputs of GSS_Display_name()
   and GSS_Display_status(), respectively, are octet strings conforming
   to the codeset of the current locale;

 - the status_string_set output GSS_Display_status() is to be localized
   to the language of the current locale.

We might have to say something about locales, but not much more than
this.

Nico
--

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Wed, 26 Apr 2006 18:29:48 -0400

> We might have to say something about locales, but not much more than

Well, I don't think we can just say "locale", but we can put in some 
language about user preferences and/or operating-system settings that means 
locale-or-whatever-equivalent.

It's not entirely clear to me that we're allowed to make this sort of 
change in GSSAPIv2u1 at all, since it's technically incompatible with the 
existing spec.  But if the reality is that that's what all the 
implementations are actually doing, then it seems like fixing the spec to 
reflect reality is the right thing.

-- Jeff

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 26 Apr 2006 17:35:56 -0500

> locale-or-whatever-equivalent.

Why not?  RFC3536 talks about locales...

> reflect reality is the right thing.

No, the GSS-APIv2u1 is *broken* in this respect.  Period.

Noone implements GSS-API mechglue that enforces the use of Latin-1 and
noone implements GSS-API applications that convert between whatever
codeset is used above to/from Latin-1 in the GSS-API calls.

We'd be correcting the spec, not changing it.

Nico
--

---

From: Martin Rex <[email-masked]>
Date: Thu, 27 Apr 2006 16:03:47 +0200 (MET DST)

> No, the GSS-APIv2u1 is *broken* in this respect.  Period.

Nope, not really.  GSS-APIv2u1 high level doesn't touch the issue,
because it is language dependent.  JAVA natively uses Unicode,
so there the issue is not visible at the language/API level,
but with Kerberos and rfc-1510 interoperability, it does
still exist at lower levels!

The C-Bindings document defines printable strings to use
iso-latin-1 single-byte codepage, NOT locale.

> codeset is used above to/from Latin-1 in the GSS-API calls.

I have shipped code that relies on the strings to be iso-latin-1
(because my code sends the printable name through the network,
 and I did have EBCDIC and ASCII machines)

and most code that I know (including MIT Kerberos) uses the
just-send-8, which is DIFFERENT from using the locale --
because depending on the situation, the locale of the KDC
or the locale of the client will determine the codepage
of what the acceptor will get returned from gss_display_name()
for the src_name of the initiator.  And the locale of the server
will likely be entirely ignored! -- which clearly indicates that
a notion that current gssapi implementations are using the locale
may often be ill-conceived.

I don't know whether OpenSSL is doing any codepage conversions
(based on locale).  The abuse of Microsoft and Netscape by
using ISO-latin-1 for subject distinguished names in X.509 certs
while tagging them as T61 seems to be an indication that
"locale" (and code-page-conversions) is rarely used and
that "just-send-8"/"just-use-8" is the most common case.

> We'd be correcting the spec, not changing it.

It would be definitely a backward incompatible change,
and therefore we can NOT invalidate the use of ISO-Latin-1.

The most common implementation (including MIT Kerberos) is
NOT to use the locale, but instead to use "just-send-8",
which is substantially different--because the use of a locale
will require you to perform conversions if the user changes
the locale -- which most code simply doesn't do today --
instead of continuing to "just-send-8".  GSS-API normally
involves two differing communicating parties (or more),
and these may have different settings and desires/requirements
for their locale.

And what do you do with characters of a name that do not exist
in the user-configured locale?  Do you abort with an error
(i.e. fail safe but uncomfortably for the user) or do you
use a dummy character indicating a failed character conversion?

The only thing that we might do about the existing spec is

 (a) document behaviour of existing implementations

 (b) recommend a particular behaviour

 (c) suggest how to correctly implement support for UTF-8
     throughout the entire infrastructure

-Martin

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 09:16:37 -0500

> iso-latin-1 single-byte codepage, NOT locale.

That's *broken*.

---

From: Jeffrey Altman <[email-masked]>
Date: Thu, 27 Apr 2006 10:32:52 -0400

>> iso-latin-1 single-byte codepage, NOT locale.

Nico:

I have to disagree with you on this one.

Its no more broken to specify the output as Latin-1 then it
is to specify the output as UTF-8.  The specification is stating
what the output is in a consistent manner.  It would be up to
the calling application to convert that output to a locale
dependent representation.

You can argue that by specifying Latin-1 that the specification is
being overly restrictive of the characters that can be represented.
However, Latin-1 is broader than US-ASCII and that is all that
Kerberos can utilize in a interoperable manner.

Jeffrey Altman

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 27 Apr 2006 11:06:55 -0400

> Kerberos can utilize in a interoperable manner.

(1) Kerberos is not the only GSS-API mechanism.
(2) Nico's point is that the spec is broken because what it describes
    is not what any of the implementations actually _do_.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 10:23:27 -0500

> I have to disagree with you on this one.

Show me one of each of these:

 - applications that convert input_name from the current locale's
   codeset to Latin-1 prior to calling GSS_Import_name() for any name
   type other than GSS_C_NT_EXPORT_NAME...

   ...and which convert the output of GSS_Display_name() and
   GSS_Display_status() from Latin-1 to the current locale's  codeset.

 - mechanisms that assume that the input_name in GSS_Import_name() is in
   Latin-1...

   ...and whose GSS_Display_name() and GSS_Display_status() output is in
   Latin-1

 - mechglues that reject non-Latin-1 input_name values  to
   GSS_Import_name()

 - mechglues that react in any way to mechanisms that output non-Latin-1
   from GSS_Display_name() or GSS_Display_status() 

There are none that I can think of.

Neither Heimdal, nor MIT, nor GNU, nor Solaris implementations of
GSS-API framework or mechanisms, nor any of the apps that use the
GSS-API that I can think of (SSHv2 [e.g., OpenSSH], SASL [e.g., Cyrus
SASL], NFS w/ RPCSEC_GSS [e.g., CITI] , and so on), qualify.

ALL just-use-8, which always amounts to: just-use-the-current-locale's-
codeset.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 10:24:56 -0500

>    is not what any of the implementations actually _do_.

Exactly.

The spec says one thing and noone does it that way.  And everyone does
it the way that I propose we change the spec.

---

From: Jeffrey Altman <[email-masked]>
Date: Thu, 27 Apr 2006 11:28:43 -0400

> (1) Kerberos is not the only GSS-API mechanism.

Obviously.  However it is an example.  Is RFC 4120 broken because
Heimdal and MIT just-send-8 and Microsoft sends Unicode?

>    is not what any of the implementations actually _do_.

The fact that the implementors that we know about do not output
Latin-1 does not make the spec broken.  It means that the implementors
didn't read it correctly when they wrote their code OR intentionally
chose to do something in violation of the spec.

I suspect that just-send-8 was used primarily because in the
environments in which the code was written US-ASCII was the predominant
character-set and at least for the Kerberos implementors the rest of
the libraries were already just-send-8 and were used in environments
with default locales of either ASCII or Latin-1.

Changing the spec to read "just-send-8" does not make GSSAPIv2 any
more interoperable then it is today.  If anything it makes it worse.

Specifying new variations of the existing functions that require UTF-8
would not be more interoperable but would support a broader user base.
We can't go back and require UTF-8 for GSSAPIv2.  That would be an
incompatible change that would not provide any additional degree of
interoperability because applications using the library would not be
able to identify what character-set to expect.

Jeffrey Altman

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 27 Apr 2006 11:27:48 -0400

>  - mechglues that reject non-Latin-1 input_name values  to

There can't be such a thing.  All octet strings are valid latin-1.

>    from GSS_Display_name() or GSS_Display_status()

Same deal.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 10:31:07 -0500

> There can't be such a thing.  All octet strings are valid latin-1.

I don't have my references handy, but I thought there were some codes in
the ISO8859-1 codeset that were not allocated.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 10:32:19 -0500

> chose to do something in violation of the spec.

Noone in their right mind will implement what the spec says in this
case.  I dare MIT to do it ;)  -- I know MIT won't, because MIT isn't
insane.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 10:35:32 -0500

> Show me one of each of these:

Or at least one of those things where you think the implementors could
be convinced to make them comply with this Latin-1 requirement.

There are none that I can think of, either way.

Martin's mechanism and application implementations might, for all I know
comply, but it seems unlikely to me.  Martin?

Nico
--

---

From: Jeffrey Altman <[email-masked]>
Date: Thu, 27 Apr 2006 11:46:19 -0400

> the ISO8859-1 codeset that were not allocated.

ISO-8859 specifies the printable characters outside the byte values
reserved for the C0 and C1 controls.

All octet strings are a valid combination of Latin-1 plus a C0 and C1
control set.  The IBM/Microsoft code pages as opposed to ISO 8859
character-sets do not designate a C1 range.  Therefore, octet strings
that are value code pages may collide with the C1 control range.

---

From: Jeffrey Altman <[email-masked]>
Date: Thu, 27 Apr 2006 11:46:59 -0400

> ALL just-use-8, which always amounts to: just-use-the-current-locale's-

What you are proposing is a protocol that only works when the
peers are either lucky enough to use the same character-set by
chance or have negotiated the character-set out of band.

There is a reason we didn't change 4120 to say just-send-8.  Its
because we care about interoperability in our specifications.
If two people independently follow the spec and produce an
implementation, they should be able to communicate with one
another.  Therefore, 4120 clarifies that the allowable character-set
is US-ASCII only.  It notes that there is not a single implementation
that we are aware of today that enforces this restriction and that
most just-send-8.  As such applications that use Kerberos use
characters outside of US-ASCII at their own peril.

The only clarification that makes sense to say for GSSAPIv2 is
a note indication that many if not all of the known implementations
just-send-8 in violation of the Latin-1 requirement.  As such it
is possible that names received may not in fact be Latin-1.

As Martin pointed out, the Java Bindings using UTF8.  That means
that we already have a character-set collision problem if one peer
is Java and the other is C.  Changing the C binding to state
"just-send-8" does not make the problem any better.

Jeffrey Altman

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 27 Apr 2006 11:52:42 -0400

> chance or have negotiated the character-set out of band.

Not at all, because we're not talking about a protocol or what happens on 
the wire.  We're talking about an API, and the caller and callee are 
assumed to have the same locale by virtue of being, at least conceptually, 
in the same process.

We're not talking about "just-send-8"; what you _send_ is up to the 
mechanism.  We're talking about "just-use-8", in an API.

-- Jeff

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 10:59:31 -0500

> What you are proposing is a protocol that only works when the
                              ^^^^^^^^

NO.  API.

---

From: Martin Rex <[email-masked]>
Date: Thu, 27 Apr 2006 18:20:50 +0200 (MET DST)

> > ALL just-use-8, which always amounts to: just-use-the-current-locale's-

I disagree, Nico.

If I install a current Linux distribution and configure the locale
for DE_de, then I get to see manpages and output of certain commands
(e.g. man pages) in german.

Code that detects the current locale and *CONVERTS* persistent or
remotely-received accordingly is locale-aware.

Kerberos, OpenSSL and a lot of other code completely ignores the
configured locale and does not perform any codepage conversions,
mostly because the printable data to which a conversion would
have to be applied is neither tagged nor sufficiently standardized
so that the code would have a chance to know where from/to a
convert to.

In absence of codepage conversions in the code, one should be very careful
to describe behaviour as "using the current locale", because it
is plain wrong, and you may realize that as soon as you *change*
the locale, because some of the results will not follow and
you may loose interoperability or cause security problems.

Documenting it as "just-use-8"/"just-send-8" means that if the subset
of characters that is used by all participants happens to use the
same codepoints, then you will get interoperability independent of
individual locales.  If all participants happen to use the same
codepage/locale, then you appear to have full interoperability
on all of the codepoints/characters -- but as soon as someone
wants to use a different codepage (including the use of any
variant of Unicode, UTF8, UCS2, UCS4,...), then you loose if
the shared/transfered printable information is neither tagged
(meta-information) nor standardized to have a particular
"transfer-encoding".

> is possible that names received may not in fact be Latin-1.

I fully agree.

> "just-send-8" does not make the problem any better.

Are you sure about UTF8 -- I thought that Java uses "wide characters"
(UCS2? USC4?), which means that a JAVA-based gssapi for Kerberos
will have to, at some point, convert from wide characters to
a single byte code page as the "transfer encoding" of principal
names in the Kerberos protocol -- and in case that UTF8 isn't
available, you will have to decide for a particular single-byte
codepage.  Choosing the current single-byte locale as the
transfer encoding may be easy, but it may certainly interoperability
problems even with other Java peers (or the KDC) if they happen
to use a different locale when the involved principal names 
are not from 7-bit ASCII...

-Martin

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 11:31:52 -0500

> (e.g. man pages) in german.

Because the man command knows how to find them -- it had to be locale
aware.

> remotely-received accordingly is locale-aware.

Yes.

GSS-API *applications* *don't* have to be.

GSS-API *mechanisms* *do* have to be.

But the spec (RFC2744) gets it backwards.

Nico
--

---

From: Martin Rex <[email-masked]>
Date: Thu, 27 Apr 2006 18:59:42 +0200 (MET DST)

> mechanism.  We're talking about "just-use-8", in an API.

The fact that the src_name which pops out of gss_accept_sec_context()
has normally not been supplied as a printable to the local GSS-API
is a vital part of the service supplied by this API.  The printable
representation created for this src_name via gss_display_name()
has been created by either by the initiator or a third party
(may be a KDC, may be a CA issuing an X.509 cert) with some
single-byte codepage that may differ from the locale/codepage
of the server/acceptor.

"Legacy" Kerberos installations (MIT,Heimdal) which have been
using non-ASCII characters in usernames (and ISO-Latin-1 locales
througout) may easily return invalid UTF-8 encodings when
you switch the locale to UTF-8, because they are locale-ignorant
and some sequences of 8-bit ISO-Latin-1 are invalid UTF-8 encodings.

There is another I18N issue: the "hostbased service names"
after the invention of Punycode.  Currently, this means that
the application has to puny-encode international host&domainnames
before passing it to gss_import_name().  However this requires
the application to parse and process names which historically
may have been opaque to the application.

Is there any document/proposal how to map between international
host&domainnames to Kerberos (hostbased) service principals?
Microsoft has been using UTF8 for their principal names,
so they might be mislead to retain UTF8 hostnames within
Kerberos principal names -- however this creates a problem
as to define who and where (API-level: kerb-mech or app) will have
to convert international host&domainnames into punycode and which
one is the "canonical" encoding at the protocol level. 

-Martin

---

From: Sam Hartman <[email-masked]>
Date: Thu, 27 Apr 2006 13:21:37 -0400

>>>>> "Martin" == Martin Rex <[email-masked]> writes:

    Martin> There is another I18N issue: the "hostbased service names"
    Martin> after the invention of Punycode.  Currently, this means
    Martin> that the application has to puny-encode international
    Martin> host&domainnames before passing it to gss_import_name().
    Martin> However this requires the application to parse and process
    Martin> names which historically may have been opaque to the
    Martin> application.

    Martin> Is there any document/proposal how to map between
    Martin> international host&domainnames to Kerberos (hostbased)
    Martin> service principals?  Microsoft has been using UTF8 for
    Martin> their principal names, so they might be mislead to retain
    Martin> UTF8 hostnames within Kerberos principal names -- however
    Martin> this creates a problem as to define who and where
    Martin> (API-level: kerb-mech or app) will have to convert
    Martin> international host&domainnames into punycode and which one
    Martin> is the "canonical" encoding at the protocol level.

There has been a lot of discussion of this issue.  I don't remember
what the final resolution was; we went back and forth a lot.

--Sam

---

From: Sam Hartman <[email-masked]>
Date: Thu, 27 Apr 2006 13:52:40 -0400

>>>>> "Ken" == Ken Raeburn <[email-masked]> writes:

    Ken> On Apr 26, 2006, at 17:34, Sam Hartman wrote:
    >> Note that if the WG does not work on internationalization, you
    >> will likely have trouble getting API drafts approved because
    >> they will not meet IETF internationalization requirements.

    Ken> If it's an IETF requirement, doesn't that implicitly put it
    Ken> in scope for any working group with a deliverable subject to
    Ken> that requirement, unless it's explicitly described as out of
    Ken> scope?

Not quite.  The IETF requirement will require anything you define now
to be properly internationalized.  However you cannot go back and
change things in the spec unrelated to what you are working on.  So,
you might end up with something silly like name attributes being UTF8
but names being ISO 8859-1.

---

From: Sam Hartman <[email-masked]>
Date: Thu, 27 Apr 2006 13:53:56 -0400

>>>>> "Nicolas" == Nicolas Williams <[email-masked]> writes:

    Nicolas> Simpler: I18N is out of scope, but GSS-APIv2u1
    Nicolas> clarifications are in scope, which means that...

    Nicolas> ...we can clarify the locale/codeset issue while leaving
    Nicolas> Unicode-specific versions of GSS_Display_name(),
    Nicolas> GSS_Display_status() (which would also need localization,
    Nicolas> i.e., langtags) and GSS_Import_name() out of scope.

Clarification does not mean changing the spec.

You cannot change the spec from 8859-1 to current locale and call that
a clarification.  You can add an interoperability note saying that
people do not follow the spec.
That's not really sufficient long-term.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 13:32:03 -0500

> That's not really sufficient long-term.

I insist that the spec is *broken* and that *noone* implements what it
says about this.  (Martin hasn't yet confirmed that this is true of his
apps/mechs.  Martin?)

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 14:05:10 -0500

>  and I did have EBCDIC and ASCII machines)

This is not conclusive.

Mechanism implementations that just-send-8 are a dime a dozen.  That's
not the issue here.

Let's be much more specific:

1) Have you shipped any GSS-API C-bindings applications that explicitly
   convert text strings[0] used as input for gss_import_name() from some
   codeset[1] to ISO8859-1?

2) Have you shipped any libraries that use the GSS-API C-bindings and
   which explicitly convert text strings[2] from some codeset[1][3] to
   ISO8859-1?

3) Have you shipped any libraries that use the GSS-API C-bindings and
   which explicitly demand inputs that ultimately are to be passed to
   gss_import_name() to be in ISO8859-1?  Or any other codeset?

   a) Do the applications that use any such libraries comply?

4) Have you shipped any mechanisms that explicitly convert
   gss_import_name() text inputs *from* ISO8859-1 to some other codeset,
   and which assume that gss_import_name() text inputs are always in
   ISO8859-1 encoding?

   a) Do the applications that use such mechanisms comply?  Or do they
      work only in Latin-1 locales?  Or only for US-ASCII text?

(1), (2), (3), (3a) are all about what applications do, including
libraries as applications, as well as consumers thereof.

If the answers to (1) or (2) are "yes," then we can't do what I propose
or any variant thereof.

If the answers to (3) _and_ (3a) are "yes," then we can't do what I
propose or any variant thereof.

If the answers to (4) _and_ (4a) are "yes," then we can't do what I
propose or any variant thereof.

Have I missed any other relevant questions?

[0] provided by users, config files, ...
[1] presumably whatever the input method/current locale used
[2] provided by applications, users, config files, ...
[3] possibly defined by the library's interface specification

Cheers,

Nico
--

---

From: Martin Rex <[email-masked]>
Date: Thu, 27 Apr 2006 23:12:59 +0200 (MET DST)

> Mechanism implementations that just-send-8 are a dime a dozen.  That's
 
Two things:

We have been (and still are) shipping a(n OEM) gssapi mechanism that
that is similar to SPKM1 and uses X.509 certificates for authentication
and exclusively uses ISO-Latin-1 on ASCII-platforms for all printable
strings at the GSS-API level (independent of locale).  Characters
outside of ISO-Latin-1 are represented with "\uNNNN" unicode codepoints.

That software implements T61STRING within distinguished names according
to the original spec and some customers were suprized when illegally
encoded ISO-Latin-1 characters in their certs were visualized differently
to what they expected.

In my (network) protocol layer I transfer the target name along in
the protocol framing around the context establishment token and
it used ISO-Latin-1 transfer encoding (this is what EBCDIC components
of my software will expect) and it de-facto relies on the gssapi
mechanism that our application calls to provide/use ISO-Latin-1
for names.  In most scenarios, this name will only be used if
context establishment fails to check whether there's a mismatch between
the target name used by the initiator and the credentials owner
of the acceptor to return a meaningful error message back to the
initiator (along with the real name of the acceptor).  This mainly
affects names of acceptors/servers, which usually have less need
for non-ASCII chars in their names and are much easier to re-personalize
(in particular for flag-day changes) if it should be necessary
than a large user base.

There's one particular situation where an acceptor can be started
without knowing the name of it's accepting credentials, and when
it receives a the initial context level token, it will use the
target name supplied by the initiator, and try to acquire credentials
for that name and use it to accept the incoming security context.

-Martin

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 16:37:59 -0500

> outside of ISO-Latin-1 are represented with "\uNNNN" unicode codepoints.

Do the applications using this mechanism convert from whatever codeset
they have to ISO8859-1?  If not, does this mean that they only work in
ISO8859-1 locales or for US-ASCII codepoints only?

Nico
--

---

From: Martin Rex <[email-masked]>
Date: Fri, 28 Apr 2006 00:10:24 +0200 (MET DST)

> ISO8859-1 locales or for US-ASCII codepoints only?

There are different levels of usage in our application. :)

It is a distributed application, and it was possible (and used)
that one could run AppServers with different codepages and all
printable data (which could include printable names in ACLs)
are codepage-converted when transfered between to Servers working
with different codepages.

My intermediate communication protocol layer strictly relies on names
being ISO-latin-1, but in most installations it didn't show, because
the codepage conversion that is performed is the identity operation.
Our "upper" application layer traditionally used "just-use-8" with
what-ever codepage is configured for our APP (but our APPs notion
is seperate from the systems notion of the OS locale and also
independent of the database locale).

However, a while ago our "upper" application layers changed to use
UCS2 internally, so I had to add a conversion.  The current codepage
conversion that is performed is ISO-Latin-1 -> UCS2 and
(due to a bug...) UCS2-> ASCII 7-bit with '#' for chars that
cannot be represented.

A customer in Korea which has been taking advantage of the "lack of
conversion" in the single-byte App was surprised that this no longer
worked in the new Unicode variant of our app.

A lot of people (unfortunately also a lot of software engineers)
have the illusion that changing an application from single byte
to unicode will improve things, while in reality it significantly
impairs things in all areas where legacy code/APIs is involved,
because all of the situations where people were taking advantage
of "just-use-8"/"just-send-8" suddenly no longer work.

During 1999-2000 I told our Unicode-pushers here that this problem
would come up with our customers one day on every occasion when
I saw them, and suggested that they should fight for the necessary
resources to design a solution ahead of time, but they decided to
ignore me.  I put on a big smile when they came with the customer
problem a few weeks ago ("told you so").

But besides trying to provide a fix that brings back the behaviour
of the single-byte application, I will likely have to design a
reliable solution for the problem to support full Unicode,
which requires UTF-8 at the GSS-API level and which is
independent of the OS locale.

-Martin

---

From: Martin Rex <[email-masked]>
Date: Fri, 28 Apr 2006 00:19:56 +0200 (MET DST)

> case.  I dare MIT to do it ;)  -- I know MIT won't, because MIT isn't

I'm sorry, I strongly disagree.
HTTP originally used ISO-Latin-1 exclusively, and quite a lot
of Browser and Server software implemented it correctly.

And I don't remember people constantly complaining that the selection/use
of ISO-Latin-1 was *broken*.  It was a deliberate choice from someone
working in Switzerland who didn't have the focus on standards when
he started, but rather the ease of use and implementation with
the immediate need to represent certain accented characters from
the ISO-Latin-1 charset.

-Martin

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 17:27:04 -0500

> (due to a bug...) UCS2-> ASCII 7-bit with '#' for chars that

OK, so you do implement the Latin-1 part of the spec.  From what you
write you did not start out doing it the way the spec said in your apps,
but did in your mechs, and ended up fixing the apps to meet the spec
also.  I soooo wish you'd done it the reasonable way to begin with, I
soooo wish IETF I18N experience had been more advanced at the time the C
bindings were standardized.

Fine.  Then I see only one way out:

 - deprecate gss_import_name() (for anything other than
   GSS_C_NT_EXPORTER_NAME at least), gss_display_name, and, possibly,
   depending on what you apps and mechs do with it, gss_display_status()

   Why?  Because other implementors have an installed base such that
   switching to being compliant with the spec on this would be
   backwards-incompatible and anyways, we need more than Latin-1.

 - issue replacements that do what rfc2744 should have said (just-use-8
   -- note: not just-send-8)...
 
   ...but, to make it easy for developers on platforms that have
   facilities that deal exclusively in UTF-8 even when the current
   locale is not a UTF-8 locale, I'd agree to a parameter for selecting
   a character encoding scheme with two possible values:
   GSS_C_CES_DEFAULT (e.g., current locale's), and GSS_C_CES_UTF8.

*sigh*

Very disappointed,

Nico
--

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 27 Apr 2006 18:37:25 -0400

>    GSS_C_CES_DEFAULT (e.g., current locale's), and GSS_C_CES_UTF8.

This is clearly beyond the realm of "clarifications".  At this point I 
think Jeff is right; the only clarification we can do to GSSAPIv2u1 is to 
add an implementation note indicating there are a lot of implementations 
that just-use-8 instead of doing what the spec says.

For GSSAPIv3, we can eliminate the 8859-1 interfaces in favor of interfaces 
that use UTF-8.

-- Jeff

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 17:52:10 -0500

> I'm sorry, I strongly disagree.

Well, you did implement that part of the spec, but noone else will now
unless they have to use your mechanisms or be used by your apps.

> the immediate need to represent certain accented characters from

What's broken is requiring that the application do codeset conversions
here.  The choice of CES was poor too, particularly considering that
choosing one of the ISO8859 codesets to the exclusion of the others
screwed everyone else.  BUT, that's just par for the course for lots of
systems and protocols older than just a few years, and therfore very
much forgivable.  Most OS vendors, such as my employer, are still
dealing with the fact that Unicode didn't exist back in the 1970s (the
world of computer science then was not sufficiently globalized), so this
is   just   one   more   thing   to   fix.

Nico
--

---

From: Martin Rex <[email-masked]>
Date: Fri, 28 Apr 2006 00:54:14 +0200 (MET DST)

> bindings were standardized.

Sorry, but the GSS-API spec is more reasonable than the "just-use-8"
implementations.  The problem that we have today is by implementors
ignoring the spec instead of complaining instead of producing an
updated spec that solves the problem in a clean fashion.

>    backwards-incompatible and anyways, we need more than Latin-1.

The GSS-API v2 spec isn't broken, and there is no problem for
installations that had either been using full ISO-Latin-1 or
pure 7-bit ASCII names with an arbitrary single or multi-byte codepage
that has ASCII in the lower 128 codepoints.

>    -- note: not just-send-8)...

As the security context establishent involves token exchanges that
will at least transfer the name of the initiator to the acceptor side,
and sometimes involve names created by a third party (KDC or CA),
we will have to document it as "just-use-8" plus "just-send-8".

>    GSS_C_CES_DEFAULT (e.g., current locale's), and GSS_C_CES_UTF8.

There is a certain problem with multi-bytes including UTF-8 already:
look at how the hostbased service name is defined.  It is specified
as being composed of a servicename part and a hostname part, glued
together with an ASCII "@" sign.  So there may be a problem today
with multi-bytes when a "just-use-8" name parser searches for "@"
without knowlegde that a multi-byte codepage (and not a single-byte
codepage) is used, as I understand it.

This search might accidentally pick the trail byte 0x40 of a multibyte
character instead of the real 0x40 ASCII char.

This does affect the internal Kerberos principal name parsing
and the '/' character as well!  You can not safely certain characters
from certain multibyte locales with "just-use-8" Kerberos implementations
today even with the native Kerberos APIs.

-Martin

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 17:54:18 -0500

Yes.

> For GSSAPIv3, we can eliminate the 8859-1 interfaces in favor of interfaces 

No, we need at least an option for "just-use-8," a.k.a.,
"just-use-the-current-locale's-ces," because I don't want to have to
change aplications to do codeset conversions if I can avoid it.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 17:56:54 -0500

> we will have to document it as "just-use-8" plus "just-send-8".

NO.  What the mechanism sends is up to the mechanism's spec.

It so happens that many implementations of the Kerberos V GSS-API
mechanism just-send-8.

BUT, just-use-8 at the API does not condemn mechanisms to just-send-8.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Apr 2006 18:03:39 -0500

> codepage) is used, as I understand it.

"just-use-8" in these API slots means
"just-use-the-current-locale's-ces."

Which means that properly implemented mechanisms need not run into this
problem.

Forcing applications to convert these inputs/outputs to/from UTF-8 is
quite obnoxious.  Forcing the mechanisms to do the conversions, which
they have to be prepared to do anyways because they should not
just-send-8, means less code has to do these conversions, which I think
is a very good result.

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Thu, 27 Apr 2006 19:19:07 -0400

> that has ASCII in the lower 128 codepoints.

It is broken, because it excludes characters which cannot be represented in 
iso-8859-1.  Now, that turns out not to be a problem for mechanisms which 
don't support more than US-ASCII anyway (for example, RFC4120-compliant 
Kerberos).  It's also not a problem for mechanisms which define a funny 
escaping mechanism to represent non-8859-1 characters across the API, 
except that then the application has to know about the funny mapping, which 
is mechanism-specific and _not_ part of the GSS-API.

The GSS-API v2 simply does not admit names which cannot be represented in 
ISO-8859-1.  I consider that a bug, and I think it's unfortunate that we 
can't fix it in v2, but it's become clear that we cannot.

> we will have to document it as "just-use-8" plus "just-send-8".

No; what mechanisms do over the wire is up to the mechanism designer. 
That's _why_ we have a well-defined API, so that neither the API nor the 
application has to deal with mechanism specifics.

If the API spec says 8859-1 and a mechanism uses something else on the 
wire, then it is up to the mechanism to convert from 8859-1 to its wire 
format and back.  The same is true if the API spec says UTF-8, and the same 
is true if the API spec says "use whatever the current locale specifies" - 
you have to assume that things on both sides of the API know what "the 
current locale" is, but that still doesn't affect the wire or the guy at 
the other end of the connection, because the wire protocol still has to be 
whatever the mechanism said the wire protocol is going to be.

For RFC4120/4121, names on the wire are US-ASCII, period.  For RFC1510ter 
they will be UTF-8, period.  If we end up updating RFC4121 for RFC1510ter 
and expect to use it under GSS-APIv2u1, then the mechanism will have to 
convert Latin-1 in the API to UTF-8 on the wire and back, and names that 
can't be representeed in Latin-1 will just not work.

We're only talking about "just-use-8", in the API.  The wire protocol is up 
to the mechanism.

> codepage) is used, as I understand it.

If you use arbitrary multi-byte encodings, you may have that problem. 
However, no one's suggesting using arbitrary encodings; we're talking about 
using UTF-8, which the IETF has selected as its standard character set and 
encoding.  The UTF-8 encoding is carefully designed so it does not have 
this problem; an octet in the range 0x00-0x7f in a UTF-8 string _always_ 
stands for itself; multi-byte sequences always have the high bit set in 
every byte of the sequence.

-- Jeff

---

From: Martin Rex <[email-masked]>
Date: Fri, 28 Apr 2006 01:34:12 +0200 (MET DST)

> unless they have to use your mechanisms or be used by your apps.

I'm not the only one!

The OEM library which we ship is also available as a Single Sign-On
tool from its vendor, and it is based on a PKI toolkit that has
used ISO-Latin-1 at all of its API calls long before I started
working for my current company and got involved with GSS-API v2
standardization.

Actually, here is an Email from John Wray (the author of both
GSS-API v1 and v2 C-bindings documents rfc1509 and rfc2744) on this issue:
:
:  Date: Tue, 23 Jan 96 11:05:35 EST
:  From: "John Wray, Digital DPE, (508) 486-5210  23-Jan-1996 1026" <[email-masked]>
:  To: [email-masked]
:  Cc: "[email-masked]"@in.ENET.dec.com, [email-masked]
:  Subject: Re: question on draft-ietf-cat-gssv2-cbind-01
 [...]
:
: ii) possible future I18N support.  Today's GSSAPI assumes that character
: strings are simple ISO-latin1 strings.  At some time in the future, we may
: want to support alternate character sets, using either multi-byte or wide
: characters.  Names are the most likely target for I18N, and APIs that support
: extended character sets should behave as similar as possible to the regular
: ISO-latin1 versions.  NUL-termination doesn't work with wide characters.

So the choice of ISO-Latin-1 wasn't accidental
-- and it definitely isn't broken.
"just-use-8" that is truely broken.

> is   just   one   more   thing   to   fix.

Allowing the use to freely choose a locale requires applications to
perform codeset conversions, otherwise run into all kinds of interoperability
problems.  It may *appear* to work within small areas, but the simple
proof that most of the time it doesn't is to change the locale and
look at the data again -- if it looks different with a different
locale, then it is obvious that locales are not supported.

-Martin