Subject: [idn] I-D ACTION:draft-ietf-idn-idna-08.txt

From: [email-masked]
Date: Thu, 23 May 2002 07:29:31 -0400

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Internationalized Domain Name Working Group of the IETF.

	Title		: Internationalizing Domain Names In Applications (IDNA)
	Author(s)	: P. Faltstrom, P. Hoffman, A. Costello
	Filename	: draft-ietf-idn-idna-08.txt
	Pages		: 
	Date		: 22-May-02
	
Until now, there has been no standard method for domain names to use
characters outside the ASCII repertoire. This document defines
internationalized domain names (IDNs) and a mechanism called IDNA for
handling them in a standard fashion. IDNs use characters drawn from a
large repertoire (Unicode), but IDNA allows the non-ASCII characters to
be represented using the same octets used in so-called host names today.
This representation allows IDNs to be introduced with minimal changes to
the existing DNS infrastructure. IDNA is only meant for processing
domain names, not free text

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-idn-idna-08.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-ietf-idn-idna-08.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-ietf-idn-idna-08.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Thu, 23 May 2002 07:05:34 -0700

Greetings again. This version of the document (hopefully) addresses 
the issues that Erik Nordmark, our Area Director, brought up after 
the WG last call. It clarifies some of the topics that we in the 
Working Group didn't notice were not covered well, and there is much 
more overview material in the document.

There is one significant technical change in the document: it now 
specifies how to break up a domain name into labels. Earlier drafts 
said that this had to be done before using ToASCII and ToUnicode, but 
didn't say how to do it. IDNA now lists the full-stop characters that 
are used to break up a name.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 23 May 2002 13:00:13 -0500

> Title    : Internationalizing Domain Names In Applications (IDNA)

This is really close to being a simple codec. The expanded character
repertoire and the removal of the mandatory transliteration "requirement"
is helpful, as those were both big problems. However, the text still
presents some of these issues:

 | 6.4 Avoiding exposing users to the raw ACE encoding
 | 
 | All applications that might show the user a domain name obtained from a
 | domain name slot, such as from gethostbyaddr or part of a mail header,
 | SHOULD be updated as soon as possible in order to prevent users from
 | seeing the ACE.

This rule still requires transliteration. As has been discussed
extensively on this list, this will cause problems with data-types which
must never be transliterated. Also interesting that a recent discussion on
the [email-masked] mailing list showed that email people prefer to see
the encoding form directly, rather than the original value (at a minimum,
this means that users have one preference while admins have a different
preference, and that these are different). Also interesting that the URI
i18n draft is specifically avoiding the use of transliteration.

In light of those facts, I will repeat my multiple earlier statements that
this text should state that transliteration MUST NOT occur unless the
governing protocols and/or data-types specifications explicitly allow it.
The text as written above is an implicit update to these specifications,
even though the authoritative parties don't want it.

 | 1.1 Brief overview for application developers

 | IDNA requires that implementations process input strings with Nameprep
 | [NAMEPREP], which is a profile of Stringprep [STRINGPREP], and then
with
 | Punycode [PUNYCODE]. Implementations of IDNA MUST fully implement
 | Nameprep and Punycode; neither Nameprep nor Punycode are optional.

IDNA needs to be available as a generic codec for use with any profile of
stringprep that works with domain names. If another WG needs to develop a
stringprep profile (eg, Kerberos), they should not be required to develop
another encoding syntax.

As such, the dependancy on nameprep should be lightened, perhaps saying
"the IDNA algorithms are specifically designed for use with nameprep but
can be used with any suitable stringprep profile which requires
conversion."

Very close otherwise.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 24 May 2002 00:35:43 +0000

"Eric A. Hall" <[email-masked]> wrote:

>  | All applications that might show the user a domain name obtained
>  | from a domain name slot, such as from gethostbyaddr or part of a
>  | mail header, SHOULD be updated as soon as possible in order to
>  | prevent users from seeing the ACE.
>
> this text should state that transliteration MUST NOT occur unless
> the governing protocols and/or data-types specifications explicitly
> allow it.  The text as written above is an implicit update to these
> specifications,

There is a rule that *requires* the ASCII form to be used in all legacy
protocols and data types, and that rule explicitly overrides the rule
about hiding ACEs from users.

IDNA does not recommend altering protocols or data types in any way; it
recommends altering user interfaces.  That is what the text you quoted
is doing.

Applications are welcome to allow users to set their own user interface
preferences.  For example, most mail applications hide most header
fields by default, but allow users to see the full header if they want.
This is the same sort of thing.  Show the non-ACE form by default, and
allow users to see the ACE form if they want.

> the dependancy on nameprep should be lightened, perhaps saying "the
> IDNA algorithms are specifically designed for use with nameprep but
> can be used with any suitable stringprep profile

A Stringprep profile suitable for use with domain names would have to be
equivalent to Nameprep-followed-by-additional-prohibitions.  Otherwise
names wouldn't compare correctly when they moved between applications
using one profile and applications using another profile.

Therefore, it is correct to say that Nameprep is mandatory for IDNA.
The specialized versions that you envision would still include Nameprep,
even if it's disguised as part of a larger Stringprep profile.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 23 May 2002 20:29:53 -0500

> recommends altering user interfaces.  That is what the text you quoted

The delineation between transport and display data is normally clear
within a single constrained function or service. The problems occur when
display data is used to feed another function. The only way that we can
reduce these risks is for the specification to take the safest, most
conservative stance -- no default conversions -- and then let the
authoritative bodies implement overrides when they know it will be safe,
or where *they* are willing to risk any problems which may result from a
looser stance. It is beyond the authority of this WG to impose these risks
on the rest of the Internet community.

I don't expect this to get changed by this team, but I am obligated to go
on record with my concerns again.

> equivalent to Nameprep-followed-by-additional-prohibitions.

The profiles do not have to be subsets of nameprep. Case variance and
character variance are both likely to occur.

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 24 May 2002 10:37:41 +0200 (CEST)

Oops - notice the number 1 below:

> label.  When requirements 2 and 3 both apply, requirement 1 takes
> precedence.

s/1/2/

  Erik

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 24 May 2002 10:43:08 +0200 (CEST)

> on the rest of the Internet community.

Eric,

The way I take this is that *if* we believe there is a strict line between
protocols and user-interfaces the document presents a reasonable compromise:
No implicit change to the protocols i.e. an authoriative body needs to
make an explicit decision to change the protocol, but user-interfaces
are changed so that the user sees the friendlier name.

It turns out that there isn't such a strict line - one can debate 
forever e.g. whether the output of a Unix command line tool that can be piped
to another tool is a protocol or a user interface; same for the cut&paste
buffer.

Given this blurring we have the choice between your conservative approach
and what the document says. The conservative approach assumes that there
are authoritaive bodies for all applications and their user-interfaces, which
in many (most?) cases do not exist. 
So the likely effect of the conservative approach is either that no application
user interface is converted to display non-ACE forms (because of wide-spread
concern), or that folks do it all over the place because they do not see the
concern. Neither would be a good result.

So I don't see how the document can provide any more useful advise to
developpers with user-interface issues than it does by e.g. saying:
> 3) ACE labels obtained from domain name slots SHOULD be hidden from
> users except when the use of the non-ASCII form would cause problems or
> when the ACE form is explicitly requested. 

Any more informative advise would have to try to enumerate places (such 
as cut&paste, Unix commands using stdout, etc) where implementors should think 
more carefully about this issue. But such a list would by its nature be
incomplete, and I don't see how it can say anything more than "think more 
carefully". Thus I think we need to defer to the judgement of the user
interface designers/implementers for this issue in all cases.

   Erik

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 24 May 2002 08:01:24 -0500

Erik Nordmark wrote:

> The conservative approach assumes that there are authoritaive bodies
> for all applications and their user-interfaces, which in many (most?)
> cases do not exist.

Clarification. I do not make any such assumption, and am fully aware that
many Internet services do not have any authoritative bodies. But let's be
realistic, if nobody is going to step up to modernize the service, then it
doesn't really need fixing does it? How hard is it to write an update to
finger and get it passed?

On the contrary, I am clinging to the robustness principle. Be
conservative in what you SEND. We're in heavy and deep violation of that
principle when we say it's okay to transliterate any data that looks like
a domain name.

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 24 May 2002 16:17:10 +0200 (CEST)

> finger and get it passed?

finger is associated with an over-the-wire procotol, so this might make sense.

But what about nslookup, dig, the addresses logged by inetd in a logfile?
I read your note as assuming that some authoritative body would make
decisions for such things as well, which might be associated with
one or more protocols, but unlike finger, isn't the embodiment of the
protocol or the main implementation of the protocol.

> principle when we say it's okay to transliterate any data that looks like

The robustness principle makes sense in many places, but the IETF 
(primarely) cares about it being applied to protocols and not APIs or 
user interfaces.

I think the situation is rather analogous to the IPv6 transition.
In that space many vendors made user-interfaces (such as netstat, or
telnet printing "Trying 1.2.3.4 ...") start outputting IPv6 addresses
to the unsuspecting user.
The IETF provided no recommendations in this space.
To the extent that piping netstat output to an application which barfs on
the IPv6 addresses, this is a vendor problem and not an IETF problem.

In the IDNA space the document at least point out that implementors should
consider the effect of outputting non-ASCII at the user interface.
Having the specification mandate UI behavior seems out of place to me.

   Erik

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 24 May 2002 09:39:04 -0500

> finger is associated with an over-the-wire procotol, so this might

Well, a better choice is WHOIS, but I don't want to go there because of
there are current activities which murk up the issues. Finger has the same
kind of issues though.

There are domain names for input (user@domain@proxy), domain names for
output (last logged into hostname), and domain names in the protocol
message (the input and output). The default condition as specified in IDNA
is for the input and output mechanisms to perform conversion, but for the
protocol to reuse the legacy format. This is a fine and dandy solution for
this particular application, but it should be stated in a standards-track
update to the Finger spec, not by us.

> But what about nslookup, dig, the addresses logged by inetd in

Whatever position we take will bleed over to these other applications.
They will look at the spec and say "SHOULD do conversion" and break
everything that depends on them. If we embraced the robustness principle,
they would look at the spec and say "MUST NOT do conversion" and they
would break nothing. To be fair, additional wording about treatment of
connection identifiers, structured data and unstructured data is probably
required if we want to be thorough.

> Having the specification mandate UI behavior seems out of place to me.

That is exactly what it does now: SHOULD transliterate anything that looks
like an ACE domain name.

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 24 May 2002 21:53:21 +0000

> character variance are both likely to occur.

If one application is using Nameprep and another application is using
some other profile with different normalization or a different mapping
table, then there are going to be pairs of names that match according to
one application and don't match according to the other.  What good can
come of that?

"Eric A. Hall" <[email-masked]> wrote:

> If we embraced the robustness principle, they would look at the spec
> and say "MUST NOT do conversion" and they would break nothing.
>
> That is exactly what it does now: SHOULD transliterate anything that
> looks like an ACE domain name.

"SHOULD" is less of a mandate than "MUST NOT", and we leave a big escape
hatch: "except when the use of the non-ASCII form would cause problems
or when the ACE form is explicitly requested".

And by the way, the recommendation does not apply to "anything that
looks like an ACE domain name".  It applies to "ACE labels obtained
from domain name slots".  Domain name slots are parts of structured
data/interfaces.  If something comes from a domain name slot, then it is
a domain name, with certainty.  If a domain label passes all the checks
performed by ToUnicode, then it is an ACE label by definition.  There is
no guessing.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 24 May 2002 18:44:33 -0500

> one application and don't match according to the other.  What good can

There is no reason that such a scenario must occur. Applications can and
do define their own domain name syntaxes (localpart is different from
hostnames, is different from SRV, is different from NetBIOS, is different
from Kerberos, and so forth). There is no reason to believe that this
scenario will not continue into the future.

Under the current wording, all of the applications must use the nameprep
rules to use IDNA encoding, even if those rules are not applicable to the
application at hand.

> "SHOULD" is less of a mandate than "MUST NOT",

RFC 2119:

 | 3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
 |    may exist valid reasons in particular circumstances to ignore a
 |    particular item, but the full implications must be understood and
 |    carefully weighed before choosing a different course.

It is not any less of a mandate for the normal case. The normal case is
precisely the scenario I am worried with.

> and we leave a big escape

IE, people can break stuff all they want until the authoritative body says
to stop it. That is exactly the wrong approach.

---

From: "Adam M. Costello" <[email-masked]>
Date: Sat, 25 May 2002 00:19:58 +0000

"Eric A. Hall" <[email-masked]> wrote:

> Applications can and do define their own domain name syntaxes
> (localpart is different from hostnames, is different from SRV, is
> different from NetBIOS, is different from Kerberos, and so forth).

Maybe you are using the term "domain name" in a more general sense
than the rest of us, but domain names are compared in a particular way
(for example, they are case insensitive) and are subject to certain
constraints (for example, their labels do not exceed 63 bytes).  If an
application's identifiers conform to all the rules for domain names,
they can be called domain names, but if they have their own syntax rules
that are incompatible with domain names (for example, if they are case
sensitive or if they contain strings longer than 63 bytes) then they are
not domain names.

> rules to use IDNA encoding, even if those rules are not applicable to the

If the email folks want to use something similar to IDNA for email
address local parts, they can.  Their spec can reference IDNA
normatively, borrow from it, or even describe only its differences from
IDNA, but it will be a standard distinct from IDNA.  IDNA itself is for
domain names.

(By the way, the method of representing email address local parts as
domain labels is a limited hack.  It cannot support general email
addresses, only those whose local parts are case-insensitive and no more
than 63 bytes.  When this hack is used, local parts get treated like
domain labels; the domain name syntax rules don't change.)

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 25 May 2002 19:27:36 -0500

"Adam M. Costello" wrote:

> constraints (for example, their labels do not exceed 63 bytes).

I don't want to get pedantic about this, but you are incorrect.

Queries treat labels as case-insensitive for comparison purposes but the
data is not case-neutral. Data must be provided in the capitalization it
was entered. There are applications which are picky about case, and this
is supported by STD13. This says nothing about the interpretation of
binary data in the eight-bit range.

Length has not and will not be an issue in the same way as capitalization,
but you are wrong about that as well. The official hostname syntax has at
times defined minimum and maximum lengths which were different from the
domain name restrictions. EG, maximum of 24 characters for hostnames in
RFC 952, even though 63 character domain names were allowed in DNS by RFC
883. These were both synchronized to 63 characters max in RFC1123 although
they still have different minimums.

> If an

A domain name is any sequence of labels that can be stored in the DNS. An
i18n domain name should be any sequence of labels which can be encoded
with IDNA. Currently, you are recursively defining i18n domain names as
anything which complies with nameprep, even though we already know that
there are applications which will not do so.

> > Under the current wording, all of the applications must use the
> > nameprep rules to use IDNA encoding, even if those rules are not
> > applicable to the application at hand.
> 
> address local parts, they can.

At least naming it will be easy: yacm  [yet another codec for mail]

---

From: Dave Crocker <[email-masked]>
Date: Sat, 25 May 2002 18:50:12 -0700

>are changed so that the user sees the friendlier name.

Erik, et al,

A fun topic, though perhaps academic for this working group.  But since the 
thread has cropped up:

Strictly speaking, IDNA very much involves a protocol 
change.  Specifically, it invents a new protocol.  One that is layered on 
top of the existing DNS (and still below the application layer.)

That it does not call for a change to the installed DNS base is, of course, 
fundamental, but IDNA is far more than a UI change.

It specifies rules for interaction between independent Internet 
systems.  It's difficult to imagine a better operational definition for 
"protocol".

>to another tool is a protocol or a user interface; same for the cut&paste

One can define a UI to be sufficiently limited -- and the API between 
processes to be sufficiently textual -- so that, yes, an interprocess 
communication convention can be the same as a UI.

That trick has served Internet protocols well, but we should not carry it 
too far.

(FTP was expected to be a direct UI.  Didn't work, very well, did it?)

In reality, the model of Unix piping being a UI is not much of a UI.

That style of Unix use a classic batch system:  specify the job.  Hope it 
works right.  If it doesn't, specify the job all over again.  As we have 
come to expect in the world of GUI's, a UI that pays attention to the needs 
and capabilities of the U(ser) is pretty poor for interprocess 
communication.  At its best, a good IPC has truly terrible human factors.

So let's not get confused.  Protocols that use textual strings and 
human-typable commands are still protocols, not UI's.

There is enormous benefit in engineering the protocol to have certain 
human-friendly characteristics, but that is a long way from being 
realistically useful as a complete user interface.

>...
>more carefully about this issue.

Might be useful for some industrious folk to write a document to assist 
implementors.   It would not be a normative specification of the protocol, 
but could look at the various choices and implications.  This would 
probably expedite deployment and successful interoperability of IDNA.

d/

----------
Dave Crocker <mailto:[email-masked]>
TribalWise, Inc. <http://www.tribalwise.com>
tel +1.408.246.8253; fax +1.408.850.1850

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 27 May 2002 15:25:40 +0200

From section 4 "Conversion operations" of draft-ietf-idn-idna-08.txt
(also discussed in section 6.7):

,----
| An application converts a domain name put into an IDN-unaware slot or
| displayed to a user. This section specifies the steps to perform in the
| conversion, and the ToASCII and ToUnicode operations.
| 
| The input to ToASCII or ToUnicode is a single label that is a sequence
| of Unicode code points (remember that all ASCII code points are also
| Unicode code points). If a domain name is represented using a character
| set other than Unicode or US-ASCII, it will first need to be transcoded
| to Unicode.
`----

This last sentence seem to brush a practical problem under the rug.
Most systems aren't Unicode based today, so in fact most systems will
have to implement this unspecified transcoding.  The Unicode
consortium has not specified how to transform Unicode to/from legacy
encodings.  There are some unofficial mappings for ISO 8859-1 charsets
on www.unicode.org/Public/MAPPINGS/, but even unofficial mappings for
other charsets (in particular CJK) is not present.

Real world scenario: My machine uses ISO-8859-1.  I enter 0xB5.  How
is this transcoded into Unicode?  U+00B5 or U+03BC?  There are many
similar examples.

I think the third paragraph of the security consideration should more
clearly express that IDNA actually is vulnerable to the attack if
machines, like most machines on the Internet, use legacy encodings.

Some high-level insight on the problem:
http://www.cl.cam.ac.uk/~mgk25/unicode.html#conv

---

From: "James Seng" <[email-masked]>
Date: Mon, 27 May 2002 21:56:46 +0800

- mapping to/from legacy encoding to ISO/IEC 10646 is available on the
CDROM, if you buy the CDROM version from your standard body.

- IDNA already specify (or suggested) that if the apps is using legacy
encodings, it should transcode to Unicode first.

-James Seng

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 27 May 2002 16:49:57 +0200

> CDROM, if you buy the CDROM version from your standard body.

Does the CD include more than what is available on the net from
www.unicode.org/Public/MAPPINGS/?

I can't find offical mappings for, e.g., ISO-2022 (RFC 1468) online.

> encodings, it should transcode to Unicode first.

Yes, but I cannot find where it says how to do it, or where it cites a
reference that explains how to do it.  That's why I think the security
consideration should be more clear that the standard will enable
trivial attacks that have security consequences if the document is
implemented, because the details of legacy transcoding is left
unspecified.

If there is no standard way to translate ISO-2022-JP into Unicode,
won't different applications implement it differently?

Many machines use legacy encodings, how IDNA ends up being implemented
on such systems seems to be up to the implementor right now.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Mon, 27 May 2002 08:55:59 -0700

>machines, like most machines on the Internet, use legacy encodings.

It isn't clear what "the attack" is. There is clearly a problem for 
the user when System A transcodes text from Encoding X into Unicode 
differently than System B does, but I don't see what the security 
issue is. Could you provide some suggested wording for the security 
consideration?

--Paul Hoffman, Director

---

From: John C Klensin <[email-masked]>
Date: Mon, 27 May 2002 12:43:50 -0400

> (FTP was expected to be a direct UI.  Didn't work, very well,

Dave,

Regardless of how I might feel about the rest of your analysis
(I'll let you and Erik debate that), the above is just not
correct.  While FTP apparently lends itself to "direct UI", it
was never intended that way.   Indeed, features of the protocol,
especially some that are not widely implemented of late,
_require_ some machinery in the server and client, rather than
being "direct" protocol features.

For example, use of "TYPE ASCII" transfers requires translation
between the character set of the sender and network ASCII and
translation by the receiver from network ASCII to local
character set and formats.  Even when those hosts use ASCII, the
translations must accomodate, e.g., conversion of end of line
conventions.

Better FTP implementations also support asynchronous command
operation, reformatting of other external data types, and so on.
But the protocol is no more designed as a "direct UI" than SMTP
or POP3 (which are at least synchronous) or IMAP (which is still
a text-based protocol).  And, for whatever it is worth, the
first FTP implementations I remember (Tenex, Multics, ITS) all
used user interfaces that were different in style from the
protocol and, generally, even in the names of some of the verbs
(where a command sequence was used, rather than a single command
with several qualifiers, as was the normal usage on Tenex).

Of course, Whois or Finger might be better examples of your
point, but those protocols did not specify the details of either
the input or output formats.

     john

---

From: "James Seng" <[email-masked]>
Date: Tue, 28 May 2002 00:56:48 +0800

> www.unicode.org/Public/MAPPINGS/?

Yes and No. I know there are very well documented mappings between ISO 10646
and national standards encoding in CDROM. There are some mappings which is
on Unicode website which is not in the CDROM.

> implemented, because the details of legacy transcoding is left

No. ISO-2022 specify a mechanism for encoding reportairs using code
switching. The transcoding varies depending on the reportairs (up to 4) you
select. And it is not a 1-to-1 mapping with Unicode.

Do you mean you want IDNA to specify transcoding algoirthm too? Then how
many do you want to support? And why only these set? What if some one
creates another encoding?

IDNA already say we start from Unicode. How you do the trancoding from your
encoding to Unicode is up to you so long you have the right Unicode to start
with, ie, the original legacy encoding is irrelevant from IDN perspective.

If you did not do your transcoding from legacy to Unicode, then there is a
security problem. But the beast you describe is not IDNA. IDNA specifically
say it operates from/to Unicode only.

> won't different applications implement it differently?

There are many ways to implement an algorithm. It does not matter how you do
it, so long the end results is the same.

> on such systems seems to be up to the implementor right now.

Yes and yes. And Toolkits are available for implementors to transcode, MS
APIs, ICU, Java I18N API etc. Lets leave the implementation issues to the
implementors.

-James Seng

---

From: Dave Crocker <[email-masked]>
Date: Mon, 27 May 2002 09:58:49 -0700

At 12:43 PM 5/27/2002 -0400, John C Klensin wrote:
>   While FTP apparently lends itself to "direct UI", it
>was never intended that way.

John,

I have a pretty clear recollection that when FTP first came out the folks 
who wrote the spec honestly expected to be able to sit at a TIP, telnet 
over to an FTP server, and type commands that would effect file transfers.

I even have a vague recollection that this was used to send data to a 
printer attached to the TIP on another port.

>character set and formats.

Not if the receiver can process network ASCII directly, as some did.

>   Even when those hosts use ASCII, the

Not if they supported CRLF directly, as some did.  In fact, that was why 
CRLF was chosen and end of line, rather than a single character.  A single 
character would have made lexical analysis notably simpler.

>But the protocol is no more designed as a "direct UI" than SMTP

SMTP came around 10 years later.

However, the original MAIL command in FTP was explicitly intended to permit 
direct typing over a telnet connection.  And it was in fact used that 
way.  (That was why there was a distinction between MAIL and MLFL which did 
the data transfer over a separate FTP data connection.)

d/

----------

---

From: John C Klensin <[email-masked]>
Date: Mon, 27 May 2002 13:37:18 -0400

--On Monday, May 27, 2002 9:58 AM -0700 Dave Crocker
<[email-masked]> wrote:

> I have a pretty clear recollection that when FTP first came
> out the folks who wrote the spec honestly expected to be able
> to sit at a TIP, telnet over to an FTP server, and type
> commands that would effect file transfers.

Well, I was part of that working group.  Yes, many things were
designed so that one could do them from TIPs, however painfully.
We could debate about whether it was a primary goal; my
recollection about FTP was that it was _designed_ for transfers
between hosts with file systems, which the TIP definitely did
not have.   One might rationally argue that the notion that
Internet protocols be based on ASCII verbs and command strings,
rather than binary ones, originated in the desire for TIP
compatibility, with the debugging and clarity advantages being
noticed later (I'm not competent to debate that either way).

But, unless I have lost all memory in the ensuing years, the
two-stream model was designed to permit asynchronous operation
(in particular, to permit the STAT and ABOR commands to operate
accurately while data were moving).   The ability to do tricks
with TIPs was, if anything, secondary.

> I even have a vague recollection that this was used to send
> data to a printer attached to the TIP on another port.

Certainly possible, but I would think somewhat later.

>> For example, use of "TYPE ASCII" transfers requires
>> translation between the character set of the sender and
>> network ASCII and translation by the receiver from network
>> ASCII to local character set and formats.
> 

If by "process directly" you would include processing in their
TTY drivers, certainly so.  But the protocol specified that such
processing had to be possible.  Note that even "translation" of
network ASCII into the 4x7+1 format used on most of the PDP-10
derived machines and into the 4x9 format used on Multics, still
required some processing at the receiving end: as far as I
recall, there were no 28 bit machines on the network.

> Not if they supported CRLF directly, as some did.  In fact,
> that was why CRLF was chosen and end of line, rather than a
> single character.  A single character would have made lexical
> analysis notably simpler.

Again, our recollections differ.  Multics used LF only because
the original version of ASCII defined it as a "new line" (first
character position on next line) character.  It was changed to
optional interpretation as an "index" character (same character
position on next line) with the first revision of ASCII, in
order to accomodate some issues with EBCDIC translation as well
as better compatibility with TOPS-10, which used CR internally,
at least in its terminal interfaces (another long story, if I
recall).  CRLF was chosen as end of line in network ASCII
because it was reasonably safe (not-data-destructive) under
either local interpretation.   If I recall, like many other
systems before and since, the TIP accepted commands to specify
what it would send when it received a raw CR or LF.

> SMTP came around 10 years later.

Yes, of course.  I didn't mean to suggest otherwise.

> However, the original MAIL command in FTP was explicitly
> intended to permit direct typing over a telnet connection.
> And it was in fact used that way.  (That was why there was a
> distinction between MAIL and MLFL which did the data transfer
> over a separate FTP data connection.)

Indeed.  And that direct use of the command connection permitted
both the use of TIPs and the use of very low-end implementations
for sending mail that could not handle two-stream FTP.   But I
would have thought that would support my point that the protocol
was not designed for general direct use as a UI.

     john

---

From: "James Seng" <[email-masked]>
Date: Tue, 28 May 2002 01:56:19 +0800

Let me clarify myself:

If you looking for a 1-to-1 code point mapping between ISO-2022 and ISO
10646, then there is no such table. ISO-2022 is design quite differently,
depending on the set of reportairs you operate upon.

-James Seng

---

From: Dave Crocker <[email-masked]>
Date: Mon, 27 May 2002 11:02:23 -0700

ain't history fun...

At 01:37 PM 5/27/2002 -0400, John C Klensin wrote:
>   Yes, many things were
>We could debate about whether it was a primary goal;

Consideration of the ability to enter user commands "directly" via TIP 
Telnet was an explicit consideration in specification meetings in those 
days.

That was my only point:  Folks thought that they were providing a UI 
environment.  In fact they were not.

They were providing an environment that permitting human data entry, which 
is a far cry from anything that can be called a UI for anything other than 
programmers, and not really even then.

>  my
>not have.

That statement does not contradict my observation about the design of the 
command and response interface.

In fact, that was the whole point:  The work at that time was an experiment 
in defining protocols that were simultaneously inter-system communication 
interfaces AND user interfaces.

>    One might rationally argue that the notion that
>compatibility,

and TIP compatibility was not for programmatic interfacing.  it was for 
humans.

That was the simplistic definition of 'user interface' back then.  Now, 
usability work pays more attention to a richer range of cognitive factors.

>accurately while data were moving).

right.  sit on a tip.  direct transfer between two remote machines.  have 
the ability to issue commands over the command channel while the transfer 
is taking place between the two remote servers.

>   The ability to do tricks

I recall it in the 1973 timeframe.  I only went to the last FTP 
specification meeting.  It was in 1973.

>recall, there were no 28 bit machines on the network.

There were all sorts of machines on the net.  I did not say that all of 
them  processed CRLF directly.  I said that some of them did.  To be more 
precise, some of them could productively take raw data from the net and 
shove it to a destination like a printer or a TTY, with no translation.

>the original version of ASCII defined it as a "new line"

Again, I did not say that all machines handled crlf unchanged.  I said that 
some did.  And I said that that was a factor in the choice of crlf as eol 
character, rather than choosing the much more convenient model of a single 
character.

>Yes, of course.  I didn't mean to suggest otherwise.

Forgive my confusion.  You were introducing a number of protocol references 
as countering my claim that application protocol work in the early/mid-70's 
was trying to make protocol command environments work both for IPC and for 
UIs.

Since I was not claiming that all Internet protocols were designed with 
this intent, I am unclear how my point is supported or refuted by citing 
protocols that were not designed with this intent, especially when they 
were designed 10 years later.

One should note, however, that SMTP did not retain the two-channel model 
and that typability was part of the factor.

d/

----------

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 27 May 2002 20:20:35 +0200

> issue is. Could you provide some suggested wording for the security

The basic attack: Alice runs on host that uses Latin-1 for
input/output and enters www.µbank.com (where µ is 8859-1 0xB5).  The
domain is registered using U+00B5, but Alice's application transcode
the string using U+03BC.  Either Alice can't connect (if the other
domain doesn't exist) or she ends up talking to someone else (if the
other domain does exist).

There are many arguments that can be raised on the applicability of
the simple attack:

1 You shouldn't map ISO-8859-1 0xB5 into U+03BC, the application is
  broken and should map it into U+00B5.  My reply: This might be true,
  but this doesn't follow from IDNA as IDNA leaves the transcoding
  issue open to the implementator.  It seems as if either IDNA need to
  reference mapping tables that MUST be used for legacy encodings, or
  state that IDNA enables the attack.

2 So what if Alice talks to someone else?  DNS can be spoofed anyway,
  so this isn't a new problem.  My reply: Such problems in DNS can be
  solved with DNSSEC.  However, even if DNSSEC is used, IDNA would
  enable this new attack => same conclusion as in 1.

3 Still, so what if Alice talks to someone else?  Alice should use TLS
  or IPSEC or SSH or CMS or Kerberos or SASL or something else to
  authenticate the endpoints and protect data.  My reply: This is
  where the subtle problems appears, and I think more investigations
  are needed here.  A (probably flawed) initial attempt:

  1 TLS and PKIX certs does not support IDNA, so we must first assume
  they are extended to support IDNA.  The security implications might
  be different depending on how IDNA is implemented, so we must study
  each approach individually.  Some approaches I can see:

    1 Put IDNA strings in DN/subjectAltName of the PKIX cert.  Alice
    compares IDNA in cert with the IDNA used to contact the host.
    While this appear to solve the problem, it really only moves the
    transcoding problem elsewhere.  Perhaps the CA had to convert an
    ISO-8859-1 string it received in mail into the IDNA when
    generating the cert.  Perhaps the machine used to apply for the
    certificate used ISO-8859-1 and had to convert it into an IDNA.
    Unless you assume the whole world switches to Unicode the day IDNA
    hits the street, the transcoding problem is present in at least
    one step in the chain, and has to be solved there.  Point is, the
    mapping tables must be standardized or you open up for attacks.

    2 Put IDNA strings in DN/subjectAltName of the PKIX cert.  Alice
    compares decoded IDNA in cert with the name used to contact the
    host.  Decoding from Unicode into legacy encodings is tricky.
    Alice must have mapping tables here as well.

    3 Use UTF-8 strings in DN/subjectAltName of the PKIX cert.  Alice
    compares ToASCII(name-in-cert) with IDNA used to contact host.
    This seems similar as 1, in that unless the whole world uses
    UTF-8, the mapping has to be done somewhere and must be well
    defined there to be secure.

    4 Add {charset, string} elements indicating the character set
    used, and the string in DN/subjectAltName of the PKIX cert.  If
    charset is the same as the charset that Alice uses, it will work
    fine.  However, in all other cases, mapping tables are needed, but
    this time O(n^2) tables must exist.

  2 SASL is just a framework, so it is each SASL mechanism that has to
  be studied.  Several SASL mechanisms (HMAC-MD5, DIGEST-MD5, SRP)
  does not include names of the endpoints, so the identity of the
  other end is only implicitly known after a succesful authentication.
  Thus it only makes man-in-the-middle attacks or password cracking
  slightly easier, no real security impact.

  ... etc, the cases for Kerberos, IPSEC and SSH seems to only repeat
  the discussions above.  I have not slept for a long time, so I'll
  spare you the discussion and me the typing. :-)

Suggested modified security consideration below.  It essentially says
that unless everyone switches to UTF-8, IDNA will enable new attacks
that has security implications.

--- draft-ietf-idn-idna-08.txt.orig	Mon May 27 18:18:58 2002
+++ draft-ietf-idn-idna-08.txt	Mon May 27 20:08:44 2002
@@ -690,10 +690,24 @@
 are introduced by the encoding process or the use of these encoded
 values, apart from those introduced by the ACE encoding itself.
 
-Domain names are used by users to connect to Internet servers. The
-security of the Internet would be compromised if a user entering a
-single internationalized name could be connected to different servers
-based on different interpretations of the internationalized domain name.
+Domain names are used by users to identify and connect to Internet
+servers. The security of the Internet is compromised if a user
+entering a single internationalized name is connected to different
+servers based on different interpretations of the internationalized
+domain name.  When all systems use ASCII or Unicode, different
+interpretations are not allowed in this specification.
+
+When involved systems use non-ASCII and non-Unicode characters (such
+as ISO-8859-1 and ISO-2022-JP, which are common on the Internet),
+however, this specification leaves the transcoding problem up to the
+application.  Thus there can not be any assurance that two
+applications will not implement different transcoding rules. When two
+applications implement different transcoding rules, they will
+(assuming both domains exists) contact different servers.  Note that
+the problem can not just easily be solved by using a security protocol
+such as TLS to identify and authenticate to end points, unless these
+protocols have already solved the problem which IDNA is trying to
+solve.
 
 Because this document normatively refers to [NAMEPREP], it includes the
 security considerations from that document as well.

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 27 May 2002 20:56:41 +0200

> say it operates from/to Unicode only.

But the Internet is not Unicode only.  I don't want to get into the
design of IDNA, I only want the security consideration to accurately
reflect reality, whatever it may be.

My interpretation of reality is that IDNA has a limitation (that it
does not define mapping tables for legacy encodings) that can be
exploited on hosts that do not use ASCII/Unicode for user
input/output.

> it, so long the end results is the same.

Yes!  But since it isn't specified, different applications may
generate different output, which in turn could be exploited.

> APIs, ICU, Java I18N API etc. Lets leave the implementation issues to the

Fine.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Mon, 27 May 2002 11:54:49 -0700

Thanks, those are very good points. A few notes on your proposed 
changes to the security considerations:

At 8:20 PM +0200 5/27/02, Simon Josefsson wrote:
>+domain name.

That seems fine, and expands from what we had before to cover your 
larger concern.

>   When all systems use ASCII or Unicode, different

I think that goes too far. A user system that "uses Unicode" could 
still make wrong judgements between the keyboard and the encoding. 
For example, on the Mac, typing Option-8 inserts a bullet character. 
There are many different bullet characters in the Unicode character 
repertoire, so entering a host name that includes one of those bullet 
symbols might get the wrong result. Thus, I'd rather not use that 
last sentence.

>+as ISO-8859-1 and ISO-2022-JP, which are common on the Internet),

We're not concerned with what is common on the Internet, but what is 
common in systems.

>+protocols have already solved the problem which IDNA is trying to

That seems like a good addition.

--Paul Hoffman, Director

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 27 May 2002 21:03:27 +0200

Paul Hoffman / IMC <[email-masked]> writes:

> still make wrong judgements between the keyboard and the encoding. For
> example, on the Mac, typing Option-8 inserts a bullet character. There
> are many different bullet characters in the Unicode character
> symbols might get the wrong result. Thus, I'd rather not use that last
> sentence.

Yes, I agree.

> We're not concerned with what is common on the Internet, but what is

Make it "common in systems on the Internet" then.  I doubt you care
about how I handle IDN's in my own disconnected system?  Or just
"common".

---

From: Keld Jørn Simonsen <[email-masked]>
Date: Mon, 27 May 2002 21:37:52 +0200

> symbols might get the wrong result. Thus, I'd rather not use that 

Well, I dont think the bullets should be in the repertoire for IDN.

> We're not concerned with what is common on the Internet, but what is 

And there we have a strong correlation, ISO-8859-1 and ISO-2022 are 
very common on systems.

Kind regards
keld

---

From: vint cerf <[email-masked]>
Date: Mon, 27 May 2002 16:54:26 -0400

Dave,

isn't there a distinction to be made between the client and the server here?
The results of the user interaction were exchanges over a TCP connection distinct
from the control connection - that was so even before TCP/IP (with NCP). 
The fact that we used TELNET to drive the FTP control doesn't make the design
any less a protocol. TELNET was a convenient way of implementing the text-based
exchanges - but they had formats that included error numbers as well as text
in the expectation that some implementations of the the client would be machine
driven, I thought. 

The server side plainly had to deal with protocol exchanges between the ftp
data transfer servers on source and sink machines, in addition to the control server
exchanges carried over the TELNET protocol. 

I'm not sure whether this thread/debate is necessarily useful, is it??

vint

At 09:58 AM 5/27/2002 -0700, Dave Crocker wrote:
>I have a pretty clear recollection that when FTP first came out the folks who wrote the spec honestly expected to be able to sit at a TIP, telnet over to an FTP server, and type commands that would effect file transfers.
>
>I even have a vague recollection that this was used to send data to a printer attached to the TIP on another port.
>
>
>Not if they supported CRLF directly, as some did.  In fact, that was why CRLF was chosen and end of line, rather than a single character.  A single character would have made lexical analysis notably simpler.
>
>
>However, the original MAIL command in FTP was explicitly intended to permit direct typing over a telnet connection.  And it was in fact used that way.  (That was why there was a distinction between MAIL and MLFL which did the data transfer over a separate FTP data connection.)
>
>d/
>
>----------

---

From: "Masahiro Sekiguchi" <[email-masked]>
Date: Tue, 28 May 2002 10:30:36 +0900

> other domain does exist).

I agree the case you described is a problem.  However, I don't
agree on the point you state is the cause, i.e., I don't think
it is a transcoding problem.

Please imagine that we are living in a ideal Unicode-only world.

Assume the bank registers its domain name using Unicode U+00B5,
intending "micro-bank."  Alice *may* type a key for U+00B5 is
she is a computer engineer, but she may type U+03BC in if she is
a Greek linguist, because her keyboard (or input mapper) will be
optimized for Greek typing, or because her thinking is biased by
her Greek familiarity (She probably read the name as "mu bank",
being puzzled what it means.)

Someone might say this is a Unicode problem.  Well, partly.  For
this particular case, Unicode could have eliminated one of
U+00B5 and U+03BC.  However, there are a lot of similar cases: l
and 1, 0 and O, ´ and ', or ° and o, even in the 8859-1 range.
We can't eliminate all of these similar lookings.

Hence, I consider the basic problem is in our writing systems
and I don't think it's feasible to fix them.

> that has security implications.

Mentioning the security implications is good.  Blaming it on
transcoding is irrelevant.  Revilutionalize the world to use
UTF-8 only doesn't completely eliminate the problem, IMHO.

> +protocols have already solved the problem which IDNA is trying to

Comparing these two wordings, I support the original one, although
the best would be something in between.

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 28 May 2002 10:47:29 +0200

> and I don't think it's feasible to fix them.

I agree it isn't feasible to fix them, so that problem cannot be
solved.  That problem should probably be mentioned in the security
considerations as well.  The user gets what she enters, and if she
enters something else than she expects to enter, there will be errors
that can have security implications.  The same problem exists today,
if you enter "mybank.com" instead of "mubank.com" no technical aid can
protect you from someone calling herself "mybank.com" setting up a
similar looking web site as "mubank.com", including server certs etc.

I think the transcoding issue is separate though.  The security
implications in the scenario above can be solved by having educated
users.  They must remember the exact spelling and exact characters
used to contact their bank (this isn't unreasonable).  However, when
the system uses transcoding to convert system characters into Unicode
characters, even a user entering the "correct" spelling cannot be
certain that she ends up at the right server because transcoding
algorithms are not specified by IDNA and is left to implementations.
The user can even look at the string she entered, and the string found
in a certificate and it is possible for them to match, octet-by-octet,
with her "correct" string, and still she is talking to the wrong
server because different mapping tables exists.

When transcoding algorithms are left unspecified, the only way for the
user to be able to verify the identity of the bank is to compare the
computed IDNA strings with what she wanted.  She enters the string
using system characters, it is converted into IDNA, the server is
contacted and a certificate is fetched.  Now, to be certain of the
identity, the application will likely compare the IDNA of the server
with the one in the certificate, but the user need to compare the
computed IDNA with something she knows, to be certain that the
application didn't use a transcoding algorithm different from what the
bank used, the CA used, and the user intended.

If transcoding algorithms was specified by IDNA, the second security
problem would be reduced into the first one (modulo any mistakes in
transcoding mapping tables -- once the tables are fixed, you can't
modify them unless you want to enable the attack again).  Instead, it
could be easier to just ignore the second security problem, assuming
that all implementations will transcode system characters into Unicode
characters in the same way.  Or that the problem is rare in practice
that it doesn't matter.  Or that the whole world will switch to
Unicode.  Or that I misunderstood everything and there isn't a problem
at all.  Either way, all I'm asking is that the problem and the
expected solution is discussed a bit further in the specification, so
that I can understand how to implement IDNA securely on my Latin-1
machine.

> UTF-8 only doesn't completely eliminate the problem, IMHO.

As illustrated above, I believe there are two separate problems.  It
might be good to make both of them explicit.

---

From: Roozbeh Pournader <[email-masked]>
Date: Tue, 28 May 2002 16:52:49 +0430 (IRST)

> other domain does exist).

I'm sorry, but your example doesn't work. In nameprep, when doing Unicode 
Normalization, U+00B5 is mapped to U+03BC. So these will be the same 
domain name, and have the same ACE label.

roozbeh

---

From: Marc Blanchet <[email-masked]>
Date: Tue, 28 May 2002 09:05:18 -0400

-- lundi, mai 27, 2002 16:54:26 -0400 vint cerf <[email-masked]> 
wrote/a écrit:

> Dave,
>
> isn't there a distinction to be made between the client and the server
> here? The results of the user interaction were exchanges over a TCP
> connection distinct from the control connection - that was so even before
> TCP/IP (with NCP).  The fact that we used TELNET to drive the FTP control
> doesn't make the design any less a protocol. TELNET was a convenient way
> of implementing the text-based exchanges - but they had formats that
> included error numbers as well as text in the expectation that some
> implementations of the the client would be machine driven, I thought.
>
> The server side plainly had to deal with protocol exchanges between the
> ftp data transfer servers on source and sink machines, in addition to the
> control server exchanges carried over the TELNET protocol.
>

While I found this thread very informative of design considerations at that 
time, I agree with Vint about the not clear usefulness of this thread for 
this wg. The internet history mailing list might be more appropriate for 
this thread.

Marc.

>> I have a pretty clear recollection that when FTP first came out the
>> folks who wrote the spec honestly expected to be able to sit at a TIP,
>> telnet over to an FTP server, and type commands that would effect file
>> transfers.
>>
>> CRLF was chosen and end of line, rather than a single character.  A
>> single character would have made lexical analysis notably simpler.
>>
>>
>> However, the original MAIL command in FTP was explicitly intended to
>> permit direct typing over a telnet connection.  And it was in fact used
>> that way.  (That was why there was a distinction between MAIL and MLFL
>> which did the data transfer over a separate FTP data connection.)
>>
>> d/
>>
>> ----------

------------------------------------------
Marc Blanchet
Viagénie
tel: +1-418-656-9254x225

------------------------------------------
http://www.freenet6.net: IPv6 connectivity
------------------------------------------
http://www.normos.org: IETF(RFC,draft),
  IANA,W3C,... standards.
------------------------------------------

---

From: "Mark Davis" <[email-masked]>
Date: Tue, 28 May 2002 07:06:05 -0700

I agree with Roozbeh; in the particular case cited, there is not a
problem.

There is a very significant problem with transcoding, because
different vendors mean different things by a tag like "SJIS". The IANA
registry, unfortunately, does not specify the mapping corresponding to
this tag in sufficient detail for precise conversions. See, for
example, http://www.w3.org/TR/japanese-xml/ for an extended
discussion. However, precise tags have been proposed to IANA to
resolve these problems (in Appendix D of this document) by providing
precise mappings, at least for Japanese.

http://oss.software.ibm.com/icu/charset/index.html lists some of the
variation among different character mapping tables for other code
sets.

However, all this being said and done, the variations are usually a
very small percent of the total, and usually restricted to a few
punctuation or symbols. And to the best of my knowledge, people do not
vary in how they interpret the ISO 8859 series. Thus while the
document wants to point to the problem, it would be misleading to give
people the impression that a large number of characters will cause
security problems, when it is really restricted to a very small number
of cases.

What would be productive and useful would be to identify and list
those characters that could have problems in practice.

Mark
__________

http://www.macchiato.com

 “Eppur si muove”
----- Original Message -----
From: "Roozbeh Pournader" <[email-masked]>
To: "IDN" <[email-masked]>
Sent: Tuesday, May 28, 2002 05:22
Subject: Re: [idn] Re: Legacy charset conversion in
draft-ietf-idn-idna-08.txt

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 28 May 2002 16:37:13 +0200

> domain name, and have the same ACE label.

You are right.  What about other examples?

ISO-8859-1 0xB5:   U+00B5 / U+03BC: Mapped to U+03BC as you indicate
ISO-8859-1 0xC5:   U+00C5 / U+212B: Mapped to U+00C5
CP437 0xE1:        U+03B2 / U+00DF: ?
CP437 0xEA:        U+03A9 / U+2126: Mapped to U+03A9
CP437 0xEE:        U+03B5 / U+2208: ?
JIS-X-0208 0x2140: U+005C / U+FF3C: ?

"?" means I could not find any KC normalization in the Unicode tables
at http://www.unicode.org/charts/normalization/, I'm not sure how to
interprete this.  Possibly it means they are not normalized, in which
case there is a problem?

I agree with Mark Davis that it would be interesting to find out which
and how many characters in commonly used legacy charsets that may
cause these problems.

Also note that, if these tables are ever changed in the future, this
could also be exploited.  Application A uses mapping table version Y
and application B uses mapping table version Y+1 which transcode
and/or normalizes characters differently.  In this case someone could
register either old or new domain and fool either new or old
applications.

---

From: "Soobok Lee" <[email-masked]>
Date: Tue, 28 May 2002 23:59:59 +0900

How do you think about the variations between  KSC5601-1987 and KSC5601-1992 ?
The latter one has thousands of new hangul syllable characters (johab), but often
has been tagged as shorter "KSC5601" or interchangably as "EUC-KR". Obviously,
This loose versioning convention,which is found everywhere from Outlook , MSIE , Hotmail, and  Netscape
Navigator,  would cause  unnoticeable failures and security problems.
For example, if  the sender encloses  hangul IRI including such hangul legacy-encoded IDN in his outgoing email message,
the recipient with old s/w  may be unable to convert  it into Unicode.
What if  that happens between  machine client and servers?

Soobok Lee

----- Original Message -----
From: "Mark Davis" <[email-masked]>

---

From: Roozbeh Pournader <[email-masked]>
Date: Tue, 28 May 2002 19:54:26 +0430 (IRST)

> CP437 0xEA:        U+03A9 / U+2126: Mapped to U+03A9

These three won't cause any problem as you noted.

> JIS-X-0208 0x2140: U+005C / U+FF3C: ?

FF3C is compatibilitily mapped to 005C. So everything is fine.

> CP437 0xEE:        U+03B5 / U+2208: ?

As far as I know, all existing CP437 tables map those to SMALL SHARP S
(U+00DF) and SMALL EPSILON (U+03B5) and not SMALL BETA or ELEMENT OF.
I just checked everywhere I could, this is the list:

	http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP437.TXT
	http://microsoft.com/globaldev/reference/oem/437.htm
	http://www.kostis.net/charsets/cp437.htm

Can you point me to anywhere disagreeing?

So you are suggesting that if one wants to use CP437 as the underlying
encoding, conform to IDNA RFCs, and then dives into Unicode charts for
finding the characters of CP437 instead of using tables he can find on the
net, he is in a real mess. I agree! The only thing we are disagreeing on 
is the real source of the mess. :-)

roozbeh

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 29 May 2002 00:23:51 +0900

One concrete example:  There is one hangul syllable char which is found only in KSC5601-1992.

"똯"  0x8C70   U+B62F  #HANGUL SYLLABLE SSANGTIKEUT WA KIYEOKSIOS

I  typed in this char "똯"  in this mesage from my Outlook Express 6.0.
If you look into the mime header of this message,
  you can find the errornous mime-charset name : "KS_C_5601-1987".  Stupid and Wrong Versioning!

Legacy encodings and its versioning conventions were not designed to be used in rigorous identifier contexts,
rather to be used in textual applications like wordprocessors or html pages.
but IRI (IDN) is about to use the legacy encodings  in its own extended way regardless of this caveat.

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 29 May 2002 00:27:15 +0900

Soobok Lee

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: "Mark Davis" <[email-masked]>; "Roozbeh Pournader" <[email-masked]>; "IDN" <[email-masked]>
Sent: Tuesday, May 28, 2002 11:59 PM
Subject: Re: [idn] Re: Legacy charset conversion in draft-ietf-idn-idna-08.txt

> > > > input/output and enters www.μbank.com (where μ is 8859-1 0xB5).
> > The

---

From: Roozbeh Pournader <[email-masked]>
Date: Tue, 28 May 2002 20:40:30 +0430 (IRST)

> Stupid and Wrong Versioning!

Sure. But no protocol can fix broken software. Nag to developers of the
software to fix it. In this case, it is passing a character onto wire,
which is not in the character set it is claiming it to be in.

If a piece of software wants to work with KSC 5601:1992, and use the 
character you used, should know its mapping to Unicode. Simple.

roozbeh

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 28 May 2002 11:10:37 -0500

[bringing this back on topic]

> the input or output formats.

That illustrates the problem exactly. The default behavior as specified in
IDNA is to perform conversion, even though these standardized protocols do
not require conversion, since they implicitly allow any charset. Ergo, any
conversion which is to be performed with those protocols must be specified
as part of an update to those protocols.

Another issue from another thread is X.509 certificates. The certificate
subjects must continue to use LDH domain names in order to interoperate
with legacy applications of legacy services. New protocols may choose to
specify the use of certificate subjects with Unicode characters, but
legacy services must use IDNA sequences in order for them to continue
functioning with legacy apps. However, if an IDNA mailer passes a
converted i18n HTTPS URL to a web browser, then the comparison operation
will fail.

In these cases (as well as with things like Message-ID) there will be
basic interoperability problems unless the governing specifications
clearly define specific behaviors. The only reasonable position for us is
to prohibit auto-conversion except where these decisions have been made.

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 29 May 2002 02:14:30 +0900

The big problem lies in that such bad and loose versioning practice is *everywhere*.
And precise and correct legacy-code versioning requires that "code-range&version" checking routines
should be inserted in every application. And such checking routines themselves also should be
versioned because legacy encodings are ever evolving. Both objectives requires upgrading all existing
i18n applications, but that may be not feasible, by the same reason why IDNA is preferred over UTF8 approach
in this WG.

For example, let's assume future Outlook Express 7.0 will fix the bug by introducing
new mime charset name "KS_C_5601-1992" . The sender posts an email message in KS_C_5601-1992
to the recipient who uses Outlook Express 6.0 which knows only "KS_C_5601-1987".
What will happen?  maybe, fallback to iso8859-1 or default locale.
direct-charset-negotiation between the sender/recipient's email clients are not possible.

A XML application server receives a XML request encoded in new KS_C_5601_1992, but the server applications
don't know the new charset. what happens if the transaction was in batch mode? There may be no immediate/interactive error report to
the orginator.

That explains how much difficult it would be to introduce new version or new legacy encoding into
the IDN repertoires of supported encodings , if a certain version of IDN/IRI would have been widely deployed.

Soobok Lee

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 28 May 2002 22:12:03 +0000

Paul Hoffman / IMC <[email-masked]> wrote:

> > When all systems use ASCII or Unicode, different interpretations are
> > not allowed in this specification.
>
> A user system that "uses Unicode" could still make wrong judgements
> between the keyboard and the encoding.  For example, on the Mac,
> typing Option-8 inserts a bullet character.  There are many different
> bullet characters in the Unicode character repertoire, so entering
> a host name that includes one of those bullet symbols might get the
> wrong result.  Thus, I'd rather not use that last sentence.

Agreed.

> > When involved systems use non-ASCII and non-Unicode characters (such
> > as ISO-8859-1 and ISO-2022-JP, which are common on the Internet),
>

I don't think we need to mention any specific character sets.  If the
reader doesn't already know that there are character sets other than
ASCII and Unicode, then none of this is going to be understandable
anyway.

> > however, this specification leaves the transcoding problem up to
> > the application.  Thus there can not be any assurance that two
> > applications will not implement different transcoding rules. When
> > two applications implement different transcoding rules, they will
> > (assuming both domains exists) contact different servers.  Note
> > that the problem can not just easily be solved by using a security
> > protocol such as TLS to identify and authenticate to end points,
> > unless these protocols have already solved the problem which IDNA is
> > trying to solve.
>

I have no objection.  We might be able to compact it a bit:

    Domain names are used by users to identify and connect to Internet
    servers.  The security of the Internet is compromised if a user
    entering a single internationalized name is connected to different
    servers based on different interpretations of the internationalized
    domain name.

    When systems use local character sets other than ASCII and Unicode,
    this specification leaves the transcoding problem up to the
    application.  If applications implement different transcoding rules,
    they could interpret the same name differently and contact different
    servers.  This problem is not solved by security protocols like TLS
    that do not take local character sets into account.

[I didn't change the first paragraph except to remove the last
sentence.]

Simon, does that still say everything you want it to?

AMC

---

From: "James Seng" <[email-masked]>
Date: Wed, 29 May 2002 07:57:56 +0800

If the problem is with the implementation, we fix the implementation, not
the protocol. We have similar arguments before.

-James Seng

----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, May 29, 2002 1:14 AM
Subject: Re: [idn] Re: Legacy charset conversion in
draft-ietf-idn-idna-08.txt (in ksc5601-1987)

> A XML application server receives a XML request encoded in new
KS_C_5601_1992, but the server applications
> don't know the new charset. what happens if the transaction was in batch
mode? There may be no immediate/interactive error report to

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 29 May 2002 09:15:05 +0900

The legacy encodings are ever evolving and ever *emerging*.
That is not the implementation issue , rather protocol/architecture issues, IMO.

Soobok Lee

---

From: "James Seng" <[email-masked]>
Date: Wed, 29 May 2002 08:25:50 +0800

> The legacy encodings are ever evolving and ever *emerging*.

Yes.

> That is not the implementation issue , rather protocol/architecture
issues, IMO.

No. The problem you describe, ie the improper handling of legacy encoding is
an implementation issue, not protocol one.

Don't confuse the two.

-James Seng

> > > i18n applications, but that may be not feasible, by the same reason
why
> > IDNA is preferred over UTF8 approach
> > > in this WG.
> > >
> > > For example, let's assume future Outlook Express 7.0 will fix the bug
by
> > introducing
> > > new mime charset name "KS_C_5601-1992" . The sender posts an email
message
> > in KS_C_5601-1992
> > > direct-charset-negotiation between the sender/recipient's email
clients
> > are not possible.
> > > That explains how much difficult it would be to introduce new version
or
> > new legacy encoding into

---

From: "Soobok Lee" <[email-masked]>
Date: Wed, 29 May 2002 09:36:48 +0900

----- Original Message ----- 
  
> an implementation issue, not protocol one.

If legacy encodings are incorporated into any IDNA libraries,
 There arise  legacy encoding versioning problems, regardles of whether
   the legacy encoding implementations were correct or not  at that time of first deployment.

In summary, the issue is that: How can we build IDNA tower on the ground
of moving legacy encodings ? Is it still  implementation issue?
In Sanfrancisco and Tokyo, every building should be well architectured
against any earthquakes and  'soften' land.

Soobok Lee

---

From: "James Seng" <[email-masked]>
Date: Wed, 29 May 2002 08:45:51 +0800

Lee,

I repeat:

legacy encoding evolves -- yes; but each one is tagged properly.

Implementation which does not handle legacy encoding properly is an
implementation problem, not a protocol one. Evolving encoding standards is
not an issue.

By all means twist the arguments round in circle, in your usual style. This
will be my last email on this thread.

-James Seng

> > > > If the problem is with the implementation, we fix the
implementation,
> > not
> > > > > The big problem lies in that such bad and loose versioning
practice is
> > > > *everywhere*.
> > > > > should be inserted in every application. And such checking
routines
> > > > themselves also should be
> > > > > versioned because legacy encodings are ever evolving. Both
objectives
> > > > requires upgrading all existing
> > > > > the IDN repertoires of supported encodings , if a certain version
of
> > > > IDN/IRI would have been widely deployed.
> > > > >
> > > > > Soobok Lee
> > > > >
> > > > > > > you can find the errornous mime-charset name :
"KS_C_5601-1987".
> > > > > > > Stupid and Wrong Versioning!
> > > > > >
> > > > > > Sure. But no protocol can fix broken software. Nag to developers
of
> > the

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 29 May 2002 02:26:29 +0000

"Eric A. Hall" <[email-masked]> wrote:

> The default behavior as specified in IDNA is to perform conversion,
> even though these standardized protocols [whois and finger] do not
> require conversion, since they implicitly allow any charset.

Just because they allow any charset does not mean ToASCII is not
necessary.

If a non-ASCII domain name falls into the hands of an IDN-unaware
application, that application (or something else downstream of it) is
likely to choke on the name or fail to compare it correctly.

[For these particular examples, I think the output of finger and whois
are plain text intended for humans, not conforming to any standard
syntax, and thus not designed for applications to be able to pick out
the domain names from the surrounding text.  If that's true, IDNA does
not require the domain names to be in ASCII form, but rather recommends
that they be in non-ACE form.]

> if an IDNA mailer passes a converted i18n HTTPS URL to a web browser,
> then the comparison operation will fail.

How?  If the URI spec does not explicitly allow IDNs in the host field,
then whoever creates the URI is required by IDNA to convert the host to
ASCII before putting it in the URI.  That is the situation with today's
URI spec.

Now suppose there is a newURI spec that explicitly allows IDNs in the
host field, in some form other than ACE.  If the browser supports
newURIs, then it is IDN-aware, and hence it knows how to compare IDNs.
If the browser does not support newURIs, then the failure is the result
of passing a newURI to a browser that doesn't support newURIs, and you
can hardly blame IDNA for that.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 28 May 2002 21:51:45 -0500

> Just because they allow any charset does not mean ToASCII is not

I didn't say it was unnecessary. In fact, I said it would be appropriate.
What I also said is that the behavior needed to be defined in a
standards-track update to those protocols so that implementors know what
to do and where to do it.

> ASCII before putting it in the URI.

There's more than one way to pass data between apps. Structured exchange
over well-defined APIs is only one of them.

---

From: <[email-masked]>
Date: Wed, 29 May 2002 13:48:39 +0900 (KST)

What if  proper and stable implementation of legacy encodings in IDNA
 is not a tangilbe object due to their frequent updates/additions ? 
stringprep/nameprep added some code prohibiting rules wrt UCS versioning. 
 And IDNA mentioned about legacy-2-unicode conversions necessary in major
 user environments. legacy versioning and UCS versioning is similar and
 interdependant issues that affect IDN security.

Soobok Lee

---

From: "Doug Ewell" <[email-masked]>
Date: Tue, 28 May 2002 22:32:20 -0700

Soobok Lee <lsb at postel3 dot postel dot co dot kr> wrote:

> What if  proper and stable implementation of legacy encodings in
> IDNA is not a tangilbe object due to their frequent updates/
> additions ?

How "frequent" are updates to legacy encodings, or updates to the
mapping tables between legacy encodings and Unicode?

As for additions, they shouldn't cause a problem anyway, because they
don't break existing legacy-to-Unicode mappings.  An often-mentioned
case of adding to a legacy encoding was when Microsoft retrofitted
U+20AC EURO SIGN onto their Windows code pages (mostly at previously
unassigned code position 0x80).  The only people who suffered at all
were the ones who thought "unassigned" somehow meant that they should
map 0x80 to U+0080.  Everyone else made it just fine.

-Doug Ewell
 Fullerton, California
 (would prefer to receive only one copy of these messages)

---

From: Soobok Lee <[email-masked]>
Date: Wed, 29 May 2002 18:30:06 +0900 (KST)

> mapping tables between legacy encodings and Unicode?

FYI, I enclose the following tables from iconv library documentations.
See Korean section below.
About recent additions (1998,2002), you can visit
http://www.xfree86.org/pipermail/i18n/2002-April/003209.html

Soobok Lee
===================================================================

.* New charsets (most from libiconv)

. + Japanese
    EUC-JP (csEUCPkdFmtJapanese, EUC_JP,
      Extended_UNIX_Code_Packed_Format_for_Japanese);
    ISO-2022-JP (csISO2022JP); ISO-2022-JP-1; ISO-2022-JP-2
(csISO2022JP2);
    JIS_C6220-1969-ro (csISO14JISC6220ro, ISO646-JP, iso-ir-14, jp);
    JIS_X0201 (csHalfWidthKatakana, JIS0201, JISX0201-1976,
JISX0201.1976-0,
       X0201);
    JIS_X0208 (csISO87JISX0208, ISO-IR-87, JIS0208, JIS_X0208.1983-0,
       JIS_X0208.1983-1, JIS_X0208-1990-0, JIS_X0208.1983-1, X0208);
    JIS_X0212 (csISO159JISX02121990, ISO-IR-159, JIS0212,
JIS_X0212.1990-0,
      JIS_X0212-1990, X0212);
    SJIS (csShiftJIS, MS_KANJI, SHIFT-JIS).

. + Chinese
    BIG5 (BIG-5, BIG-FIVE, BIGFIVE, CN-BIG5 csBig5);
    EUC-CN (CN-GB, csGB2312, EUC_CN, GB2312); EUC-TW (csEUCTW, EUC_TW);
    HZ (HZ-GB-2312); ISO-2022-CN (csISO2022CN); ISO-2022-CN-EXT;
    GB_1988-80 (cn, csISO57GB1988, ISO646-CN, iso-ir-57);
    GB_2312-80 (CHINESE, csISO58GB231280, GB2312.1980-0, ISO-IR-58);
    ISO-IR-165 (CN-GB-ISOIR165).

. + Korean
    JOHAB (CP1361); EUC-KR (csEUCKR, EUC_KR); GBK (CP936);
    ISO-2022-KR (csISO2022KR);
    KSC_5601 (CP949, csKSC56011987, ISO-IR-149, KOREAN, KSC5601.1987-0,
      KS_C_5601-1987, KS_C_5601-1989, KSX1001:1992).

. + Vietnamese (independently of libiconv)
    TCVN; VIQR; VISCII; VNI; VPS.

. + Other languages
    ARMSCII-8; Georgian-Academy; Georgian-PS; WINDOWS-874 (CP874);
    MuleLao-1; CP1133 (IBM-CP1133); CP1258 (WINDOWS-1258);
    TIS-620 (ISO-IR-166, TIS620, TIS620.2529-1, TIS620-0, TIS620.2533-0,
      TIS620.2533-1).

. + Apple specifics
    MacArabic; MacCentralEurope; MacCroatian; MacCyrillic; MacGreek;
    MacHebrew; MacIceland; MacRomania; MacThai; MacTurkish; MacUkraine

. + Unicode
    JAVA; UCS-2-INTERNAL; UCS-2LE (UnicodeLITTLE); UCS-2-SWAPPED; UCS-4BE;
    UCS-4-INTERNAL; UCS-4LE; UCS-4-SWAPPED; UTF-16BE; UTF-16LE.

. + Others
    CP932; CP949 (UHC); CP950; CP866 (866, csIBM866, IBM866).
    ISO-8859-16 (ISO-IR-226, ISO_8859-16:2000).

. + Recode internal
    :libiconv: (:)   [so option -x: avoids going through libiconv]

.* New aliases (from libiconv) [list to be revised]
   csASCII (for ANSI_X3.4-1968); csHPRoman8 (for hp-roman8);
   csISOLatin1 (for ISO-8859-1); csISOLatin2 (for ISO-8859-2);
   csISOLatin3 (for ISO-8859-3); csISOLatin4 (for ISO-8859-4);
   csISOLatin5 (for ISO-8859-9);
   csISOLatin6 and ISO_8859-10:1992 (for ISO-8859-10);
   csISOLatinArabic (for ISO-8859-6); csISOLatinCyrillic (for ISO-8859-5);
   csISOLatinGreek (for ISO-8859-7); csISOLatinHebrew (for ISO-8859-8);
   csKOI8R (for KOI8-R); csPC850Multilingual (for IBM850);
   csUCS4 (for ISO-10646-UCS-4);
   csUnicode, csUnicode11, UCS-2BE, UnicodeBIG (for ISO-10646-UCS-2);
   csUnicode11UTF7 (for UNICODE-1-1-UTF-7);
   csVISCII and VISCII1.1-1 (for VISCII);
   ISO-IR-179 (for ISO-8859-13); csMacintosh and MacRoman (for macintosh);
   TCVN5712-1, TCVN5712-1:1993 and TCVN-5712 (for TCVN).

.* New surfaces
   tree (experimental).

* Version 3.5 - Francois Pinard, 1999-05.

.* Incompatible changes
. + A double dot `..' should now be used instead of a colon `:'.
. + Option --force (-f) is needed to pursue recoding despite errors.
. + There is no more quoting for special characters within charsets names.
. + Auto check (`-a') and popen (`-o') options have been withdrawn.
. + Some charsets and aliases were deleted, see `Charsets & aliases'
below.

.* Extended features
. + Program messages are available in localised form for many languages.
. + Long character names are available in French, if LANGUAGE is set to
`fr'.
. + A new request syntax allows for recode chaining, and for surfaces.
. + Option --header-file (-h) accepts a language parameter, and Perl is
new.
. + Full charset listings now show the UCS-2 value for characters.
. + Option --known=PAIRS (-k) also accepts octal and hexadecimal numbers.
. + Option --list (-l) better sorts charsets and aliases, also fully
written.
. + Charset `RFC1345' implements mnemonic+ascii+38, and is now reversible.
. + HTML is not limited anymore to Latin-1, HTML 4.0 entities are
supported.

.* New features
. + Euro support.
. + Updated RFC 1345 set of tables, from Keld Simonsen.
. + Some African charsets and transliterated forms.
. + Conversions for ISO 10646 and Unicode.
. + Combining or explosion of UCS-2 diacriticized characters and
ligatures.
. + Implementation of surfaces, see `Surfaces & aliases' below.
. + Mixed mode for recoding only comments and strings in C sources or PO
files.
. + A stand-alone recoding library gets installed, often as a shared
library.
. + Option --find-subsets (-T) lists charsets which are subsets of
another.
. + The library may generate testing data, and study character
frequencies.

.* Charsets & aliases

. + New ISO 10646 and Unicode charsets
.  - combined-UCS-2: pseudo-charset.
.  - count-characters: pseudo-charset.
.  - dump-with-names: pseudo-charset.
.  - ISO-10646-UCS-2 (UNICODE-1-1, BMP, rune, u2).
.  - ISO-10646-UCS-4 (10646, ISO-10646, UCS-4, u4).
.  - UNICODE-1-1-UTF-7 (TF-7, u7).
.  - UTF-8 (UTF-2, UTF-FSS, FSS_UTF, TF-8, u8).
.  - UTF-16 (Unicode, TF-16, u6).

. + RFC 1345.bis matters
.  - Deleted charsets
     dk-us, us-dk (because of &duplicate which `recode' does not handle
yet).
.  - New charsets
     baltic (alias is iso-ir-179); CP1250 (1250, ms-ee, windows-1250);
     CP1251 (1251, ms-cyrl, windows-1251);
     CP1252 (1252, ms-ansi, windows-1252);
     CP1253 (1253, ms-greek, windows-1253);
     CP1254 (1254, ms-turk, windows-1254);
     CP1255 (1255, ms-hebr, windows-1255);
     CP1256 (1256, ms-arab, windows-1256);
     CP1257 (1257, WinBaltRim, windows-1257);
     CWI (CWI-2, cp-hu); EBCDIC-IS-FRISS (friss);
     GOST_19768-87 with aliases of previous GOST_19768-74;
     IBM256 (256, CP256, EBCDIC-INT1); IBM875 (875, CP875, EBCDIC-Greek);
     IBM1004 (1004, CP1004, os2latin1); IBM1047 (1047, CP1047);
     ISO-8859-13 (ISO_8859-13:1998, iso-baltic, iso-ir-179a, l7, latin7);
     ISO-8859-14 (ISO_8859-14:1998, iso-celtic, iso-ir-199, l8, latin8);
     ISO-8859-15 (ISO_8859-15:1998, iso-ir-203, l9, latin9);
     KOI-7; KOI-8 (GOST_19768-74); KOI8-R; KOI8-RU; KOI8-U;
     macintosh_ce (macce); mac-is;
     NeXTSTEP (next) yet previous `recode' had it outside RFC 1345.
.  - Alias promoted to charset (with previous charset becoming alias)
     ISO-646.basic (with ISO-646.basic:1983); ISO-646.irv
(ISO-646.irv:1983);
     ISO_5427-ext (ISO_5427:1981); ISO_5428 (ISO_5428:1980);
     ISO-8859-1 (ISO_8859-1:1987); ISO-8859-2 (ISO_8859-2:1987);
     ISO-8859-3 (ISO_8859-3:1988); ISO-8859-4 (ISO_8859-4:1988);
     ISO-8859-5 (ISO_8859-5:1988); ISO-8859-6 (ISO_8859-6:1987);
     ISO-8859-7 (ISO_8859-7:1987); ISO-8859-8 (ISO_8859-8:1988);
     ISO-8859-9 (ISO_8859-9:1989); ISO-8859-10 (latin6);
     NC_NC00-10 (NC_NC00-10:81); sami (latin-lap).
.  - New aliases
     037 (for charset IBM037); 038 (IBM038); 273 (IBM273); 274 (IBM274);
     275 (IBM275); 278 (IBM278); 280 (IBM280); 281 (IBM281); 284 (IBM284);
     285 (IBM285); 290 (IBM290); 297 (IBM297); 367 (ANSI_X3.4-1968);
     420 (IBM420); 423 (IBM423); 424 (IBM424); 500, 500V1 (IBM500);
     819 (ISO-8859-1); 864 (IBM864); 868 (IBM868); 870 (IBM870);
     871 (IBM871); 880 (IBM880); 891 (IBM891); 903 (IBM903); 905 (IBM905);
     912, CP912, IBM912 (ISO-8859-2); 918 (IBM918); 1026 (IBM1026);
     ECMA-113, ECMA-113:1986 (ECMA-Cyrillic); GOST_19768-74 (KOI8);
     ISO_8859-N (ISO-8859-N) for N = 1 through 10 and 13 through 15;
     ISO_8859-10:1993 (ISO-8869-10); iso-ir-170 (INVARIANT);
     KOI8_L2 (CSN_369103); pclatin2, pcl2 (IBM852); SS636127
(SEN_850200_B).

. + New African charsets
    AFRL1-101-BPI_OCIL (t-francais, t-fra);
    AFRFUL-102-BPI_OCIL (bambara, bra, ewondo, fulfulde);
    AFRFUL-103-BPI_OCIL (t-bambara, t-bra, t-ewondo, t-fulfulde);
    AFRLIN-104-BPI_OCIL (lingala, lin, sango, wolof);
    AFRLIN-105-BPI_OCIL (t-lingala, t-lin, t-sango, t-wolof).

. + Extra miscellaneous charsets
     KEYBCS2 (Kamenicky); CORK (T1); KOI-8_CS2.

. + New HTML pseudo-charsets
    HTML_1.1 (h1);  HTML_2.0 (RFC 1866, 1866, h2); HTML-i18n (RFC 2070);
    HTML_3.2 (h3) reimplemented; HTML_4.0 (h4, HTML, h);
    deleted aliases HTF, 8859, ISO 8859, Entities, SGML, WWW, w3.

.* Surfaces & aliases
   Base64 (64, b64); Quoted-Printable (qp, Quote-Printable);
   21-Permutation (swabytes); 4321-Permutation; CR; CR-LF (cl);
   Decimal-1 (d, d1); Decimal-2 (d2), Decimal-4 (d4);
   Hexadecimal-1 (x, x1); Hexadecimal-2 (x2); Hexadecimal-4 (x4);
   Octal-1 (o, o1); Octal-2 (o2); Octal-4 (o4).
   data; test7; test8; test15; test16.

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 29 May 2002 11:39:16 +0200

> mapping tables between legacy encodings and Unicode?

The problem seem to be that there are no official mapping tables
between legacy encodings and Unicode.  So each implementator can in
theory chose whatever mapping tables she wishes, and change it every
other day as well.  In practice, implementator will pick a table they
find on the net.  Question is if this is sufficient or not, are all
mapping tables "on the net" consistent and non-evolving?

>  (would prefer to receive only one copy of these messages)

Setting "Mail-Copies-To: never" helps.

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 29 May 2002 11:51:09 +0200

> Simon, does that still say everything you want it to?

Sure.

Before the last sentence it could be useful to also add something
like: "Furthermore, if a single application uses one mapping table in
one version, and a subsequent version of the application uses a
modified mapping table, different interpretations of the same
internationalized text string may be possible even within the same
application which has security implications."

Perhaps it didn't come out very clearly, I was trying to explain that
if mapping tables are modified over time (in a way that Unicode CK
normalization does not cancel out) it will be exploitable.

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 29 May 2002 12:32:33 +0200

> Can you point me to anywhere disagreeing?

No.  I was thinking of mapping from Unicode into legacy charsets here,
which probably wasn't clear (I probably hadn't realized I made that
assumption when I wrote that message).

To understand why that is ever relevant, I think it is useful to
summarize the security related problems with internationalized text
identifiers in applications.  We have only talked about the first two
yet.

1) Internationalized text strings extend the set of possible
   characters to use in identifiers.  Traditionally a-z0-9- is such a
   small set that users can separate two elements of the set easily
   ("oh! mybank.com is not the same string as mubank.com, I must be
   under attack!").  With Unicode, users cannot easily do this
   anymore.  I think the point of CK normalization is to mitigate this
   problem, but the basic problem will still be there.  Users must
   understand that different but similar looking characters are
   different.  This problem cannot be solved entirely, and people will
   exploit this fact.

2) On systems with non-ASCII non-Unicode charsets, applications need
   to transcode strings entered using the system charset into Unicode
   before applying IDN stuff.  If different applications use different
   mapping tables, or if those mapping tables change, in a way that CK
   normalization does not cancel out, there will be additional
   attacks.

3) On systems with non-ASCII non-Unicode charsets in security
   applications, the system will have to convert Unicode in IDNA into
   system charset before sending it to those security applications.
   Here is the problem I was referring to above.  Consider a user
   entering a string in the browser containing CP437 0xE1, it is
   IDNAlized within the resolver and the user connects to the server,
   receives (for security purposes, in a TLS stream for instance),
   e.g., a certificate containing an IDNA.  If IDNA is not supposed to
   force the entire system to switch to Unicode, the application will
   have to convert the IDNA string into the system charset to display
   it to the user.  So it converts U+03B2 into CP437 0xE1 and displays
   it, the user compares the strings (possibly even by studying the
   byte sequence to be sure, if a system charset with similar looking
   symbols are used) and can verify it.  However, it is not
   unreasonable for the application to convert U+00DF into CP437 0xE1
   as well.  So the attack will only have to register the domain using
   U+00DF instead of U+03B2 in order to mount an attack.

Ok, I admit that the case in 3) can be solved in two ways.  Either the
user is shown the IDNA strings and is allowed to compare them.  Or the
system is upgraded to use Unicode in all involved applications,
including the display engine.  My point is that this attack isn't
discussed in the security considerations.  Also, the two solutions
aren't very good, IMHO, since the first one will generate a bad user
experience and the second will take years to implement.  And bad
solutions are only implemented badly or not at all, so this will most
likely generate security incidents.  Which for prudence should at
least be mentioned in security considerations.

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 31 May 2002 00:56:05 +0900

By "additions", i mean the required new tag for new version of legacy encoding, like "ks_c_5601-1992"
which should have been used, but never have been, as far as i know. Is there any central
registry that maintain the correct tag values for vaiour versions of numorous legacy encodings ??
If not, how to ensure stable and interoperable legacy-2-unicode conversion among myriads of applications ?

Soobok Lee

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 31 May 2002 01:20:38 +0900

As for the case of new character addition to legacy encodings you mentioned below,
 The real problem emerges when the older version of receiving application has
  received new-version-legacy-encoded IDN(IRI) from the new version of sender application and 
  it tries to  convert that received IDN(IRI) into unicode using the *OLDER* version of
  legacy-2-unicode convesion table, but surely it will fail in some cases.

Soobok Lee

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 30 May 2002 10:02:18 -0700

There is, to my knowledge, no such central registry. The IANA names
are, unfortunately, neither unambiguous nor comprehensive (see other
mail).

The closest I know of to this is that the ICU team has been gathering
data on different character sets based on a programmatic analysis of
how conversion actually operates on different platforms (which is
often different than the documentation of that mapping!). We have made
provision for (but have not gathered), historical data -- unless a
mapping is also in modern use on one of the platforms we collected
from: aix, glibc, hpux,os390, os400, java, solaris, windows.

See http://oss.software.ibm.com/icu/charset/ for an overview;
http://oss.software.ibm.com/cvs/icu/charset/data/xml/ for the data in
xml.

However, I want to emphasize that this is an open-source project,
*not* a registry.

Mark
__________

http://www.macchiato.com

 "Eppur si muove"
----- Original Message -----
From: "Soobok Lee" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, May 30, 2002 08:56
Subject: Re: [idn] Re: Legacy charset conversion in

> If not, how to ensure stable and interoperable legacy-2-unicode
conversion among myriads of applications ?

---

From: Keld Jørn Simonsen <[email-masked]>
Date: Thu, 30 May 2002 19:29:57 +0200

> If not, how to ensure stable and interoperable legacy-2-unicode conversion among myriads of applications ?

IANA has a registry of charsets, and many of them have mappings defined
for UCS. There is also an ISO register that has mappings between
legacy charsets and UCS, available at
http://www.dkuug.dk/cultreg/registrations/charmap

Kind regards
Keld

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 30 May 2002 21:25:28 +0000

Simon Josefsson <[email-masked]> wrote:

> >     When systems use local character sets other than ASCII
> >     and Unicode, this specification leaves the transcoding
> >     problem up to the application.  If applications implement
> >     different transcoding rules, they could interpret the same name
> >     differently and contact different servers.  This problem is not
> >     solved by security protocols like TLS that do not take local
> >     character sets into account.
>
> like: "Furthermore, if a single application uses one mapping table
> in one version, and a subsequent version of the application uses
> a modified mapping table, different interpretations of the same

How about simply changing "applications" to "different applications (or
different versions of one application)"?

AMC

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 31 May 2002 23:19:39 +0200

> different versions of one application)"?

Yes, that would be clearer.  What do the editors think about adopting
these changes?