Subject: Re: [idn] WG last call summary

From: "D. J. Bernstein" <[email-masked]>
Date: 11 Mar 2002 03:58:13 -0000

Marc Blanchet writes:
> We will be sending the documents for IESG consideration for Proposed 
> Standard on March 11th 2002.

That's outrageous. IDNA has received strong written objections from at
least fifteen regular WG participants and _hundreds_ of other people.
IDNA will cause a tremendous amount of damage, including bounced email,
web link failures, widespread user confusion, and massive costs---much
higher than necessary---for software development and deployment.

You say that you are obliged to ignore all these objections because the
IDN WG has to _do something_. But the IETF procedures don't say ``It's
okay to make an incredibly destructive modification to the Internet
protocol suite if you have to _do something_.''

I hope that the IDN WG can settle on a safe course of action. However,
until that happens, we will have to stick to the status quo.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: Dave Crocker <[email-masked]>
Date: Sun, 10 Mar 2002 20:19:07 -0800

At 03:58 AM 3/11/2002 +0000, D. J. Bernstein wrote:
>That's outrageous.

Hyperbole and sweeping, undocumented assertions are outrageous.

Forwarding specifications that have been carefully responsive to the 
problem that the group was tasked with solving... now that is NOT 
outrageous.  It's what supposed to happen in the IETF and what has, in 
fact, been done in IDN.

>IDNA will cause a tremendous amount of damage,

Such false claims have been dealt with repeatedly.  Continuing to make a 
false claim does not make it true.

>IDN WG has to _do something_.

He did not say that.  Hence there is no need to respond to the derivative 
assertions from that falsehood.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  (new)fax +1.408.850.1850

---

From: "Soobok Lee" <[email-masked]>
Date: Mon, 11 Mar 2002 15:36:22 +0900

----- Original Message ----- 
From: "D. J. Bernstein" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; <[email-masked]>
Sent: Monday, March 11, 2002 12:58 PM
Subject: Re: [idn] WG last call summary

> protocol suite if you have to _do something_.''

I agree with you.
I believe IESG+IAB wouldn't be a rubber stamp this time.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 12 Mar 2002 10:27:59 -0800

>IDN WG has to _do something_.

You are lying again, Dan. Marc never said that, and you know it.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "James Seng" <[email-masked]>
Date: Fri, 15 Mar 2002 09:44:06 +0800

Since some of them are my response which is taken out of context, it is my
duty to clarify any misleading statements.

> Once again: IDNA has received strong written objections from at least
> fifteen regular WG participants and _hundreds_ of other people.

The email protests from Taiwan is not against the current set of drafts but
the lack of handling of TC/SC within it.

The problem of TC/SC is not solved in IDNA either. If it were to be done, it
is either nameprep or punycode or both or another draft.

Therefore, it is misleading to suggest the problem exist in or the protest
is directed at IDNA particularly.

>    * ``Just protesting doesn't count, if an alternative or fix isn't
>      included'';

(1) IETF progress with internet drafts. A proactive participation in IETF
includes writing I-Ds that contributes to the work. Saying "I disagree with
it" without explaining why and proposed alternative solution or fix is not
constructive.

>    * ``the Chair's responsibility ... is to move work along ...
>      discouraging discussion of problems ... for which realistic
>      solutions ... have not been proposed'';

This is John's statement taken out of context in an exchange Deng Xiang have
with me. But I do not see anything wrong with it - it is the chairs
responsibility to move forward *any* drafts that have rough consensus within
the group.

>    * ``unless you have a TC/SC solution which you willing to contributed
>      to the group, I consider this discussion closed'';

See (1)

>    * ``You have only repeated problems that we already knows. You have
>      not demonstrated any solution which is technical possible now.''

See (1)

> These responses are all missing the point. When a user objects to IDNA,
> saying--for example---that IDNA will produce ``conflicts and chaos for
> Internet users of Han characters,'' you can't dismiss his objection by
> saying that you believe that the other proposals are even worse.

This is misleading, reading far too much.
No one say other proposals are even worse.

The statements you quoted above basically:
- Stop repeating about your problem, we all know it by now.
- None of the draft written on the problem have reach wide acceptence
  and therefore no rough consensus to move it forward

> As I commented before, the IETF procedures don't say ``It's okay to make
> an incredibly destructive modification to the Internet protocol suite if
> you have to _do something_.'' Until the IDN WG settles on a safe course
> of action, we will have to stick to the status quo.

Once again, Marc did not say "we have to _do something_".
You probably hear it in your dream.

-James Seng

---

From: "James Seng" <[email-masked]>
Date: Fri, 15 Mar 2002 11:23:21 +0800

> James Seng writes:
> > The email protests from Taiwan is not against the current set of drafts
>
> Liar.

Really? Do I need to response to this futher then?
I mean, for what I say, to you is just a lie. *laugh*.

Life would be so much better if everyone who disagree with you is a liar
isn't it :-)

-James Seng

---

From: "D. J. Bernstein" <[email-masked]>
Date: 14 Mar 2002 21:19:39 -0000

Here are some typical quotes from IDNA proponents explicitly refusing to
take these objections into account:

I also summarized why people are objecting to IDNA: ``IDNA will cause a
tremendous amount of damage, including bounced email, web link failures,
widespread user confusion, and massive costs---much higher than
necessary---for software development and deployment.'' Crocker asserts
that ``such false claims have been dealt with repeatedly.'' Let's go
back to the videotape:

   * IDNA co-author Adam Costello claimed in an IDN message on Sun, 27
     May 2001 21:30:52 +0000 that, under IDNA, ``nothing will actually
     break (mail will get through, web pages will load, etc).''

   * After the IDN WG identified several serious interoperability
     problems in the IDNA architecture---the result being that mail
     would bounce, web links would fail, etc.---I challenged Costello's
     ``nothing will actually break'' claim.

   * In an IDN message on Thu, 19 Jul 2001 04:31:48 +0000, Costello
     admitted that IDNA _would_ break things, and that his previous
     claim was wrong: ``I overstated it. I was wrong. Sorry.''
     (Naturally, he continued by saying that even more things would be
     broken by another proposal.)

I recently asked a simple question about how IDNA is supposed to work,
from a programmer's perspective: under UNIX, if LANG is en_US.UTF-8,
should the MH/NMH ``show'' mail-displaying program convert names from
the IDNA character set to UTF-8? Costello, aware that a ``yes'' answer
would cause interoperability problems and that a ``no'' answer would
mean that users see gobbledygook instead of non-ASCII glyphs, ignored
the question.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 15 Mar 2002 02:04:58 -0000

Adam M. Costello writes:
> If this were considered a show-stopper, the working group could have
> conceded defeat and disbanded long ago.

Interoperability failures _are_ a show-stopper. Imagine looking a user
in the eye and explaining to him why you screwed up his email delivery.

The whole point of IDNA was the promise of immediate IDN deployment. You
should have stopped pushing IDNA last July, when we discovered that the
deployment plan was inherently broken.

You now attempt to defend IDNA by attacking other proposals. This is the
ethically bankrupt ``We have to _do something_'' argument once again.

If every proposal before the IDN WG is harmful, then the IDN WG will
have to adopt _none_ of the proposals. The IETF procedures do not say
``It's okay to make an incredibly destructive modification to the
Internet protocol suite if you have to _do something_.'' How many times
do I have to say this before it sinks in?

(In fact, there are proposals that _don't_ break anything. I could easily
shred your ``no matter what we do, things will break'' claim. But this
doesn't matter right now.)

IDNA is harmful. Therefore, IDNA must not move forward. Once again: IDNA
has received strong written objections from at least fifteen regular WG
participants and _hundreds_ of other people. The merits of other
proposals are simply not relevant.

---

From: "William Blackwood" <[email-masked]>
Date: Thu, 14 Mar 2002 20:45:13 -0500

Bernstein's misrepresentations of fact and quotes are frequent and well
documented.  Bernstein lacks credibility.

Best to just ignore Bernstein's senseless drivel.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 15 Mar 2002 03:07:49 -0000

> The email protests from Taiwan is not against the current set of drafts

Liar. The people you're talking about wrote in response to your Last
Call for IDNA. They told you that adopting IDNA would ``damage Internet
community in China and Taiwan'' and that adopting IDNA would cause
``conflicts and chaos for Internet users of Han characters.''

I agree that these objections are based on particular deficiencies in
the basic IDNA design decisions, and that other proposals with the same
deficiencies would receive the same objections. So what?

People are telling you that IDNA will damage the Internet community. You
are saying that other proposals will damage the Internet community too.
Don't you realize how stupid this sounds?

The bottom line, once again, is that at least fifteen regular WG
participants and _hundreds_ of other people have written messages to the
IDN mailing list in opposition to IDNA.

---

From: Randy Bush <[email-masked]>
Date: Fri, 15 Mar 2002 09:29:03 -0800

dan, for those of us out here who are mono-lingual honkeys, easily
confused when we cross a county line, blah blah blan, could you
give one *simple* *concrete* example of the failure mode you fear?
and try not to use adjectives.  thanks.

randy

---

From: Randy Bush <[email-masked]>
Date: Fri, 15 Mar 2002 13:07:15 -0800

> It's possible to eliminate the interoperability failures by upgrading
> _everything_, or by ugprading _nothing_. The problem is the transition.

so, it sounds like the problem is not the proposed standard in question,
but the lack of a transition plan without holes.  shades of ipv6!

so it sounds as if the wg is missing a work item.

randy

ps: i find your arrogant mail robots sufficiently annoying and abusive
    of my time that i will no longer be corresponding with you.  oddly,
    we all think we're more important than everybody else.  of course,
    none of us are.

---

From: Lloyd Wood <[email-masked]>
Date: Fri, 15 Mar 2002 22:59:04 +0000 (GMT)

On Fri, 15 Mar 2002, Randy Bush wrote to D.J.Bernstein:

>     of my time that i will no longer be corresponding with you.

You're now objecting to anti-spam measures?

L.

thought qmail's 'sorry it didn't work out' its best feature.

<[email-masked]>PGP<http://www.ee.surrey.ac.uk/Personal/L.Wood/>

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 15 Mar 2002 18:46:05 -0600

Dave Crocker wrote:

> The transition issues for IDNA that actually pertain to Internet
> protocol behavior have been described repeatedly and are identical
> with the kind of changes that were required to adopt MIME.

I can see why you'd want to say that, but it's a poor analogy.

MIME imposed structure on unstructured data by using a subset of values,
and specifically limiting itself to particular technologies. I mean, even
RFC2047 only allowed some particular kinds of data (mostly unstructured)
to be extended, and marked structured data as off-limits. Conversely, IDNA
modifies structured data-elements, and applies to all technologies.
Resistance is futile, you will be transliterated.

A much better analogy for IDNs in general is IPv6, where every
participating protocol, data-element and application that uses the
previous data-format has to be modified if it wants to use the new
data-format. But since IDNA seeks to avoid these costs, in the end, the
protocols, data-elements and applications don't actually obtain any real
benefits from internationalization; it's all just a facade, taped onto the
surface of the Internet, sometimes in harmful ways given the artifacts of
mandatory transliteration. In this regard, the best analogy would be what
IPv6 *DIDN'T* do: pass IPv6 inside of IPv4 options and call the work
finished. That's pretty much what IDNA does.

As for breakage, it will undoubtedly occur, including in those places that
Dan mentions, but it will also happen inside protocols and namespaces.
Message-IDs will absolutely get corrupted and reinserted into the
messaging system, Kerberos and NetBIOS will get broken and then get panic
fixes, etc.

> The example given had nothing to do with Internet protocols, but rather
> with how to transition the software on a computer system to support a
> richer set of characters.  That is, of course, an interesting problem,
> but it is not part of IETF scope.

Dan's argument is actually the easiest to deal with, since it can be
addressed with the judicious and cautious use of "MAY" along with the
appropriate warnings.

Extending all known data-structures which happen to make use of DNS domain
names as key elements is the larger problem, and it is certainly within
the scope of the IETF.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/

---

From: Keith Moore <[email-masked]>
Date: Fri, 15 Mar 2002 22:00:37 -0500

> to be extended, and marked structured data as off-limits. 

more precisely, RFC 2047 was intended only for human-readable text, 
because at the time we hadn't figured out how to solve the various 
problems associated with having multiple representations of 
machine-readable text.

today we have a much better idea for how to do that.

I agree with Dave that IDNA is very similar to things we did with MIME
in both the header and the message body, and I think the experience
with MIME will be a good predictor of experience with IDNA.  

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Fri, 15 Mar 2002 21:54:27 -0600

> today we have a much better idea for how to do that.

At least IDNA only has one conversion form. Still, the point is valid that
something as precise and scope-limited as 2047 only worked well within the
scope it was intended for, namely presentation of unstructured header
fields within a specific service. I mean, I don't know of any email
clients that perform 2047 encoding on search string input, do you?

> in both the header and the message body

The notion of transliterate-for-display-so-we-don't-upgrade is similar,
but the scopes are completely different. If you are using a 2047-capable
email client, copy-n-paste between From: and To: is practically guaranteed
to work out. But all IDN approaches introduce copy-n-paste between web
browsers, email clients, WHOIS clients, ad nauseum. That also introduces
all kinds of other issues, like, "is that i18n URL I copied actually a
legal URL? [not today, no]" That's why this is closer to IPv6 than MIME.

Just to be clear, my position on this subject remains that IDNA is a
necessary and useful transfer encoding for legacy systems that need access
to i18n domain name data, but that the mandatory transliteration of all
domain names as any kind of solution to the problem is just reckless.
These problems won't go away until they are accepted and designed for.

---

From: Keith Moore <[email-masked]>
Date: Fri, 15 Mar 2002 23:50:03 -0500

> clients that perform 2047 encoding on search string input, do you?

No, but that would be a really lousy way to implement searching for 
non-ASCII text in message headers.   If I were implementing such 
a search function, I'd build an index of decoded and canonicalized 
header text rather than trying to match the query string against 
the 2047 encoding.
  
> browsers, email clients, WHOIS clients, ad nauseum. 

Copy and paste is also an issue for 2047.  The only difference is that
since 2047 doesn't apply to addresses, but only to human-readable text,
a transliteration bug is less likely to cause your mail to go to the 
wrong address.  But such bugs do sometimes cause names or subjects to 
be garbled.

But there are copy-and-paste issues even for ASCII URLs, due to %-encoding
(if you're copying from a poorly "decoded" form of the URL) and line 
wrapping.  Perhaps in hindsight URLs could have been designed differently.
But with so many people writing so many implementations of things that
handle URLs, subtleties tend to get overlooked by large numbers of 
implementors.  That's also the worst problem with doing IDNs, but it would
exist no matter what scheme were chosen.  

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 15 Mar 2002 20:56:02 -0000

Randy Bush writes:
> could you give one *simple* *concrete* example of the failure mode you
> fear?

I assume that by ``failure mode'' you're referring specifically to
interoperability problems. Here's another example.

The ezmlm-list program takes a local mailing-list name on its command
line and sends the subscriber list to stdout. People use this program in
two ways:

   (1) Displaying the results on the screen, for human consumption. For
       example, the ezmlm-list output is often piped to the standard
       ``less'' pager: ezmlm-list ~/sos | less

   (2) Feeding the results to another program. For example, people have
       shell scripts along the lines of

          #!/bin/sh
          ezmlm-list ~/sos | ( while read sub
          do
            echo ... | qmail-inject -- "$sub"
          done )

       to send personalized email to all the subscribers.

Under the IDNA deployment plan, names in the IDNA character encoding
will start showing up in subscription lists immediately. Those names
will be displayed as gobbledygook. The ezmlm-list author will be forced
to have ezmlm-list convert addresses to the local character encoding.

However, if ezmlm-list converts from the IDNA character encoding to the
local character encoding, then the script shown above will break. Mail
will bounce. That's an interoperability disaster.

What is the ezmlm-list author supposed to do? How do we make IDNA work
without encountering this failure?

``Disable the conversion by resetting LANG inside the shell script,''
you say? Yes, we could change the shell script to eliminate the problem.
But that isn't under the control of the ezmlm-list author, and the IDNA
deployment plan doesn't wait for it.

Deliberately screwing up a user's email, and then telling him how to fix
the problem, is not interoperability. It's sabotage.

``Upgrade qmail-inject to support IDNA,'' you say? Yes, I could change
qmail-inject to convert addresses to the IDNA character encoding, and
that would make the script work if the local character set is Unicode
(encoded as UTF-8, presumably). But the IDNA deployment plan doesn't
wait for that either.

There is, of course, no safe order of package upgrades if IDNA is
deployed immediately. For example, addresses are copied from web
browsers to mail programs, and from mail programs back to web browsers.

---

From: Dave Crocker <[email-masked]>
Date: Fri, 15 Mar 2002 13:59:55 -0800

>so it sounds as if the wg is missing a work item.

Well, no, not really.

The example given had nothing to do with Internet protocols, but rather
with how to transition the software on a computer system to support a
richer set of characters.  That is, of course, an interesting problem, but
it is not part of IETF scope.

The transition issues for IDNA that actually pertain to Internet protocol
behavior have been described repeatedly and are identical with the kind of
changes that were required to adopt MIME.

d/
----------
Dave Crocker <mailto:[email-masked]>
TribalWise, Inc. <http://www.tribalwise.com>
tel +1.408.246.8253; fax +1.408.850.1850

---

From: "D. J. Bernstein" <[email-masked]>
Date: 16 Mar 2002 02:21:34 -0000

> but the lack of a transition plan without holes.  shades of ipv6!

No.

IDNA is being proposed as something that new sites can use _right now_.
In contrast, nobody is telling sites that they can rely on IPv6 for
complete Internet service today.

More importantly, IDNA is a set of changes without a long-term strategy.
In contrast, specifying the desired final architecture is an explicit
part of the IPv6 project; there are many documents along these lines,
separate from the transition documents.

There certainly isn't IDN WG consensus on IDNA as the long-term IDN
solution. In fact, many of us suspect that there's consensus on the
following statement, which implies that IDNA is merely temporary:

   The long-term IDN solution will encode Unicode characters as UTF-8 on
   the wire.

The IDN WG chairs have repeatedly refused to run a straw poll on this
statement, never mind actually settling all the other important details
of what we're really trying to achieve!

If we were faced with a document proposing IDNA as a long-term solution,
I'd agree that the interoperability issues were only with the transition
plan. It _is_ possible to make a world of multiple character encodings
work correctly, even though it's remarkably bad software engineering.

Of course, the lack of interoperability is only one of IDNA's problems.

---

From: Keith Moore <[email-masked]>
Date: Sat, 16 Mar 2002 10:51:20 -0500

>    The long-term IDN solution will encode Unicode characters as UTF-8 on

I doubt we'd reach consensus on that either, since many of us suspect 
that the following statement is closer to the truth:

	The on-the-wire encoding of IDNs is irrelevant; what matters is the 
 	behavior experienced by users.

Keith

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 16 Mar 2002 11:01:53 -0600

>     behavior experienced by users.

Obviously. It's a question of getting there, however.

The final, ultimate argument in this logic-chain is that native
representation of the data in the protocol message is the ultimate design
solution, since it means less implementation work, fewer errors introduced
by wayward codecs, highest reusability by other services, etc. Managed
facades are not long-term solutions to anything, and in fact, tend to
introduce as many problems as they try to fix.

Obviously, direct encapsulation is harder with some services than others.
That doesn't mean that direct encapsulation should not be the ultimate
design target, it only means that some protocols will have to use a patch
(like IDNA) until they can either be extended or replaced.

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 16 Mar 2002 11:34:35 -0600

Dave Crocker wrote:

> The reference to searching nicely underscores that this entire thread
> pertains to local implementation issues,

"Searching" in the context of RFC2047 doesn't just apply to local issues.
Consider that IMAP SEARCH and NNTP XPAT both provide protocol mechanisms
to search server-based data. However, I don't know of any clients that
convert "ë" to "=EB" for those searches.

That's partly a local issue, but it is also a protocol issue, and in the
end it is also a service-wide issue. At some point, somebody has to say
"okay we either mandate back-end conversion to allow for unencoded
searches or we mandate encoded search behavior." Globally, it is a
user-interaction issue, where the facade failed, due to the lack of direct
integration.

This same kind of issue will apply equally to searches for an "ë" that is
embedded into a domain name component of an email address, and which has
been ACE-encoded. There are multiple secondary considerations here as
well. Searching for a single "ë" that has been encoded with ACE is not as
simple as searching for an "=EB" sequence with 2047, for example. I dare
say that substring searches are likely to prove impossible with
ACE-encoded strings.

Anyway, clearly these aren't just local implementation issues.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sat, 16 Mar 2002 11:52:02 -0600

> will bounce. That's an interoperability disaster.

under the scenario dan presents:

1) upgrading the qmail-inject program to support encoding will do no harm. 
Anything that will be changed by encoding was an illegal email address 
under the old rules.

2) upgrading both ezmlm-list and qmail-inject does no harm in the scenario 
given.

thus, I think the defensive behaviour for ezmlm-list is to implement a 
switch:

ezmlm-list 
--yes-I-have-really-upgraded-the-thing-that-will-read-this-to-understand-em
ail-addresses-with-utf-8-characters-in-them

given that in the endgame of essentially all programs being upgraded to 
undestand UTF-8 email addresses, this switch will be a bother, it should 
probably be capable of being specified in a configuration file, so that it 
can be left on and negated when needed.

3) contrary to some beliefs, watching gobbledygook is not known to cause 
permanent damage in human beings. So not upgrading anything does not cause 
much harm either.

4) yes, I do agree that specifying this level of behaviour is out of scope 
for the IETF.

                 Harald

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sat, 16 Mar 2002 11:56:01 -0600

>    The long-term IDN solution will encode Unicode characters as UTF-8 on

If this was reformulated as

  The long-term IDN solution will not use DNS

it would have my vote.

---

From: Dave Crocker <[email-masked]>
Date: Sat, 16 Mar 2002 05:54:58 -0800

>non-ASCII text in message headers.   If I were implementing such

Indeed.

The reference to searching nicely underscores that this entire thread
pertains to local implementation issues, rather than having anything at all
to do with the DNS.

The DNS does not have searching.

And with respect to the comparison with MIME, IDNA lays a specialized
character format onto an existing string environment, thereby requiring
change only for participants that which to enjoy the new feature.  It
requires no change to the infrastructure and has no impact on
non-participating DNS entities.  That is the architectural basis for
comparing IDNA to MIME.

In other words, folks, this entire thread is caused by misunderstandings of
the difference between protocol and implementation.

d/

----------

---

From: Keith Moore <[email-masked]>
Date: Sat, 16 Mar 2002 16:53:00 -0500

> introduce as many problems as they try to fix.

Every single one of the above arguments is highly dubious.

First, it's naive to assume that UTF-8 will be the native 
representation on everybody's platform - it has a number of nice
properties but it's far from clear that everybody is going to
drop what they're doing now to support it.  Second, the portion of
IDNA that does ASCII encoding is such a trivial bit of code that 
the number of failures introduced by that code will pale in 
comparison to those introduced by the other code needed to handle 
10646 (normalization, etc) which would be needed no matter what 
encoding were used.   Third, your statement about "managed facades"
is a gross over-generalization.  Numerous examples demonstrate 
that transition issues are often paramount in determining whether 
a new technology can succeed.

Simplicity is often a virtue, but IDN is inherently complex - it
reflects the tremendous variety in the world's languages and 
writing systems.  And blind faith in some vague notion of 
cleanliness is a poor substitute for engineering analysis.

Having said that, I'm all for having an alternate method to do
UTF-8 queries on-the-wire, provided it produces consistent
results with the IDNA queries, with similar efficiency and
reliability.  But the need to allow incremental upgrade of 
legacy application components strongly compels IDNA, and the 
incremental benefit of a native UTF-8 query interface beyond 
that of IDNA does not appear to justify the additional complexity.  
Nor does it justify delaying approval of IDNA.  Most of the 
problems that are anticipated with IDNA - including the various
kinds of transcription problems - will occur no matter what 
encoding is used for on-the-wire queries.

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 17 Mar 2002 00:09:25 -0000

Users suffer every day from the current character-encoding mess. Anyone
who ventures beyond ASCII is faced with endless trouble.

There's no theoretical obstacle to making multiple character encodings
work---make sure that there is, implicitly or explicitly, a perfectly
clear character encoding for every byte stream; make sure that every
copy and comparison includes all necessary conversions---but in practice
this is a disaster. It's too much work to specify the encodings. It's
far too much work to program all the necessary conversions.

Look at IDNA. Yet another character encoding. An amazingly unclear
specification of which byte streams are supposed to use that encoding.
Massive redeployment of, at a minimum, every web browser in the world.
And that's just for domain names! Is every worldwide identifier---for
example, mailbox names---supposed to have its own massive upgrade?

UTF-8 offers a way out of this mess. We do _one_ upgrade to make sure
that UTF-8 works everywhere. For example, RFC 2277, IETF Policy on
Character Sets and Languages, requires UTF-8 support in all protocols.
Then we convert all stored data to UTF-8. Then, finally, we can drop
support for the other character encodings.

Do we want programmers in twenty years to be faced with the same mess
that we have today? Or do we want them focusing on positive features for
the users?

>  	behavior experienced by users.

Everything is judged by the user experience, yes, but you are clearly
incorrect in saying that the encoding is irrelevant.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 17 Mar 2002 00:52:02 -0000

Harald Tveit Alvestrand writes:
> permanent damage in human beings.

Internationalized domain names are a failure if non-ASCII glyphs don't
appear on the screen. Have you completely lost sight of the goal here?

The interoperability problems begin when IDNA gobbledygook is converted
to the local character encoding for display. If the result is copied to
another program---through pipes, copy-and-paste, whatever---then that
program's lookups will fail. Mail will bounce, web links will fail, etc.

The conversions I'm talking about are specified in the IDNA documents.
They will produce interoperability problems visible on the wire. You are
incorrect in saying that this is outside the IETF's scope.

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 16 Mar 2002 19:42:13 -0600

Keith Moore wrote:

> representation on everybody's platform

Clarification. I did not say anything about UTF-8 on platforms, but
instead cited native representation in protocol messages. However,
if UTF-8 is the encoding of choice for a particular protocol's data or
message formats, then we know that UTF-8 is also going to be incorporated
into the necessary supporting functions at the participating end-points.
This doesn't mean that the whole box has to use UTF-8. Frankly, I don't
even think that's relevant. Instead, it is a question of whether or not
the related components (searching, as in the previous example) will be
likely to deal with UTF-8, rather than having to selectively graft an
exraneous encoding into select portions of that service in order to
provide simple functionality (as with 2047 and searching, again).

But this is also entirely irrelevant. By your argument that the transfer
encoding is irrelevant, I would like to hear your arguments as to how,
say, using EBCDIC to pass ASCII data around could possibly be seen as
reasonable design. Of course the native encodings are always best. The
fact that most of the apps are heading towards UTF-8 should tell us that
we should be designing for a long-term support infrastructure that
provides the data in the format it is going to be used in. Furthermore,
whenever the remaining services get upgraded or replaced, they should be
able to use something a little better than the best technology that 1968
money can buy.

> Second, the portion of IDNA that does ASCII encoding is such a trivial
> bit of code that the number of failures introduced by that code will
> pale in comparison to those introduced by the other code needed to
> handle 10646 (normalization, etc) which would be needed no matter what
> encoding were used.

Getting new problems in addition to shared problems is hardly an argument
in your favor. You've already conceded that 2047 has some problems with
transliteration goofiness, and that restricting it to unstructured data
limits the real damage that is caused. Are we to believe that extending
structured data with mandatory transliteration will not cause the problems
you thankfully avoided?

> Numerous examples demonstrate that transition issues are often
> paramount in determining whether a new technology can succeed.

I agree that transitional services are important. I also think that the
evidence shows that end-station layering works well when existing formats
are used as carriers for *subsets*, and when it is targeted to a specific
usage. That isn't what's being done here, though. Instead, well-known and
commmonly-used data-types will get *extended* into broader and
incompatible forms by default, and it will happen purposefully and
accidentally. This is not transitional probing, it is knowing that stuff
will break and doing it anyway.

Cripes, why do we have to do it all in a big-bang? Can't we start with the
transfer encoding (no required upgrades for anything), incrementally add
transliteration where we know it will be safe and robust (some upgrades),
and then add UTF-8 for those newer services that can make use of it (some
more upgrades)? What is the problem with this?

> cleanliness is a poor substitute for engineering analysis.

That's almost a fair shot. I do put a bunch of faith into transparent
data-types and structures. Dunno about "blind". ASCII is always best when
it's encoded as ASCII, after all.

> that of IDNA does not appear to justify the additional complexity.

The complexity required for a direct UTF-8 name-resolution service in
conjunction with simple passthru-everywhere is minor in comparison to the
complexity of transliterate-everywhere.

---

From: Keith Moore <[email-masked]>
Date: Sat, 16 Mar 2002 22:20:56 -0500

> provide simple functionality (as with 2047 and searching, again).

again, the effort to get the ASCII encoding right is trivial compared
to the effort required to do string comparisons in a way that's effective 
for searches. 

> reasonable design. 

it would be quite reasonable for applications that already had a 
deeply-wired assumption that character strings were in EBCDIC.
Which is why on BITNET and related networks, the applications 
generally exchanged data in EBCDIC even between ASCII-native hosts.

> Of course the native encodings are always best. 

utter hogwash.  first, there is no single "native" encoding of UCS, 
second; there's no encoding of IDNs that is native to the vast majority 
of deployed applications.  

> The fact that most of the apps are heading towards UTF-8 should tell us 
> that we should be designing for a long-term support infrastructure that
> provides the data in the format it is going to be used in. 

again, that's hogwash, because it completely ignores issues associated 
with transitioning existing apps and/or with maintaining multiple 
interfaces to DNS.

> Furthermore,

that's like saying that now that we have automobiles, we should completely 
rearrange all of those cities that were designed on a walking scale, that
we should change the width of roads so that they're more appropriate for
motor-driven vehicles than horse-driven vehicles, etc.  the fact is that 
once conventions get established, it's often more effecient overall to 
stick with a convention that is sub-optimal than it is to rebuild everything 
to be optimized for current conditions - particularly when those conditions
will probably change again anyway.  (e.g. it's much easier to build mass 
transit for cities designed on a walking scale...commuting by automobile 
sucks!)

if and when the rest of the protocol uses UTF-8, then the worst that 
happens is that there's an extra layer of encoding that happens before 
a DNS query is sent.  it's isolated to a single routine.  it's not 
going to hurt enough to matter.   and having the client do encoding
prior to lookup is less complex than the extra cruft that's required
to take advantage of having two kinds of DNS queries without introducing
a significant new source of failures or delays.

like I said, blind faith in cleanliness is no substitute for analysis.

> in your favor. 

no, but you're ignoring the set of new problems that comes with either
a) providing multiple DNS query interfaces or 
b) failing to provide an ascii-compatible encoding of IDNs that can be
   tolerating by existing apps
but sure enough, if you close your eyes, you can pretend that the room 
is empty.

> will break and doing it anyway.

the alternative is to say that all apps have to be rewritten to take
advantage of IDNs, and the likely result is to fragment existing apps
into >> 2 non-interoperable enclaves.  the IDNA path causes less harm.

> more upgrades)? What is the problem with this?

it completely ignores the forces that drive deployment of upgrades. 

> complexity of transliterate-everywhere.

only if you have the luxury of starting from scratch, before any
applications are written.`

Keith

---

From: Joel Rowbottom <[email-masked]>
Date: Sun, 17 Mar 2002 01:15:36 +0000

D. J. Bernstein wrote:

> that UTF-8 works everywhere.

This is wonderful, right up to the point of character equivalence. E != e !=
e(acute) yadayada. Or maybe it does...!

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 16 Mar 2002 23:17:20 -0600

Keith Moore wrote:

> > I would like to hear your arguments as to how, say, using EBCDIC
> > to pass ASCII data around could possibly be seen as reasonable
> > design.
> 

Mapped to the current topic, you are saying that ~if the current apps were
already deeply-wired for IDNA, then it would be reasonable to use. Of
course, I would agree with that, if it were anything resembling the
current situation. Too bad nothing in the known universe makes any use
whatsoever of IDNA or (more importantly) its conversion routines.

> utter hogwash.  first, there is no single "native" encoding of UCS,

You already know that UTF-8 is the implicit default encoding for it here.

> of deployed applications.

Keith, there's no IDNA encodings of IDNs that are native to *any*
applications, either current or scheduled.

[big snip]

> a DNS query is sent.

No, the "worst" that happens is that well-known and widely-used data-types
get extended by a third-party process, and then get reused.

>  and having the client do encoding

There is definitely some transitional pain in getting to direct UTF-8, but
it is non-fatal here, there, and in-between. It doesn't rewrite data, it
doesn't require a forklift upgrade of the entire Internet for rendering
purposes, and the entire Industry would be better off when it was done.
None of that can be said for IDNA transliteration. Well, the pain part
still applies.

> you're ignoring the set of new problems that comes with either

Also necessary for IDNA sooner or later, mark my words.

>    tolerating by existing apps

To repeat myself for the record, I think that the IDNA transfer-encoding
portion is necessary to provide legacy applications with access to
resources in the IDN namespace. What I am specifically arguing against is
transliteration. It is just foolish to expect that well-known and
widely-used data-types can survive being extended beyond their spec and
still function as expected. I have even provided you with evidence proof
to the contrary.

Look, if a private venture were to come out pitching an untested add-on
that rewrote every domain name which appeared on screen, the greybeards
would trot out to make measured statements cautioning against mangling of
well-known datatypes. What's the difference here?

Oh, shit, we *ARE* the greybeards!

---

From: Keith Moore <[email-masked]>
Date: Sun, 17 Mar 2002 08:39:07 -0500

> appear on the screen. Have you completely lost sight of the goal here?

by that logic, IDNs are doomed to fail, because regardless of their
representation, few existing apps will be able to properly display them.
similarly, the entire Internet is a failure because it fails to route
packets to networks that don't exist yet.

not everyone accepts your criteria for success.  actually, it appears 
that your criteria for success are incompatible with many others'
criteria for success - in particular, that existing apps be able to
accept IDNs in the spaces that DNS names are currently acceptable.

Keith

---

From: Robert Elz <[email-masked]>
Date: Sun, 17 Mar 2002 23:37:21 +0700

Date:        Sat, 16 Mar 2002 05:54:58 -0800
    From:        Dave Crocker <[email-masked]>
    Message-ID:  <[email-masked]>

  | The DNS does not have searching.

The sad thing about this whole debate is that the DNS has nothing
to do with it at all.

It is touted as "international domain names", which would make you
think that the DNS was deeply involved, but it really isn't.

Aside from the A==a rule in the DNS, it cares nothing at all about
characters or their meanings.

All the work here is for the benefit of other protocols, the DNS (which
never had a problem with domain names in any char set whatever) is being
mangled to make life easier for other protocols that really should have
been fixed years ago.

So, what the DNS has, or doesn't have, is irrelevant - what is important
is what all the protocols that are the users of domain names (every one of
them) has, or doesn't have.

kre

ps: this whole discussion is silly anyway - there's too much potential profit
to be made selling domain names in non ascii based countries for anyone to
care much about doing things the right way - all that matters is that something
be done immediately, so that profit potential can be realised.

---

From: Paul Robinson <[email-masked]>
Date: Sun, 17 Mar 2002 19:23:11 +0000

> appear on the screen. Have you completely lost sight of the goal here?

I'm just going to whack my 2p in here...

Would I be right in understanding that what you're saying is that if a given
application does not display non-ASCII glyphs correctly, when attempting to
display IDNs, then you would categorise IDNs as having been a failure?

If that is the case, you had best prepare for failure, as what you're asking
for is for every OS vendor/contributor (*BSD, Linux as well as the MS, Sun,
Cisco(!!), and VMS crowds) to start supporting IDNs before they can be
deployed. In fact, we should be altering it at an internal level within
every OS, and ensure old OSes are destroyed, so as to make sure application
developers don't have to worry about it. But of course, they will, and so
all software ever written needs to be modified if it ever had the ability to
understand domain names.

What you're suggesting is IMHO ludicrous. If that is what you are 
suggesting...
 
> program's lookups will fail. Mail will bounce, web links will fail, etc.

That's not actually completely true. Badly written software will break. In 
particular throwing up chars to the shell might break things, but that's 
particularly specific to a platform. Pipes won't break. Copy and Paste may 
depending on OS, and 'from where and to where'. Generally, it won't. If it 
does, it's because somebody has written some *really* crusty code that can't 
handle ASCII outside 0-127. The IETF shouldn't be in the business of making 
sure everythign works with badly written code.
 
> incorrect in saying that this is outside the IETF's scope.

You are kind of right, but your premise is all over the shop. The example
you've given is outside the scope of IETF (IMHO), but the broader issues
obviously are worth a look at. What I'm having problems understanding, and I
suspect others are feeling the same, is what exactly you are proposing
should be done about it apart from telling the WG to drop everything and
listen to you?

-- 
Paul Robinson

---

From: [email-masked]
Date: Sun, 17 Mar 2002 19:03:47 -0500

> sure everythign works with badly written code.

Hmm.. so you're saying that *ALL* that code out there that double-checked that
things that claimed (possibly implicitly) to be USASCII were in fact in the
0-127 range are "crusty" code?

Damn.  Sendmail 8.12.3.Beta1 is crusty - it actually bothers checking.

What's wrong with this picture?
-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

---

From: [email-masked]
Date: Sun, 17 Mar 2002 19:11:02 -0500

> generally exchanged data in EBCDIC even between ASCII-native hosts.

Damn you Keith.  I had *almost* managed to repress the joyful memories of
trying to debug what happened when a document managed to start off life
as ASCII, get translated to EBCDIC for Bitnet, to/from ASCII again for
an intermediate hop, back to EBCDIC again for another hop, and then back
to ASCII at the tail end of the journey.  And yes, that translate count
is correct, and that document was... umm... interesting. ;)

However, the day-to-day joys of Bitnet ASCII-EBCDIC translation and what
it did to uuencode was one of the *primary* motivating reasons why Base64
happened instead of uuencode.   Let's not lose sight of that debacle. ;)
-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

---

From: "James Seng" <[email-masked]>
Date: Mon, 18 Mar 2002 13:44:11 +0800

> appear on the screen. Have you completely lost sight of the goal here?

Dan,

On 28th May 2001 (oh boy, we been aroud for so long?), I response to you on
the similar discussion: See
http://www.imc.org/idn/mail-archive/msg02789.html

Particularly, it will explain why display of non-ASCII glyphs isnt as simple
as "just use UTF-8 and everything is okay".

-James Seng

---

From: "D. J. Bernstein" <[email-masked]>
Date: 18 Mar 2002 06:20:56 -0000

Let me emphasize that the IDN WG has received public written objections
to IDNA from at least fifteen regular WG participants and _hundreds_ of
other people. Everyone agrees that the objections together accuse IDNA
of causing a tremendous amount of damage, including bounced email, web
link failures, widespread user confusion, and more.

The IDN WG chairs have nevertheless sent IDNA to the IESG. That's
outrageous. It is, within the IETF, the most blatant abuse of power I've
ever seen.

I'm willing to spend some time discussing the technical issues that led
to these objections; but I'd like readers to keep the context in mind.
These specifications should never have been sent to the IESG.

Now, back to the technical discussion.

Keith Moore writes:
> not everyone accepts your criteria for success.

``Internationalized domain names are a failure if non-ASCII glyphs don't
appear on the screen.'' What kind of idiot would disagree with that?
Would you like to explain, Keith, how IDNs can be a success without
non-ASCII glyphs appearing on the screen?

Paul Robinson asks whether I'm saying that _one_ bad display means that
IDNs are a _complete_ failure. Of course not. One bad display means one
failure. One billion bad displays mean one billion failures.

Obviously the IDNA authors understand the desire to put non-ASCII glyphs
on the screen. Why is this obvious? Because the IDNA documents say that
the ad-hoc IDNA character encoding ``SHOULD be hidden from users,'' with
various exceptions. The documents spend quite a bit of time explaining
how to convert strings out of the IDNA character encoding.

The problem is that this conversion creates interoperability failures.
Mail will bounce. Web pages will fail to load. The same failures would
not have occurred with ASCII domain names.

These failures are visible on the wire, and are caused by a proposed
IETF specification. The assertion that they're outside the IETF's scope
is patently absurd.

On Sun, 27 May 2001 21:30:52 +0000, IDNA coauthor Adam Costello claimed
on the IDN WG mailing list that nothing would actually break. On Thu, 19
Jul 2001 04:31:48 +0000, Costello admitted on the IDN WG mailing list
that his claim was wrong. Things will break.

To answer another question from Paul: The problem happens when the
converted data is fed to _any_ network program that isn't aware of IDNA.
It doesn't matter whether that program is an obsolete 7-bit application
or a modern 8-bit-clean application; what matters is that the program
isn't undoing the IDNA conversion.

When I give specific examples of these IDNA interoperability problems,
someone inevitably responds that we can fix those examples by turning
off the initial IDNA conversion. That is a silly response, for two
reasons:

   * Preventing _all_ the interoperability problems means turning off
     conversion for _every_ display subject to IDNA-unaware piping,
     IDNA-unaware copy-and-paste, etc.---and that's a massive failure to
     achieve the IDN goal.

   * The proposal on the table, IDNA, does not disable these
     conversions. It explicitly encourages these conversions.

Anyway, while some IDNA proponents obviously want to slither between
interoperability failures and display failures, Costello has already
admitted that IDNA will cause both types of failures.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 18 Mar 2002 07:08:19 -0000

[email-masked] writes:
> Hmm.. so you're saying that *ALL* that code out there that
> double-checked that things that claimed (possibly implicitly) to be
> USASCII were in fact in the 0-127 range are "crusty" code?

Time for some facts.

Sendmail, by default, does _not_ enforce the 0-127 restriction for mail
message headers. It allows bytes 160-255. Otherwise European users would
be dumping Sendmail even more quickly than they are today; ISO 8859-1
Subject lines are extremely popular.

Sendmail _does_ discard bytes 128-159 in mail message headers, because
it uses those bytes internally for its internal macro handling. Those
bytes aren't used in ISO 8859-1, but they are used in UTF-8. See
http://pi.cr.yp.to for a concrete example.

I sent Allman some email in February 1999 suggesting that he convert

   128 -> 255 160
   129 -> 255 161
   ...
   159 -> 255 191
   255 -> 255 255

with the opposite conversion on output. There have been several
security-fix releases of sendmail since then, so we could have had the
128-159 problem fixed on a huge number of machines. But he ignored the
suggestion. Apparently he doesn't care about international users.

People proposed more than a decade ago that the IETF require 8-bit-clean
mail software. (See, for example, Andre Pirard's ietf-smtp message dated
Tue, 19 Feb 91 12:08:00 +0100.) The only objection to this requirement
was the claim that 8-bit support would take a long time to be deployed.
Paul Vixie said that he had some seven-year-old sendmail binaries, for
example, and concluded ``with near-certainty'' that ``any changes to the
SMTP spec will take at least a decade to reach 90% of the critical
server population.''

In fact, it took less than a decade for every critical server to add
support for 8-bit message bodies, even though the IETF _still_ doesn't
require this. If the SMTP specification had been changed in 1991 to
require transparent 8-bit handling in both the header and the body, we
wouldn't have Sendmail's UTF-8 problems today.

Sendmail's continued data corruption is an embarrassment to the Sendmail
company. The fact that RFC 2821 and RFC 2822 allow this garbage is an
embarrassment to the IETF.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 18 Mar 2002 08:42:27 -0000

James Seng writes:
> Particularly, it will explain why display of non-ASCII glyphs isnt as
> simple as "just use UTF-8 and everything is okay".

Here we go again: IDN WG co-chair James Seng responds to a discussion of
IDNA's flaws by attacking another proposal.

Think about that for a moment. IDNA has received massive objections
because it's horribly destructive. Seng has sent it to the IESG anyway.
How does he defend himself? By claiming that other proposals are worse.

What Seng fails to do is compare IDNA to the status quo. Sure, the
status quo forces sites to stick to ASCII, which is visually unpleasant
for many users. But the status quo also gets the mail delivered, makes
web links work, avoids the chaos and confusion that IDNA would produce,
avoids massive software upgrades and redeployment, etc.

Refusing to consider the status quo as an option is another way of
saying ``We have to _do something_.'' That's a recipe for the Internet
protocol suite to spiral down into a neverending nightmare. That's not
how the IETF works.

More importantly, in the absence of consensus, the status quo wins.
Something is seriously wrong when an internationalization proposal draws
objections from hundreds of Chinese-speaking users, for example.

P.S. If you'd like to see an example of a serious UTF-8 IDN proposal,
try http://cr.yp.to/proto/idnc3.html.

---

From: "James Seng" <[email-masked]>
Date: Mon, 18 Mar 2002 17:19:00 +0800

> IDNA's flaws by attacking another proposal.

Nope. If you read my mail again, you realise that the display problem which
you keep banging on occurs for both UTF-8 & punycode or whatever system you
come up with.

Display problem is problem at a higher level, one that can only be solved by
implementation and deployment, not protocol standardisation. If you
disagree, then explain how protocol can enforce what the display/render
engine do?

> How does he defend himself? By claiming that other proposals are worse.

Nope. The co-chairs believe we have rough consensus within the working group
to move forward these documents. The merits of other proposal(s) are
irrelevant to the status of the ones we move forward.

If and when the other documents have rough consensus, the co-chairs will
move them forward likewise.

> protocol suite to spiral down into a neverending nightmare. That's not

Once again, neither Marc or myself have said "we have to _do something_".
You hear it in your dream (or nightmare). Repeating it again and again does
not make it true, even if you wish it is so.

The IETF works by moving documents which have rough consensus. And that is
what we are doing here.

> objections from hundreds of Chinese-speaking users, for example.

In the absense of rough consensus, yes, status quo wins. But not in this
case.

The objections from the Taiwanese (non-wg members btw) are noted to the
group.
See http://www.imc.org/idn/mail-archive/msg05977.html

None of them provide any useful technical information to the last call.
Neither are the protest within the IETF process as described in RFC2026
Section 6.5.

-James Seng

---

From: Masataka Ohta <[email-masked]>
Date: Mon, 18 Mar 2002 18:35:35 +0859 ()

D. J. Bernstein;

> for many users.

Though I, personally, can read, for example, Hangul (Korean Alphabet),
for almost all international users, it is much much more unpleasant to be
displayed a domain name in Hangul characters than be displayed Latin
character equivalent of it.

Though I, personally, can read many Chineses characters, for almost
all Japanese users, it is much much more unpleasant to be displayed
a domain name in Chinese characters than be displayed Latin character
equivalent of it. Note that Kanji characters of Japan is not Chinese
characters any more, just as Greek characters are not Latin characters.

So, even if your software can display Hangul and/or Chinese characters,
it does not make the situation better.

							Masataka Ohta

---

From: Thor Harald Johansen <[email-masked]>
Date: Mon, 18 Mar 2002 11:14:44 +0100 (CET)

> as "just use UTF-8 and everything is okay".

I've been silently following this discussion for a while. Why is it that
international domain names can't be implemented with some Unicode
encapsulating 7-bit encoding? I understand that there will be issues of
case sensitivity (for latin alphabets) on servers and incorrect display on
old clients, but how many of you type your URLs in mixed case? The
transition will go largely unnoticed. It'll be the responsibility of the
creator of the domain to make sure his/her site is available, anyway.

Have I missed an important point here, or...?
-- 
Thor Harald Johansen

---

From: Paul Robinson <[email-masked]>
Date: Mon, 18 Mar 2002 11:57:17 +0000

> 0-127 range are "crusty" code?

No. I'm saying that if a piece of software gets input that is unexpectedly 
'out of range' and then crashes and burns it is badly written. Sure, 
checking data is a good thing. Not checking it and letting things 'just 
break' is stupid. There are 14-year olds in high school in their second 
class of visual basic programming who know this.

To go back to the original argument, trying to put a stop on a standard
getting through because it breaks a piece of softwre written some time ago
that a small percentage of people use today is a dumb idea. OK, let's take
IDNA. How many people are there in China that would benefit from this
process? Let's say the population is 2 billion, and it would benefit 5%.  
That's 100 million people. How many people would be 'hurt' by the example
given breaking? Are there 100 million people out there who would be
affected? If yes, then the process needs to stop and take that into account.

To argue that it's upsetting the existing user base is also flawed - this is 
not about keeping things cosy for ourselves at a cost to those outside of 
the Latin alphabet. If somebody can show how this process will break things 
for the majority of users (in other words using IE 4 and above, with Outlook 
Express for mail), THEN it would be a good idea to question 
interoperability. I'm not saying that we should be considering IE and 
Outlook Express as de facto standards, just as a benchmark to how many 
people are likely to experience 'breakages' as a result of IDNA things being 
pushed through.
 
> What's wrong with this picture?

There's too much blue in it, and its skewed to the left a little. :-)

-- 
Paul Robinson

---

From: Paul Robinson <[email-masked]>
Date: Mon, 18 Mar 2002 12:22:51 +0000

> objections from hundreds of Chinese-speaking users, for example.

Now *that's* a convincing argument. However, the status quo in this case is 
broken. OK, let me rephrase - it's broken for people in China who don't 
understand the latin alphabet or who want their e-mail addresses outside of 
USASCII 0-127. Something *should* be done, but your argument has a hint of 
'I never want anything done, ever' about it, which is putting people off.

My current understanding is that the majority of debate appears to be about
interoperability with legacy applications - to me, it sounds a little like
the brain-dead 'we can't roll out IPv6 as it breaks IPv4' argument. In other
words, the code re-write and upgrde is going to have to happen sometime. You
just appear to want to put it off whilst we all sit around and procrastinate
for a while. History has shown this is not a Good Plan. However, the more I
start to re-read docs and think about it, the more I am convincing myself
that you may be right.

-- 
Paul Robinson

---

From: Keith Moore <[email-masked]>
Date: Mon, 18 Mar 2002 11:07:15 -0500

> appear on the screen.'' What kind of idiot would disagree with that?

anybody with two working brain cells should be able to figure out
that IDNs won't be able to be displayed with software that was
designed and written before IDNs existed.   even if IDNs were bare
UTF-8, they still wouldn't be able to be input from software
that predated IDNs because such software wouldn't do the appropriate
canonicalization/normalization and wouldn't know how to properly
compare two IDNs for equivalence.

but you know this already.  you just choose to ignore it so that
you can berate people who don't want to rewrite all applications to
suit your whim.

Keith

---

From: Keith Moore <[email-masked]>
Date: Mon, 18 Mar 2002 11:13:09 -0500

> Neither are the protest within the IETF process as described in RFC2026

I have to disagree with this.

first of all, there's no such thing as a wg member.   anybody who
contributes to the discussion is as much a member as anyone else.

second, this is useful technical input, and the problems with
TC/SC equivalence are serious.  even if they can't be fixed 
reasonably with the current DNS protocol, they do have some
bearing on the applicability of IDN.

third, I don't see why you say these are not within the IETF process,
but I find no justification for that statement.  if nothing else,
the spirit of IETF has always been to consider any constructive input,
even if it were not presented in exactly the correct way.

Keith

---

From: Dave Crocker <[email-masked]>
Date: Mon, 18 Mar 2002 10:23:21 -0600

>What Seng fails to do is compare IDNA to the status quo. Sure, the

this has been done many times.  there is no need to repeat just because a 
participant failed to acknowledge or understand it.

d/

----------

---

From: Keith Moore <[email-masked]>
Date: Mon, 18 Mar 2002 11:29:24 -0500

> In other words, the code re-write and upgrde is going to have to happen 
> sometime. 

right.  but it's not in our power to tell people when to do the rewrite
and upgrade.  and insisting that everyone upgrade all applications to
accept IDNs before anyone uses IDNs won't work.  there's too much demand
for IDNs right now, and people are already deploying it.  since we don't
have the luxury of upgrading everything first, we need a solution that
minimizes the harm when IDNs are used with legacy software.

Keith

---

From: [email-masked]
Date: Mon, 18 Mar 2002 06:08:25 -0800 (PST)

> that a small percentage of people use today is a dumb idea.

I have very little sympathy for software that breaks when given unexpected
input. But this is not the concern. The concern is that there is a bunch of
widely used software that prohibits certain inputs. It does so because the
standards in effect at the time the software was written said those inputs are
illegal.

> interoperability.

Here's a quote from a message sent by Carl Gutekunst,
<[email-masked]>, to the IDN list on 10-Feb-2000 that directly
addresses this:

 I conducted some experiments with raw 8-bit data in message header fields a
 year or so ago in response to requests from customers in Japan and China.
 There were (and are) some native Japanese and Chinese language E-mail clients
 that quite happily stuff unencoded Shift-JIS, EUC JIS, Big 5, et al into the
 Subject and the phrase fields (but not the route-addr) of addresses.

 The results were similar to what we saw with Just-Send-8 a decade ago. As long
 as the entire end-to-end path had been carefully managed to specifically
 handle these messages, they would be received and displayed correctly. But if
 any SMTP relay or client wasn't party to the bilateral agreement, all kinds of
 damage happened, including RFC2047 conversions, nonsensical character set
 conversions, seemingly random mangling, 8-bit truncation, and non-deliveries.
 (And I'm *not* including untagged charset mismatches, which would not be an
 issue if we used UTF-8 exclusively.)

Carl later stated that he tested using sendmail 8.9.3, UofW Imapd 4.5, and
Netscape Communicator 4.7. The latter dumped core.

I've done similar tests and reached similar conclusions.

In short, this proposal DOES break things, so interoperability has to be
questioned.

				Ned

---

From: Daniel Senie <[email-masked]>
Date: Mon, 18 Mar 2002 14:26:44 -0500

At 11:29 AM 3/18/02, Keith Moore wrote:
>accept IDNs before anyone uses IDNs won't work.

And there are some places where, frankly, trying to display 
internationalized characters is a non-starter. Some things in the world 
still use ASCII displays or ASCII serial interfaces, including router 
consoles. I'd still prefer to be able to type in some equivalency string 
for a domain name and be able to do traces.

>   there's too much demand

Precisely.

Dan
-----------------------------------------------------------------
Daniel Senie                                        [email-masked]
Amaranth Networks Inc.                    http://www.amaranth.com

---

From: [email-masked]
Date: Mon, 18 Mar 2002 14:36:57 -0500

> class of visual basic programming who know this.

OK.. Thanks for the clarification. The distinction is important, because:

> standards in effect at the time the software was written said those inputs are

And let's remember - currently, if it's *not* ascii and *not* RFC2047-encoded,
it is *illegal* in email headers and *IF* it works, it only does so either
*BY ACCIDENT* or *BY MUTUAL PRIVATE AGREEMENT*.

As such, I have to insist that *any* IDN proposal be required to be backwards
transparent to the current 2821/2822/2045-2049 standards the same way that
MIME/ESMTP was required to be backwards transparent to the then-existing 821/822
infrastructure.

On the other hand, I *will* accept an IDN proposal that looks as ugly
to non-upgraded software as 2047-encoding does to a non-2047 capable MUA.
-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

---

From: Pete Resnick <[email-masked]>
Date: Mon, 18 Mar 2002 14:38:38 -0600

On 3/18/02 at 6:20 AM +0000, D. J. Bernstein wrote:

>``Internationalized domain names are a failure if non-ASCII glyphs 
>don't appear on the screen.'' What kind of idiot would disagree with 
>that?

I'm one of those kind of idiots. I expect that there will be many 
applications that won't put non-US-ASCII glyphs on the screen at the 
get-go. The three criteria for success (for me) are:

1. Applications, if they are properly coded, *can* display non-US-ASCII glyphs.
2. All of the new domain names can appear (in some form) in legacy 
applications without breaking those applications.
2. All applications, legacy or otherwise, are able to resolve new 
domain names. If there exists an domain name (and I'm referring to 
the logical entity, not some particular glyph representation of it) 
that a legacy application can't address, that's a failure.

That is to say, balkanization during the transition is a failure of 
the protocol. Inability to display non-US-ASCII glyphs by legacy 
applications is not a failure.

>Obviously the IDNA authors understand the desire to put non-ASCII 
>glyphs on the screen.

To what is this a response? Has anyone claimed that it is 
*undesireable* to display non-US-ASCII glyphs?

>IDNA-unaware copy-and-paste, etc.---and that's a massive failure to 

I cannot disagree more; this is not a massive failure. Unless of 
course MIME has also been a massive failure to send around binary 
attachments. In the same way, if I use an e-mail client that decodes 
base64 into binary and pipes the output to a program that expects 
US-ASCII input, I've done something broken. If I decode IDNA names 
and pipe them to a program that expects US-ASCII domain names, I have 
again done something broken. And in IDNA, we do MIME one better: The 
undecoded IDNA names are usable, even if they aren't pretty. A base64 
encoded file is near useless to anything without decoding.

>conversions. It explicitly encourages these conversions.

Textual support? 6.1 and 6.4 seem to give lots of reasons not to do 
the conversions.

pr
-- 
Pete Resnick <mailto:[email-masked]>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: Thor Harald Johansen <[email-masked]>
Date: Mon, 18 Mar 2002 23:33:25 +0100 (CET)

> minimizes the harm when IDNs are used with legacy software.

This could be done by alternative means of accessing the sites the IDNs in
question are refering/pointing to. People who want IDNs also have the
responsibility of not forcing people to use these to access their servers.
IDNs in e-mails would pose no real problem, as anyone replying in a legacy
client to an account on an IDN adressed mail erver would see the
encapsulated encoding, and reply to the encoded address.

When he then upgrades his client, he would either change to typing the
addresses unencoded or just continue typing in the encoded IDNs (and the
new client would see this and refrain from doing any conversion).

The hypothetical Norwegian web page 'www.søråsen.no' could have an
alternate representation (a separate domain) of 'www.sorasen.no' or
'www.soeraasen.no' for people with old clients. The domain's owner could
also simply give the encoded address to the users of the site, and the new
clients would decode these. The user would then learn the easier way of
typing, or (if the IDN was in a non-latin alphabet and he would lack the
proper keyboard map to type it) just type in the encoded version.
-- 
Thor Harald Johansen
[email-masked]

---

From: "James Seng" <[email-masked]>
Date: Tue, 19 Mar 2002 10:34:59 +0800

> contributes to the discussion is as much a member as anyone else.

I meant to say they are not subscriber of the wg.

(btw, it is not funny to get 353 bounce mails, sorting them out and manually
replying to each of them.)

> bearing on the applicability of IDN.

It is useful technical input on the first time.
By the 353th time, someone got to ask what else are they contributing?

> even if it were not presented in exactly the correct way.

It is a protest and appeal against the last call. The IETF process specify
the exact process to do so in RFC2026 Section 6.5.

-James Seng

---

From: Keith Moore <[email-masked]>
Date: Mon, 18 Mar 2002 22:05:40 -0500

> By the 353th time, someone got to ask what else are they contributing?

of course once should be sufficient.   
some people don't realize that we don't do voting.

> the exact process to do so in RFC2026 Section 6.5.

perhaps, but this isn't a justification for ignoring valid concerns.

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 19 Mar 2002 03:45:56 -0000

[email-masked] writes:
> As such, I have to insist that *any* IDN proposal be required to be
> backwards transparent to the current 2821/2822/2045-2049 standards

Putting UTF-8 into email headers right now would trigger a serious
Sendmail bug, so of course I don't encourage anyone to do that.

However, the Sendmail bug should be fixed as soon as possible. Do you
disagree with this?

Furthermore, the IETF specifications that allow 7-bit software should be
fixed as soon as possible. Do you disagree with this?

Or do you want these bugs to continue to plague programmers in 10 years?
20 years? 50 years?

---

From: "D. J. Bernstein" <[email-masked]>
Date: 19 Mar 2002 06:47:20 -0000

Robert Elz writes:
> All the work here is for the benefit of other protocols, the DNS
> (which never had a problem with domain names in any char set whatever)
> is being mangled to make life easier for other protocols that really
> should have been fixed years ago.

Exactly.

The IDNA documents claim, in attempting to justify IDNA's archaic 7-bit
character encoding, that 8-bit names would break DNS servers. There is
not a shred of evidence for that claim. See the message from the BIND
company to the IDN mailing list dated Mon, 12 Mar 2001 11:39:29 +1100.

The real problems are the 8-bit bugs in some HTTP clients, some SMTP
clients, and Sendmail. See http://pi.cr.yp.to. DNS is not the issue.

Of course, IDNA demands an upgrade from every client in the world,
including clients that already work fine with UTF-8: the MacOS X version
of Internet Explorer 5.1, for example, and many UNIX programs running
under the UTF-8 xterm. Even worse, we're going to have to go through
_another_ massive upgrade for mailbox names and for every other
internationalization target.

How much pain is the IETF willing to inflict upon the universe before
admitting that 7-bit software is obsolete? Is it really so hard to
understand how much the users will benefit from settling on UTF-8?

---

From: Thor Harald Johansen <[email-masked]>
Date: Tue, 19 Mar 2002 12:06:04 +0100 (CET)

> Or do you want these bugs to continue to plague programmers in 10 years?

I'm having trouble understanding why we're still using these old
protocols. Does a standardized transparent (in the "binary" sense of the
word) mail protocol with built-in support for multi-part messages exist?
Then why isn't it being implemented all over the place?

It would make things a whole lot simpler: no UTF-8 or MIME encoding of
international characters would be nessecary (use Unicode), binary
attachments would take up significantly less space and Unicode IDNs could
easily be supported.
-- 
Thor Harald Johansen