Subject: Re: [idn] WG last call summary

From: "D. J. Bernstein" <[email-masked]>
Date: 19 Mar 2002 00:29:29 -0000

Paul Robinson writes:
> Something *should* be done, but your argument has a hint of
> 'I never want anything done, ever' about it, which is putting people off.

I have put a huge amount of effort into evaluating the costs of various
IDN proposals. Please read http://cr.yp.to/proto/idnc3.html before you
make any more comments about what I'm trying to do.

I have been trying to fix obsolete 7-bit IETF specifications for years.
I have been advocating for a year that the IDN WG _immediately_ declare
7-bit software to be obsolete. See the DRUMS archives for more examples.

I have a detailed web page, http://pi.cr.yp.to, tracking 8-bit bugs in a
variety of programs. I've spent quite a bit of time talking to other
implementors about this. It has been more than three years since I
suggested an easy Sendmail fix to Eric Allman.

You may be aware that I'm the author of the mail software with the
Internet's largest increase in SMTP-server deployment over the past 18
months (reaching #2 in total deployment, behind only Sendmail), and the
DNS software with the Internet's largest increase in domain-name
deployment over the past 18 months (reaching #2 in total deployment,
behind only BIND). My programs relay 8-bit data without trouble, even
though certain people obviously don't understand why this is important.

I want internationalized domain names (and mailbox names and so on) to
work. But I can't support the IDNA proposal; IDNA is a disaster. I also
can't accept having _any_ protocol move forward over so many objections.
Even the strongest desire to _do something_ is less important than the
requirement to obtain consensus for any change.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: Paul Robinson <[email-masked]>
Date: Tue, 19 Mar 2002 20:44:07 +0000

> make any more comments about what I'm trying to do.

I stand by my original statement. Something should be done, but your 
document make you look like a typical whiner - you point out all the 
problems, but offer no solutions to some of the problems you raise. The 
solutions you do offer will take at least 4 years IMHO to be effective, and 
your phase 2 will actually INTRODUCE more problems, as well as allowing the 
continuation of other problems you raise.

> though certain people obviously don't understand why this is important.

I know who you are. I know your software. I have spent many hours looking at
your code. It's why I'm suspicious of anything you ever write or say. Don't
get me wrong, I'm not lacking in respect for some of the things you've done,
but I think you're shooting yourself in the foot with the way you'er going
about this. I don't want to have an argument with you, but you're
approaching this from the wrong angle. All very wrong. It's what you do, and
it makes sense to you, but shouting at everybody THAT THEY'RE ALL WRONG AND
IT MUST ALL STOP RIGHT NOW OR ELSE THE WORLD WILL END AND MAIL WILL BREAK is
not the way to get IDNA fixed. Perhaps expanding out IDNC3 to something more
than 'these pieces of software need to be 8-bit clean and then we start
registration' might be somewhere to head next...
 
> requirement to obtain consensus for any change.

Don't you understand that to get what you want, all Unix boxen have to be 
*completely* 8-bit OK, all MS boxen have to be *completely* 8-bit OK and a 
good 20% of sites out there will just have to shut down ops permanently? How 
are you proposing to display alpha-ol.com on a VT100?

Had you not considered (and excuse me if I'm being dumb here), that just 
modifying a DNS resolver to understnad PunyCode might just be a nice short 
cut? That perhaps, just maybe, software that recognises e-mail addresses as 
being any continuous text with @ in them, and URLs as beginning http:// is 
going to be able to just throw a string down to a resolver that knows that 
it's doing? And that maybe replacing the DNS resolver on all the machine s 
out there to be able to do lookups with PunyCode might be a TAD more 
realistic than trying to get EVERYTHING, EVERYWHERE to be good with 8-bit?

I must be missing something here... I've *got* to be missing something 
here... please, somebody tell me what it is...

-- 
Paul Robinson

---

From: "D. J. Bernstein" <[email-masked]>
Date: 20 Mar 2002 04:32:50 -0000

Paul Robinson writes:
> You tell him that although it's gobbledygook to people without greek
> alphabet support, it will still work. It's not convenient, but it WILL
> work. Guaranteed.

False. IDNA does _not_ work. IDNA causes interoperability failures. Mail
will bounce, for example, in situations where ASCII domain names would
have worked fine. IDNA coauthor Adam Costello has admitted this.

> And that maybe replacing the DNS resolver on all the machines out
> there to be able to do lookups with PunyCode might be a TAD more 

Here you are assuming that the only problem is the DNS resolver---that
the conversion between the local character encoding and the IDNA
character encoding can be handled entirely by the DNS resolver.

That assumption is false. Consider, for example, an MTA configured to
accept mail for pi.cr.yp.to, with a Greek pi. The MTA compares the
incoming domain name to pi.cr.yp.to. That doesn't involve the resolver.

People who say that IDN is purely a DNS issue are confused.

> Making every piece of software and display device that might ever have
> to deal with IDNs capable of handling UTF-8?

Here you're being simultaneously inconsistent and shortsighted. Fixing
bad displays is part of the cost of IDNs. In the context of UTF-8, you
agree with me that this is a cost; in the context of IDNA, you ignore
the cost completely.

In fact, the cost of fixing UTF-8 displays is much _smaller_ than the
cost of fixing IDNA displays. UTF-8 has been around for many years, has
built up incredible momentum (as illustrated by RFC 2277), and already
works in a huge number of programs.

The extra programs hurt by IDNA aren't just UTF-8-aware clients. Fixing
the IDNA display failures also means changing web servers, mail servers,
DNS servers, etc., so that the sysadmin can put a properly displayed IDN
into his server configuration files. Think about the above pi.cr.yp.to
example again.

> The solutions you do offer will take at least 4 years IMHO to be
> effective

Let's suppose 4 years is right, and let's compare the results to IDNA
after 4 years.

IDNC3 requires 8-bit fixes to some widely deployed programs, certainly.
But IDNA needs _much larger_ changes in _many more_ programs. So, after
the same 4 years, only a fraction of the IDNA work will be done. IDNA
will still have an incredible number of display failures, plus the
interoperability failures and all the other IDNA problems.

Even worse, IDNA doesn't do _anything_ to fix the other half of the
email problem. Do you seriously believe that Chinese users will be
satisfied with email addresses where the domain part can contain Chinese
characters but the box part is still required to be ASCII? It's obvious
how to fix this with UTF-8; how, pray tell, do we fix it with IDNA?

I presume that you're not one of the 7-bit-forever crackpots. How do you
propose migrating from IDNA to UTF-8? This is much more costly than
moving directly to UTF-8, because it needs a compatibility period during
which everyone supports two different encodings of the same character.
Doesn't it bother you that the IDNA documents don't discuss this at all?

What makes your position particularly shameful is the fact that people
proposed requiring 8-bit transparency _eleven years ago_. If it hadn't
been for Paul Vixie et al. making your ``it'll take years!'' argument
back then, we would have had 8-bit transparency today. Do you want to be
facing the same stupid bugs in another eleven years?

> Until then, he has to get a 'normal' domain to see himself over.

Correct. Your example Greek user has an ASCII domain name that's always
displayed with an ASCII d instead of the truly desirable Greek delta. 

Now, please explain why the same user should prefer a domain name that's
_occasionally_ displayed with the desired delta but _usually_ displayed
as incomprehensible gobbledygook.

Your answer, of course, will be something like this: ``The gobbledygook
is a temporary problem. In twenty years, after the massive IDNA upgrade
is complete, everyone will see a delta!''

In short, you're looking at the long-term IDNA benefits (never mind the
interoperability failures and all the other problems) but refusing to
look at the long-term UTF-8 benefits. Inconsistent once again.

> Something should be done, but your document make you look like a
> typical whiner - you point out all the problems, but offer no
> solutions to some of the problems you raise.

False. http://cr.yp.to/proto/idnc3.html explains how IDNC3 offers
solutions to every one of the IDNA problems that it points out:

   * interoperability failures;
   * inconsistent displays of the same name;
   * unnecessary implementation and deployment costs;
   * multiple semantically similar names;
   * identical displays of different names; and
   * typing failures.

Each solution is listed right next to the problem, so I can't imagine
how you missed this.

> What you are proposing IS introducing an interoperability failure,

False. Every step in http://cr.yp.to/proto/idnc3.html preserves
interoperability.

> How are you proposing to display alpha-ol.com on a VT100?

There are several options. One option is to work around the hardware
limitations in software, displaying something like

          |
   /\/ /\ |   /^ /\ /\ /\
   \/\ \/ | * \_ \/ | | |

Another, much more popular, option is to move your email reading, web
browsing, etc. from your 1970s-vintage VT100 to a graphics terminal.
Have you considered the VT340, for example? Or an IBM PC, model 5150?

> a good 20% of sites out there will just have to shut down ops permanently

Get a grip, Paul.

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 20 Mar 2002 14:32:41 +0859 ()

D. J. Bernstein;

> False. IDNA does _not_ work. IDNA causes interoperability failures.

IDNA does _not_ work, because Unicode does not work in International
context.

> People who say that IDN is purely a DNS issue are confused.

It's purely a cultural issue.

> works in a huge number of programs.

In international context, it is technically impossible to properly
display Unicode characters.

There is no implementation exist.

While some implementations work in some localized context, local
character set serves better for the context.

						Masataka Ohta

---

From: Robert Elz <[email-masked]>
Date: Wed, 20 Mar 2002 14:21:12 +0700

Date:        Wed, 20 Mar 2002 14:32:41 +0859 ()
    From:        Masataka Ohta <[email-masked]>
    Message-ID:  <[email-masked]>

  | IDNA does _not_ work, because Unicode does not work in International
  | context.

This argument is bogus, and always has been.   If (and where) unicode
is defective, the right thing to do is to fix unicode.

That is, it isn't the principle of a single encoding of all characters
that anyone is objecting to here, it is that some specific characters
have been implemented incorrectly ("merged" with others) as I understand it.

I'm not competent to decide how important this problem is, and this is
not the forum to debate it anyway (so please don't reply just to tell me
how significant the problem is, nor why).   Do that with whoever maintains
unicode.

If you can't get enough of the unicode experts to agree that there's a
problem that needs fixing, then by definition, there isn't.  That's just
the same way as the IETF works (whether the unicode group actually work
this way or not - if not, they should...) - that a few people believe
something is broken is irrelevant if they can't demonstrate that well
enough to sway others to agree with them.

So, stop arguing against unicode (10646) - just fix any problems it has.

kre

---

From: Masataka Ohta <[email-masked]>
Date: Thu, 21 Mar 2002 00:57:18 +0859 ()

Kre;

> is defective, the right thing to do is to fix unicode.

Unicode is not usable in international context.

There is no unicode implementaion work in international context.

Unicode is usable in some local context.

There is some unicode implementaion work in local contexts.

However, the context information must be supplied out of band.

And, the out of band information is equivalent to "charset"
information, regardless of whether you call it "charset" or
not.

> So, stop arguing against unicode (10646) - just fix any problems it has.

Fix is to supply context information out of band to specify which
Unicode-based local character set to use.

With MIME, it is doable by using different charset names for
different local character set.

See, for example, RFC1815.

As for IDN, it can't just say "use charset of utf-7" or "use charset
of utf-8".

IDN can say "for Japanese, use charset of utf-7-japanese".

Or, if you insist not to distinguish different local character sets by
MIME charset names, IDN can say "use charset of utf-7, but, for
Japanese, use Japan local version of utf-7" and somehow specify
how a name is used for Japanese.

Anyway, with the fix, there is no reason to prefer Unicode-based
local character sets, which is not widely used today, than existing
local character sets already used world wide.

						Masataka Ohta

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 20 Mar 2002 12:29:10 -0600

dropping a few mailing lists off the CC:

> Unicode is usable in some local context.

Agreed. Note that "some" is changing to "many" as time goes on.
> However, the context information must be supplied out of band.

Agreed.
> information, regardless of whether you call it "charset" or

Do not agree.
for most values of what we currently have registered as "charset",  it is 
not sufficient to identify the context.

Therefore, depending on "charset" to identify context is not only useless, 
but actively harmful.
My opinion, which I stated in RFC 1766, and have found no reason to change.
The IETF rough consensus seemed to agree at the time.

                 Harald

---

From: Paul Robinson <[email-masked]>
Date: Wed, 20 Mar 2002 21:22:05 +0000

> False. IDNA does _not_ work. IDNA causes interoperability failures. Mail

... with the current DNS resolvers in place...

OK, others have pointed out failures with things like SSL/HTTPS (which is 
broken in several interesting way anyway from the point of view of 
scalability and long-term usefullness), but even then I can see work 
arounds. All of them are more preferable to throwing away every serial 
console in the world because a greek delta doesn't display properly on it 
and therefore has a 'display failure'.

> incoming domain name to pi.cr.yp.to. That doesn't involve the resolver.

Well, funnily enough I have to tell my MTA about .co.uk domains AND .com 
domains as well, even though the bit in front of them is IDENTICAL. If I 
register the .net I have to tell it about that as well. They're different 
domains. If I buy an IDN with a Greek letter in it, I'm going to have to 
tell my MTA about that. If think that by adding a new domain to your config 
you shouldn't have to tell your MTA about that, you're obviously not a very 
experienced admin.
 
> as incomprehensible gobbledygook.

Until he fixes his machine to have proper Greek alphabet support. However, 
when it comes over to my PDP-8 it appears as gobbledygook. But it still 
works. Sure, that's REAL broken.
 
> look at the long-term UTF-8 benefits. Inconsistent once again.

The IETF is about the long-term. It not about the next 2 years. Even I know 
that. You should certainly know that.
 
>    * interoperability failures;

That don't exist. Or the ones that do can be given work-around far simpler 
than replacing every piece of equipment in th eknown universe and updating 
every piece of software ever written. Ever.

>    * inconsistent displays of the same name;

That don't matter.

>    * unnecessary implementation and deployment costs;

The fact you're suggesting that what *I'm* proposing and what the IDNA is 
proposing requires unnecessary implementation and deployment costs, quite 
frankly, makes me snort my coffee up my nose.

>    * multiple semantically similar names;

OK, let's not allow uppercase Greek Alpha. Does that fix one of your 
problems? Sheesh.

>    * identical displays of different names; and

Not a problem for the IETF. Not even a problem for users. In Greece A means 
Alpha. In the US it means 'Ay'. If you're a Greek in New York, you might 
have problems. You're going to have problems anyway, but at least with IDNA 
PunyCode is your friend.

>    * typing failures.

We get those anyway. It's why slahdot.org exists.
 
> False. Every step in http://cr.yp.to/proto/idnc3.html preserves

Fixing half a dozen pieces of software and then just allowing every mand and 
his dog to register special Bernsteinised-domains? Right. I see.
 
> limitations in software, displaying something like

*ROFL*

# cat Mail/inbox | utf8decode | figlet

That's how I like to read my mail in the morning!

Seriosuly man, I'm sat here on a customer's site in the middle of the South 
Atlantic looking forward to an 18 hour RAF flight home to the UK, and I'm 
crying with laughter. You really have cheered me up. That was so 
off-the-wall and unexpected - to propose IDNs to be displayed as ASCII art - 
that I'm going to be smiling and giggling to myself for days. You have 
confirmed what I should have known - you're stark raving insane. Quite, 
quite, quite mad. Fantastic.
 
> Have you considered the VT340, for example? Or an IBM PC, model 5150?

You're proposing the abondonment of legacy hardware when there is no need. 
That costs money. Serious money. If I have my hostname setup as 
pi.alpha-ol.com and I need to gain access over the serial port at boot with 
a VT100, the option of anything else is not available. 
 
> > a good 20% of sites out there will just have to shut down ops permanently

I'm not the one proposing we turn the world into one great big ASCII art 
factory. 

Anyway, I'm going to be away from mail for a few days due to travelling 
around - don't think I'm giving up on you yet though. :-)

Seriously... ascii art... hehehe...

-- 
Paul Robinson

---

From: Masataka Ohta <[email-masked]>
Date: Thu, 21 Mar 2002 06:37:15 +0859 ()

Harald;

> Agreed. Note that "some" is changing to "many" as time goes on.

Irreleant, because some contexts are not compatible.

The point here is that there can be no universal context.

> > However, the context information must be supplied out of band.

Then, how can you provide the information with IDN?

> not sufficient to identify the context.

That is a problem of current and past charset reviewers including you.

> Therefore, depending on "charset" to identify context is not only useless, 

Agreed. ISO 2022 escape sequence is the way to go.

> My opinion, which I stated in RFC 1766, and have found no reason to change.

It was already denied by real world examples.

						Masataka Ohta

---

From: George Michaelson <[email-masked]>
Date: Thu, 21 Mar 2002 08:39:35 +1000

So is teardown on a non-IETF day scheduled for 4am or 5am?

No, really: How long can the itnerant ex-attendee sit in the lounge and
be on line? Or, how long at the Britpub with the feed?

	-George

PS I have been dying to ask: does anybody measure the size of the suspected
   pool of locals who get slashdotted into driving 40ks for free connectivity?

   not that you'd want to stop this mind, but I do wonder if some of the
   bodies are actually *only* here for the 802.11

---

From: Masataka Ohta <[email-masked]>
Date: Thu, 21 Mar 2002 16:47:58 +0859 ()

Mark;

As my mailer says;

	[Charset utf-8 unsupported, skipping...]

you should learn that Unicode is not usable in international context
of IETF.

> ...
> It would not be worth replying to these threadworn and repeated
> assertions by Mr. Ohta, except that some members of this list may not
> be that familiar with Unicode. Clearly Unicode is being used
> successfully in a huge variety of products in international contexts.

That is not so totally untrue. There do exist some Unicode products
valuable in international contexts, because they are useful in some (or
many) local contexts.

However, as Harald and I agreed, such products need local
context information OOB to work properly.

Then, the problem is that some local contexts are mutually exclusive.

There is no instance of Unicode-based-local-character-set
internationally useful.

Some product using unicode may be usable in multiple local context,
if the context information is given OOB, and the product is valuable
in international market.

However, unicode, still, is not usable in international context unless
OOB information, which, thanks to poor charset reviewing, essentaily is
yet another charset designaiton, is attached.

						Masataka Ohta

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 21 Mar 2002 09:51:56 -0600

> context information OOB to work properly.

apologies for indicating too strong an agreement....
we agree that for many functions that users would like software to perform 
(sorting being an obvious example), context information is needed to work 
properly.
> There is no instance of Unicode-based-local-character-set

I assert that for many functions (such as non-high-quality text display) in 
many locales, having information in Unicode-based character sets are useful 
without OOB information. Thus, Unicode is internationally useful - but we 
are deluding ourselves if we claim that it solves ALL problems.

> yet another charset designaiton, is attached.

We disagree on this point.
I assert that the fact that charset does not give the required information 
is by DESIGN of the charset mechanism, not by "poor review".
As a citizen of a continent where > 10 nations use the same character set 
(ISO 8859-1), but where locale requirements vary across the continent, I 
believe that separation of those concerns was a correct engineering 
decision.

Just to make it clearer where Ohta and I agree, and where we disagree.....

                           Harald

---

From: "Mark Davis" <[email-masked]>
Date: Wed, 20 Mar 2002 19:18:14 -0800

...

For more information on the CJK repertoire (which is the part Mr. Ohta
objects to), see http://www.unicode.org/unicode/faq/han_cjk.html.

Mark
—————

Γνῶθι σαυτόν — Θαλῆς
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

----- Original Message -----
From: "Masataka Ohta" <[email-masked]>
To: "Robert Elz" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; <[email-masked]>;
<[email-masked]>
Sent: Wednesday, March 20, 2002 07:58
Subject: [idn] Re: I don't want to be facing 8-bit bugs in 2013

---

From: tedd <[email-masked]>
Date: Thu, 21 Mar 2002 10:17:26 -0500

>>  Go sell a Greek user an ``internationalized domain name'' with a delta,
>>  Pete. Then tell him that most of his correspondents will see the delta
>>  as incomprehensible gobbledygook rather than a delta. See what he says.

Okay, I'm not Greek, but as the registered owner of .com, I'll tell you*.

I expect that some windoze users will display it (.com) as a box dot
com; mac users will display it as a delta dot com; and everyone else
will display it as a bq--eida.com (RACE), or dq--uuyg.com (DUDE), or
zq---h9g.com (AMZ_C), or punycode, or whatever -- AND, they will
continue to do so until they have the appropriate software on their
machines to handle the conversion. That's what I expect.

On the other hand, I do not expect that the IDN will eventually
produce something that will transparently produce the appropriate
glyphs on millions of users machines. In my mind, that's simply not
possible because I think it requires that every machine on this
planet have a complete copy of UNICODE installed and available.

Granted, I am clueless about a lot of this IDN stuff and I may be
"hubris" (as Mr. Paul Hoffman once informed me) in my offering my
opinion to the "Gods of the Internet", but I do think I see the
distinction between which side of the modem one is on and what
responsibility lies in each camp.

>OK, scenario 1:
>alphabet support, it will still work. It's not convenient, but it WILL work.
>Guaranteed. For his business colleagues and friends in Greece, who DO have
>the latest and greatest software, it will display as a delta. His ISP hasn't
>had to upgrade, and everybody in the world can use his domain - eventually
>they will see it as a delta as well, but for now they see it as an encoded
>string they can still use no problem.
><-snip>
>If you were IT director of a large firm, and you had a choice as to which to
>roll out, which would you choose?
>
>--
>Paul Robinson

I would vote for scenario 1. Besides, IMHO (in my hubris opinion)
that's the only way it's going to work.

tedd

* The Greek version of "delta" registers a internal error -- it's the
Math language set that provides an acceptable "delta".
-- 
http://sperling.com

---

From: Robert Elz <[email-masked]>
Date: Fri, 22 Mar 2002 00:39:56 +0700

Date:        Thu, 21 Mar 2002 00:57:18 +0859 ()
    From:        Masataka Ohta <[email-masked]>
    Message-ID:  <[email-masked]>

Otha-san

  | Anyway, with the fix, there is no reason to prefer Unicode-based
  | local character sets, which is not widely used today, than existing
  | local character sets already used world wide.

Let's assume that local char sets, and an explicit indication of which
to use is adequate for this purpose (as Harald has said, and I agree,
that's not sufficient for all purposes, but for stuff like domain names,
file names, etc, I suspect it is).

Then, let's take all the MIME names of those charsets, and number them,
0, 1, 2, 3, ... (as many as it takes), that's a 1::1 mapping, after all
the mime charset names are in ascii, we wouldn't want to keep that.

Then, rather than labelling whole names with a charset, we'll label
every individual character, so if, by some strange chance, ascii (or 8859-1)
happened to be allocated number 0, then 'A' would be 0-65.
This way we can mix and match names with characters from any random
character set (it may be a little less efficient than one label per name
but that's OK, assume we'll pay that price for the flexibility).

No, we'll just list all the possible characters in all the char sets,
with their char set numbers attached, in one big table.

What we have at that point is (essentially) 10646 - unicode.   Just up
above you said this works (throwing in some redundant labelling cannot
cause it to stop working, nor can altering the labels from ascii to binary
numerals).

Of course, a large fraction of the char sets that we just listed in a big
table contain ascii as a subset,so we end up with dozens of versions of
A (it's in 8859-1 8859-2 ... tis-620 (Thai) ...).  All of those A's are
in all respects identical, keeping them will do no more than cause
interoperability problems.   So let's go through and squash all the duplicates.
and then renumber everything to be a bit more compact (the renumbering is
a 1::1 operation that alters nothing important).

Having done that, we have exactly 10646 (or can have, if we picked the
right character sets for our initial big list of everything, gave them
the appropriate char set numbers, and compressed the number spaces in
just the right way).

Again, you said above, this works.

The only place in all of this where there's any possibility of a problem is
in the "squash all the duplicates" - if some characters that are duplicates
aren't removed, or some that aren't are treated as if they are.  If this
happened, then there'd be a bug in the actual character tables (too many, or
too few) - but this doesn't alter the principle of the thing.

If there is a bug like this (which I am not able to judge) then someone
should get it fixed.

Whether or not there is a bug, the unicode/10646 approach is clearly the most
flexible way, and is perfectly adequate for, labelling things using
whatever characters anyone wants to use - internationally or locally.

There is simply no way to rationally claim that a local char set, plus label,
is adequate and unicode is not.

kre