Subject: Re: [idn] We are quibbling about WHAT?

From: Dave Crocker <[email-masked]>
Date: Thu, 02 Aug 2001 10:36:03 -0400

At 08:13 PM 7/29/2001, Eric Brunner-Williams in Portland Maine wrote:
>I think the encapsulate vs extend debate is about architecture. It
>just happens that 63 bytes does focus some attention on effective
>characters per mechanism, and encapsulation schemes using only 5 bits
>plus a nickle per octet are at a disadvantage over extension schemes
>that use at least 7 bits per octet.

1.  My impression is that, in fact, expressions of concern have more often 
had, really, to do with encoding efficiency than anything else.  Or, at 
least, the arguments with the most substance.  And to continue the 
analytical quibble, my impression is that the character sets that are more 
restricted with UTF-8 *AND* with ACE have more information density that 
such un-dense stuff as ASCII.

2.  The name of "architecture" is a very double-edged sword.  Architecture 
that worries only about computer science purity, for example, often winds 
up being highly impractical.  It does not attend to a sufficient range of 
real-world requirements/constraints.

That was why I went through the transition discussion, separately.  The 
difference in transition ease, risk, etc., between ACE and UTF-8 is exactly 
the reason that the horrible technology, called MIME, was used for 
introducing binary, segmented, labeled data to email content.

>The transition impact discussion can't be refuted. It was the subject
>of RFC 801, and others. I'm particularly fond of lines 51 - 64 of
>that transition plan by the way.
>Time to choose an ACE and move on? I'll pass thanks. A half-dozen
>browser-hack marketeers, a trio of monopolies, and the DNSO food
>fight, is inadequate motivation to solve a complex problem. A good
>solution is, and differences in that are why we are where we are.

Having not yet seen any response to the analysis I offered, I beginning 
whether folks really are not interested in attending to transition pragmatics.

Where is the detailed response, rather than just rejection or 
acceptance?  Please skip the personalization of this.

Just as the tendency in the IETF is to say that the primary -- and maybe 
the only -- design concern is scaling, I've come to believe that the only 
concern when upgrading an installed base is the transition ease.

d/

----------
Dave Crocker  <mailto:[email-masked]>
Brandenburg InternetWorking  <http://www.brandenburg.com>
tel +1.408.246.8253;  fax +1.408.273.6464

---

From: Dave Crocker <[email-masked]>
Date: Fri, 03 Aug 2001 13:52:31 -0400

At 11:55 AM 7/30/2001, David Hopwood wrote:
>The debate between ACE and UTF-8 has nothing to do with encoding
>efficiency. The benefit of UTF-8 is in having DNS names in the same
>encoding as the surrounding text, rather than having to treat them as a
>special case.

"surrounding text"???  you mean like this message?

sorry, no.  DNS strings occur in many situations.  Most do not, today, use 
UTF-8,

As to predictions of the future, well, thety are always wrong.  (No doubt 
including this one.)

d/

----------

---

From: David Hopwood <[email-masked]>
Date: Thu, 09 Aug 2001 02:13:49 +0100

-----BEGIN PGP SIGNED MESSAGE-----

Dave Crocker wrote:
> Content-Type: text/plain; charset=us-ascii; format=flowed
> ...
>

Yes, like that message: the domain name "www.brandenburg.com" and the
email address "[email-masked]" were in the same encoding as the
surrounding text. That's why they looked right when I viewed the message,
rather than looking like gibberish.

But, you may say, that encoding wasn't UTF-8. Doesn't matter. Suppose for
the sake of argument that the message had contained characters outside the
US-ASCII repertoire (in a domain name or elsewhere). In that case, your
mail client would have automatically chosen an encoding that is capable
of representing those characters. That still might not be UTF-8, but the
important point is that a text message or file can always be losslessly
transcoded into any UTF, including UTF-8 [*]. So, any domain names in
the message (regardless of whether they are pasted, typed, come from a
.signature file, or whatever), will both "look right", and will be
losslessly convertable to the format needed for a lookup. They can also
be pasted into other applications, and will still look right - even in
applications that don't and shouldn't need to know anything about IDNs.

[*] The transcoder may or may not be NFC-normalizing; that doesn't matter
    because nameprep(NFC(X)) = NFC(nameprep(X)) = nameprep(X) for all X.

Although most of the discussion on this list has been categorising the
possible architectures as "UTF-8-only", "UTF-8 + ACE", and "ACE-only",
this misses the point slightly, and I should have been clearer about
that in my previous message. Call a domain name representation within
some surrounding text (or other protocol elements) "transparent", if the
representation is of a character string such that each character in the
string stands for the same character in the domain name. I.e.:

                charset
  octet string  <----->  character string     =     domain name

as opposed to:

                charset                      ACE
  octet string  <----->  character string  <----->  domain name

A more fundamental categorisation of IDN architectures is then
"transparent-only" vs. "transparent + ACE" vs. "ACE-only".

The representation of LDH domain names has always been transparent: even
if you transcode a message containing domain names into, say, EBCDIC,
each character still stands for itself in the EBCDIC encoding.
Similarly, in proposals that support transparent non-ASCII domain
names, there will be no difficulty with transcoding a message into
UTF-16, UTF-EBCDIC, or anything else that is capable of representing
all the characters in that message.

[I use EBCDIC and UTF-EBCDIC as examples, to make the point that it's not
the fact that the encoding is a superset of US-ASCII that is relied on;
only that it is capable of representing all the characters in the name
(up to Unicode compatibility equivalence), and all the characters in the
surrounding message text. Of course, using an encoding that is a superset
of US-ASCII is very convenient in practice, and a UTF will always be able
to represent the necessary characters - hence the attractiveness of UTF-8.)

The thesis behind the ACE-only proposals seems to be that most existing
protocols, formats, and implementations assume that domain names only
use US-ASCII characters, and so using a non-ASCII representation will
break them. In the vast majority of cases, that simply isn't true: most
things don't assume that the representation is ASCII, but they *do* very
often assume that it is transparent. The main flaw in the argument that
ACE is just another encoding, is that it ignores how important
transparency is as a simplification.

> sorry, no.  DNS strings occur in many situations.  Most do not, today,
> use UTF-8,

I've been searching through all RFCs to try to produce a list of all
the IETF-standardized protocols where domain names are encoded (including
email addresses and URIs/URLs), and actually quite a large proportion of
them use UTF-8 already for the rest of the protocol. There are very, very
few protocols that require use of a different 8-bit character set.
Overall, I don't see anything that would interfere with straightforwardly
adopting UTF-8 as the preferred encoding for domain names, with ACE just
used for a handful of problematic cases such as SMTP.

In the case of protocols where messages are tagged with their charsets,
receiving software might have to transcode a message from whatever it is
tagged as into a UTF before further processing. However, that is what a
Unicode-based program would normally do anyway, and after that, the entire
message (or file, or whatever) can be treated uniformly. ACE domain names,
OTOH, will always have to be treated as a special case. So transparent
solutions have an advantage over ACE-only solutions even when the encoding
is not initially UTF-8.

If anything, maximum name lengths are a bigger compatibility problem than
encodings, and in that respect there is little difference between UTF-8
and the proposed ACEs (some languages win, some lose, but either UTF-8 or
AMC-ACE-Z would be adequate in practice).

> As to predictions of the future, well, they are always wrong.

That's obvious nonsense. It is a perfectly safe prediction that the
default encoding (often the only encoding) used in almost all Internet
protocols in the near future will be UTF-8. In a sense there is nothing
to predict; the decision has already been made in RFC 2277 / BCP 18, and
the process of making the necessary modifications to existing standards
is already well underway.

- -- 
David Hopwood <[email-masked]>

Home page & PGP public key: http://www.users.zetnet.co.uk/hopwood/
RSA 2048-bit; fingerprint 71 8E A6 23 0E D3 4C E5  0F 69 8C D4 FA 66 15 01
Nothing in this message is intended to be legally binding. If I revoke a
public key but refuse to specify why, it is because the private key has been
seized under the Regulation of Investigatory Powers Act; see www.fipr.org/rip

---

From: "Adam M. Costello" <[email-masked]>
Date: Thu, 9 Aug 2001 02:09:12 +0000

David Hopwood <[email-masked]> wrote:

> The thesis behind the ACE-only proposals

What ACE-only proposals?  I think the strongest ACE supporters
merely say that you have to obey existing standards.  If a protocol
specification (like the URI spec, or the SMTP spec, or the TLS spec)
says that the domain name in a particular slot may contain only certain
characters, then ACE is the only way you can use IDNs while still
obeying the standard.

But new standards can of course specify looser rules.  I don't think
anyone here wants to prohibit new protocols/interfaces from using UTF-8
(or their favorite charset).

> most things don't assume that the representation is ASCII

As Dan Bernstein's recent tests have shown, lots of software stops
working when domain names contain bytes greater than 127, including
browsers, resolvers, mail user agents, and mail transfer agents.

And one thing's for sure:  No deployed software today knows how to
properly compare two IDNs (which requires nameprep).

Even if you're right about most things not breaking on 8-bit domain
names, things are connected to each other.  Domain names get passed
around.  If you feed an 8-bit domain name to an IDN-unaware agent, it
might not break that agent, but it's probably going to break something
else a little later.  Therefore 8-bit domain names ought to be passed
around only among IDN-aware agents/protocols, and converted to ACE when
interfacing with IDN-unaware agents/protocols.

AMC

---

From: "D. J. Bernstein" <[email-masked]>
Date: 9 Aug 2001 06:47:12 -0000

> browsers, resolvers, mail user agents, and mail transfer agents.

Lots of software _already_ works with UTF-8 IDNs, including browsers,
resolvers, mail user agents, and mail transfer agents. A small number of
easily fixed bugs account for all the reported failures.

UTF-8 is way ahead of ACE.

---Dan

---

From: Dave Crocker <[email-masked]>
Date: Thu, 09 Aug 2001 10:31:19 +0100

At 02:13 AM 8/9/2001, David Hopwood wrote:
>But, you may say, that encoding wasn't UTF-8. Doesn't matter.

well, actually it does.

>Suppose for
>US-ASCII repertoire (in a domain name or elsewhere).

If the text switches from one character set to another, then no, it is 
unlikely the processing software will handle that well.  If you mean that 
the text is all in the UTF-8, then, no, it isn't.  Mine is in ASCII.  Hence 
your suggestion means we would have mixed Ascii, UTF-8.

The real point is that you appear to be assuming a much more simplified and 
consistent processing environment for characters, across applications, than 
actually exists.

>Call a domain name representation within
>string stands for the same character in the domain name.

By that definition, UTF-8 is not transparent.  It is an encoding scheme, as 
is ACE.

>break them. he vast majority of cases, that simply isn't true

The premise has been present in the design of Internet protocols from the 
beginning.  Much has changed since then, but not everything.

The premise that all software will correctly process UTF-8 has been 
demonstrated to be false.

>protocols in the near future will be UTF-8.

I do not know how many such confident predictions of the future you have 
suffered through, but many of us have watched them made repeatedly over the 
last 25 years and, I am sorry to say, they are usually incorrect.  Not 
always, no.  But frequently.

>  In a sense there is nothing

You should review the history of "decisions" that were made around the 
world, by businesses and governments, for many other technical choices, 
such as ADA, OSI, CMIP, ATM, ...

Nothing is over until it is over.  No decision is firm until it is in the past.

d/

----------

---

From: Dave Crocker <[email-masked]>
Date: Thu, 09 Aug 2001 10:34:22 +0100

>UTF-8 is way ahead of ACE.

Fortunately, the IETF has a history of choosing to be less cavalier about 
global impact of transition processes.

d/

----------

---

From: David Hopwood <[email-masked]>
Date: Fri, 10 Aug 2001 00:11:13 +0100

-----BEGIN PGP SIGNED MESSAGE-----

Dave Crocker wrote:
> >Suppose for the sake of argument that the message had contained characters
> >outside the US-ASCII repertoire (in a domain name or elsewhere).
> 
> If the text switches from one character set to another,

When did I say anything about switching charsets within a text? The example
you chose didn't support your argument because the message only contained
characters from the US-ASCII repertoire (and so it obviously wouldn't show
up differences between any IDN proposals). I'm talking about a message that
includes some abstract characters with Unicode code points > 0x7F.

> then no, it is unlikely the processing software will handle that well.
> If you mean that the text is all in the UTF-8, then, no, it isn't.

If I had meant that, I would have said that.

> Mine is in ASCII. Hence your suggestion means we would have mixed Ascii, UTF-8.

Mixed ASCII and UTF-8 is identical to UTF-8, which is one reason why it would
have been much clearer for you to use a different example.

More generally, the sending MUA can choose any single encoding for a MIME text
part, but a properly internationalized MUA should always choose an encoding
that is able to represent all the characters that are used in the text. [It's
possible to use multiparts to switch charsets in a somewhat klunky fashion,
but practically no user agents actually do that. If a compliant receiving MUA
did come across such a message, it would interpret each part according to its
tagged charset, though, so there would be no problem.]

For example, if the message contains only characters from the ISO-8859-1
repertoire, then a compliant MUA might send it as "text/plain; charset=iso-8859-1".
In that case a transparent representation means that any domain names, email
addresses, URIs, etc. will also be ISO-8859-1. The receiving internationalized
MUA transcodes the message into a UTF, so that if the user clicks on a link or
cuts/copies text from the message, it will be encoded correctly. In no case
does a MUA (or any other application) have to do any additional work in order
to display transparently represented domain names properly.

> consistent processing environment for characters, across applications, than

I'm assuming that users that want internationalization support will use
software that provides that support. All the proposals assume that.

> By that definition, UTF-8 is not transparent.

Again you're missing the point. Transparency is a property of "a domain name
representation within some surrounding text"; it's not dependent on which
charset is used to represent that text. For any charset X, a domain name
encoded using X is transparent if it appears in a message or protocol that
also uses X. That property never applies to a name encoded as ACE, because
ACE will never be used as a general-purpose charset.

[...]
> >break them. In the vast majority of cases, that simply isn't true
> 

My argument doesn't depend on all software being able to correctly process
UTF-8. No software correctly processes ACE at the moment. Remember that
the point of IDNs is to be able to enter and display non-ASCII characters
correctly; if that doesn't work, then a more meaningful ASCII alias for the
domain might as well have been used instead.

> always, no.  But frequently.

Congratulations; you've gone from "always" to "usually", then "frequently",
all in one paragraph. "frequently" is much better, although I would say
"sometimes". There is also a reporting bias to be taken into account: cases
where the prediction was wrong are more interesting, so they get cited more
often. Confident predictions of the near future that turn out to be absolutely
correct are rarely commented on.

In any case, my statement stands: "It is a perfectly safe prediction that
the default encoding (often the only encoding) used in almost all Internet
protocols in the near future will be UTF-8." Note that the effect of BCP 18
is to block progression on the standards track of new protocols and protocol
updates that don't allow UTF-8, unless an exemption is granted.

- --