Subject: [idn] Moving forward

From: C C Magnus Gustavsson <[email-masked]>
Date: Mon, 28 May 2001 19:46:52 +0200 (MET DST)

The current discussion is not leading anywere. The work group seemed
to make some progress half a year ago but now it's stuck again.

We should ask ourselves the following questions:

1. Do we require or want a rough consensus?

   There is a significant amount of people who demand ACE in the short
   term and another significant amount of people who demand UTF-8 in
   the long term.

   So if the answer is "no" we can run one part over and go for a ACE
   only or UTF-8 only solution.

   If the answer is "yes" we need to realize that there is only one
   solution that we might reach consensus about: arch-2 or "ACE now -
   UTF-8 eventually".

   It should be obvious by now that we can never agree on anything
   else even if we have this discussion until the End of Time. So, if
   we want consensus we should now move on to discuss the following:

2. Given arch-2, how can we make ACE and UTF-8 co-exist?

3. How is ACE to be phased out in the future?

I think we can agree then that some fallback solution is needed. If
UTF-8 is possible, the software must use it. Otherwise, use ACE. New
software must be able to use UTF-8 to comply to standards.

Then we know that ACE will eventually be phased out as soon as all
systems are upgraded. Then we don't need a deadline; we wouldn't be
able to enforce a deadline anyway.

Do we want consensus? Is this an acceptable way to move forward?

/Magnus (who didn't vote in the straw poll, lacking good alternatives)

---

From: "D. J. Bernstein" <[email-masked]>
Date: 28 May 2001 19:17:02 -0000

> term and another significant amount of people who demand UTF-8 in

I'd like to see a poll on the question of whether UTF-8 will be the
long-term IDN solution. I suspect we have consensus that it will be. The
only person arguing for ACE-forever is Keith ``Q-P forever'' Moore.

It will help us move forward if we have a clear UTF-8 goal in mind. We
can, for example, agree that sendmail's destruction of 8-bit characters
is a bug, and has to be fixed.

The real decision is between ACE-then-UTF-8 and a direct move to UTF-8.
There appear to be more people in this group supporting ACE-then-UTF-8.
However, (1) many people here have been misled by incorrect statements
of the costs of ACE and UTF-8, and (2) UTF-8 appears to be the de facto
standard for working IDN experiments today.

---Dan

---

From: Keith Moore <[email-masked]>
Date: Mon, 28 May 2001 15:58:18 -0400

> We can, for example, agree that sendmail's destruction of 8-bit 
>characters is a bug, and has to be fixed.

or we might agree that Dan's misrepresentations of other people's 
software, and other people's opinions, and operational reality in
general, are bugs which need to be fixed.  I suspect that we would 
not get consensus on this, but that we would get as least as much
agreement to this statement as to Dan's statement above.

Keith

---

From: "Brian W. Spolarich" <[email-masked]>
Date: Mon, 28 May 2001 17:44:29 -0400

| 
| The real decision is between ACE-then-UTF-8 and a direct move to UTF-8.
| There appear to be more people in this group supporting ACE-then-UTF-8.
| However, (1) many people here have been misled by incorrect statements
| of the costs of ACE and UTF-8, and (2) UTF-8 appears to be the de facto
| standard for working IDN experiments today.

  Depends entirely on your definition of 'working'.  From what I've 
seen of the 'just send UTF-8' approaches out there they 1) completely
ignore the normalization issues and 2) completely ignore the applications,
protocols, etc. that are not directly user-facing.

  -bws

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Mon, 28 May 2001 17:26:29 -0700

>only person arguing for ACE-forever is Keith ``Q-P forever'' Moore.

Keith is not the only person arguing for it, and he is not the only 
person supporting it. Many people would rather be poked in the eye 
with a sharp stick than to try to argue with you. As has been seen 
over the years, many people also would rather be silent than agree 
with you because you often turn and attack your supporters.

Thus, the small number of people on the list who are willing to take 
either position doesn't mean that a small number of people have an 
opinion.

The best way to resolve this is for someone who supports the "first 
ACE then UTF-8" position to write an Internet Draft. That person 
needs to be someone who is willing to later edit the draft to include 
other people's opinions, and to revise it in a timely manner when 
people comment on it. The draft needs to cover how the new protocols, 
and the change between them, affects applications, protocols, and 
users. Until there is a draft that gives the arguments in detail, 
there is no reasonable way to compare it to IDNA.

And, yes, Patrik and I need to add more about all of this in the IDNA 
draft. Look for something this week.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "D. J. Bernstein" <[email-masked]>
Date: 29 May 2001 04:25:57 -0000

I suspect that we have consensus that the long-term IDN solution will
encode Unicode characters as UTF-8 on the wire. I request that the WG
chairs run a poll to see whether we have consensus.

If we do, in fact, have consensus on this point, then we can stop
wasting time considering ACE-now-and-forever. We can instead compare
the actual possibilities: (1) ACE-now-UTF-8-later; (2) simply UTF-8.

---Dan

P.S. I'm not going to bother responding to Hoffman's ad-hominem attacks.

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 29 May 2001 04:49:21 +0000

C C Magnus Gustavsson <[email-masked]> wrote:

> we need to realize that there is only one solution that we might reach
> consensus about: arch-2 or "ACE now - UTF-8 eventually".

I agree if you mean that any solution with a chance of fostering
consensus must allow ACE now and must encourage UTF-8 in the future.  I
suspect that we will be even more likely to reach consensus if we allow
UTF-8 to coexist with ACE as soon as possible, and if we don't require a
complete ACE phase-out any time soon.

How about this model:

Just as there are now multiple ways to represent domain names (ASCII,
EBCDIC, ink on paper, ...) and one especially customary way (ASCII),
there will be many ways to represent IDNs (ACE, UTF-8, UTF-16,
iso-2022-jp, koi8, ink on paper, ...) and two especially customary ways
(ACE and UTF-8).

Obviously any existing protocol that defines a hostname as a sequence
of dot-separated LDH labels has no choice but to use ACE.  New versions
of those protocols may opt to support UTF-8 in addition.  New protocols
must support UTF-8 domain names (I think that follows from an existing
IETF policy regarding UTF-8), which implies that they can also handle
ASCII names, including ACE names.

Senders must not send UTF-8 domain names unless they know that the
receiver can handle it (based on the protocol or protocol-version).
Any program that might sometimes send a domain name to a receiver that
cannot handle UTF-8 names must be prepared to perform ACE encoding.
Notice that this does not apply to new protocols, since new protocols
are required to support UTF-8 names.

Programs that send domain names to receivers known to handle UTF-8 names
may perform ACE decoding on ACEs before sending, but are not required or
even encouraged to do so (because maybe someday ACE will be rare and it
would be useless functionality).

Programs that display domain names to users should perform ACE decoding.
Maybe someday, if ACE becomes rare, this could be weakened from "should"
to "may".

DNS servers must accept both UTF-8 and ACE queries (how it accepts
UTF-8 queries is an open question).  Maybe someday DNS will be the only
protocol left that uses ACE.  If/when that happens, we can revisit the
phase-out question.

"D. J. Bernstein" <[email-masked]> wrote:

> characters is a bug

Given that RFC 821 specifies that messages shall be 7-bit, I don't think
it's fair to characterize sendmail's handling of 8-bit characters as a
bug.  We might agree that it should change anyway, but I think that's a
matter for a different working group.

AMC

---

From: Keith Moore <[email-masked]>
Date: Tue, 29 May 2001 02:09:44 -0400

> chairs run a poll to see whether we have consensus.

I doubt that we have consensus that this is "the" long-term solution,
i.e. that UTF-8 will completely replace ACE in finite time.
(If/when it does happen, I expect it will happen because DNS itself 
is replaced...but I don't think we are even close to ready to touch that)

I suspect we are closer to consensus that we can have a way to 
use UTF-8 on the wire in DNS queries and responses.  Though I cannot 
justify it on a purely technical basis due to transition issues (it's 
hard to imagine the circumstances where a UTF-8 query will be more 
reliable than, and as fast as, an ACE query), to me it would be worth 
it to agree to define a UTF-8 query option if it allows us to move forward.

Keith

---

From: "James Seng/Personal" <[email-masked]>
Date: Tue, 29 May 2001 14:05:04 +0800

I would like to hear some support on this before we run this poll.

-James Seng

---

From: Keith Moore <[email-masked]>
Date: Tue, 29 May 2001 02:35:30 -0400

Adam,

I think this framework would be a good basis for moving forward, perhaps
with one exception:

> even encouraged to do so.

I'm not sure this is a good idea for all applications - in particular,
I think it would be a bad idea for email.  Partially because email is more
reliable if MTAs don't do unnecessary transformations on messages, and
partially because in email there's really no way to know the capabilities
of the "ultimate" receiver (there may in fact be multiple receivers).

It might be that more precise wording of your statement would address
this concern, though offhand I cannot suggest text that would do so.

Keith

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 29 May 2001 07:32:16 +0000

Keith Moore <[email-masked]> wrote:

> > Programs that send domain names to receivers known to handle UTF-8
> > names may perform ACE decoding on ACEs before sending, but are not
> > required or even encouraged to do so.
>
> I think it would be a bad idea for email.  Partially because email
> is more reliable if MTAs don't do unnecessary transformations on
> messages, and partially because in email there's really no way to know
> the capabilities of the "ultimate" receiver (there may in fact be
> multiple receivers).

Yes, MTAs are already recommended not to alter messages, and that would
still apply.

> this concern, though offhand I cannot suggest text that would do so.

I guess I should say that ACE decoding is permissible only when the
sender of a protocol message knows that all recipients of that message
(both direct and indirect) can handle it.  In the case of email headers,
I don't see how the sender could know that.  But in the case of SMTP
MAIL and RCPT commands, only the next-hop MTA receives those commands,
so ACE decoding would be safe there.  Probably pointless though.

The reason I included this option in the model was to allow for the
reduction of ACE-leakage.  Example scenario:  After IDN is standardized,
a new application protocol is defined that uses nameprep'd UTF-8.  Since
this application talks only to itself, it doesn't bother implementing
ACE decoding when displaying names to the user (it is recommended, but
not required, to decode ACE); it just handles conversion of UTF-8 to the
local charset.  That works fairly well until someone writes a gateway
between this application and some other protocol.  Now it might be nice
to put ACE decoding in the gateway, since the application doesn't do it
itself.  This is safe because the gateway knows that this application
handles UTF-8.

AMC

---

From: "Bruce Thomson" <[email-masked]>
Date: Tue, 29 May 2001 16:33:55 +0900

> I would like to hear some support on this before we run this poll.

It is likely that we have consensus on this point, and it would be well
worth the effort of a straw poll to establish this, so we can move on.

Bruce Thomson

---

From: John C Klensin <[email-masked]>
Date: Tue, 29 May 2001 05:12:31 -0400

> I would like to hear some support on this before we run this

The third alternative, which seemed to have some support in
Minneapolis and other places, is

ACE and/or UTF-8 now, some sort of search/directory system in
the long term.  We should not let that one get washed out in the
heat of the current discussions (see my tediously long note,
just posted).

    john

---

From: Mats Dufberg <[email-masked]>
Date: Tue, 29 May 2001 12:13:26 +0200 (CEST)

> chairs run a poll to see whether we have consensus.

Independent of IDN scheme it is important that the the solution is
considered as an upgrade of DNS, not a temporary hack, to have it
uniformly implemented.

If IDN is implemented as Nameprep+ACE I do not think that we will see
enough reason to replace ACE with UTF-8 on the wire in a foreseeable
future. Such a change would require that all DNS infrastructure is
upgraded. If UTF-8 is used, the ACE-to-UTF-8 and UTF-8-to-ACE conversions
can easily be handled in the interface to the DNS system.

Today we have three alternatives, as far as I can see:

1. UTF-8 without nameprep
2. UTF-8 with nameprep
3. ACE with nameprep

I do not consider "ACE now, UTF-8 later" an alternative.

Mats

-----------------------------------------------------------------
Mats Dufberg                                     +46-8-545 857 06
[email-masked]                           fax: +46-8-545 857 29

---

From: Keith Moore <[email-masked]>
Date: Tue, 29 May 2001 09:30:03 -0400

> so ACE decoding would be safe there.  Probably pointless though.

probably so, since users don't see this most of the time.  

also there is the potential for utf-8 IDNs to leak out in delivery status
notifications (DSNs) which expect addresses in ASCII.

> reduction of ACE-leakage.  

sure.  but utf-8 leakage is also a concern.  one guy's improvement
is another guy's leakage :)

> Example scenario:  After IDN is standardized,

this would be fine.  but we usually don't specify the behavior of gateways.
there are too many details to track down for marginal cases.

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 29 May 2001 09:30:03 -0400

Keith

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 29 May 2001 09:07:07 -0700

At 4:25 AM +0000 5/29/01, D. J. Bernstein wrote:
>encode Unicode characters as UTF-8 on the wire.

I suspect this is incorrect.

If ACE works for all the IDN strings we want, then there is no reason 
to do the second transition to UTF-8.

If ACE doesn't work for al the IDN strings we want, then it should 
not be deployed at all.

And, for those of you who are supporting the idea of the transition, 
please consider the companies and people who register names that are 
legal in ACE but illegal in UTF-8 because of the 63-octet limit on 
name parts. As has been pointed out here earlier, many ACEs compress 
some scripts better than UTF-8. Thus, if we do "ACE first then 
UTF-8", there will some names that are legal ACE and illegal UTF-8. 
Is this a good policy?

The reverse is also true: there are many names that will will fit in 
63 octets of UTF-8 that cannot fit in 63 octets of any of the ACEs 
proposed to date. Those names will be usable to some people (the ones 
who have made the transition) but not to the people who haven't. How 
is this transition supposed to work?

No transition is needed. Either we do ACE, or we do non-ACE. 
Currently, ACE seems to the be the strongly preferred solution in the 
WG.

--Paul Hoffman, Director

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Tue, 29 May 2001 12:40:27 -0400

Paul,

Just what was the point of having arch-2 in your comparison document?

Eric

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 29 May 2001 10:18:14 -0700

>Just what was the point of having arch-2 in your comparison document?

arch-2 had been proposed in the WG, and the original idea for the 
comparison document was to cover all the proposals. Now that the 
number of proposals is huge, and the fact that many of the 
most-talked-about proposals don't have active Internet Drafts, and 
the fact that I have become a proponent of one of the solutions, and 
the fact that no one commented on the comparision document, I have 
given up on the document. The WG chairs may want to get someone who 
is neutral to keep it alive, but I'm not sure it is still useful.

--Paul Hoffman, Director

---

From: Eric Brunner-Williams in Portland Maine <[email-masked]>
Date: Tue, 29 May 2001 14:35:58 -0400

Paul,

The point isn't authorship of a document, it was the presentation of a
middle that wasn't fictive.

Eric

---

From: "D. J. Bernstein" <[email-masked]>
Date: 29 May 2001 20:07:12 -0000

> to do the second transition to UTF-8.

Consider a programmer writing network software five years from now, or
ten years from now.

He'll be working with text in UTF-8. But wait, you say, he has to
convert to some screwy ACE format on the wire. When he asks why all the
Internet protocols have this strange complication, you mumble something
about ancient versions of Sendmail that didn't support 8-bit text.

So he'll have to deal with ACE, taking time away from implementing new
features that users actually want.

You should be ashamed of yourself.

---Dan

---

From: Keith Moore <[email-masked]>
Date: Tue, 29 May 2001 20:41:28 -0400

> name parts.

That's an interesting point.  However, I expect that we'll need some 
DNS exetnsion to do UTF-8 anyway.  Might such an extension allow us to
do away with the 63 octet limit?

Keith

---

From: Keith Moore <[email-masked]>
Date: Tue, 29 May 2001 20:52:46 -0400

> convert to some screwy ACE format on the wire. 

yes, but there will be a library routine for that.  the programmer
will still have to deal with it, but it's not much worse than 
the effort that would otherwise be required to deal reasonably 
with message headers.

do you really think we'll be using a paper-tape I/O model 10 years 
from now?  even on UNIX?  maybe we will - it's endured a long time -
but I wouldn't bet the farm on it.

> When he asks why all the

no, that's what *you* would say.  the rest of us might mumble something
about the need for backward compatibility with ASCII mail standards,
because we understand that sendmail is not the source of the problem.

Keith

p.s. if and when qmail has been around for 20+ years,  and if it still 
has significant popularity, some young upstart is going to trash your
code for being shortsighted and brain-damaged.  and you are not going
to want to rewrite qmail - because you will have spent too much sweat
on it and fought too many interoperability battles for that code to 
give up on it, and those things that you did to make the code ugly
will seem (even in hindsight) to have been perfectly defensible at the 
time.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 29 May 2001 18:04:00 -0700

>convert to some screwy ACE format on the wire.

It's true.  Practical standards are really a drag.

We should change them every year... no, that won't be quick enough.

Let's say every 3 months, to reflect all of the most modern changes in 
technical practise.

Each 3 months, it will of course be obvious what the new practises, 5 years 
from now, will be.

After all, we all have such a good track record when predicting the future, 
don't we.  In 1992 we were assured that Unicode was instantly going to be 
in mass use, for example.

But wait.  It takes years to adopt and deploy a standard.  That means that 
we will never have any stability, since we will be spending all our time 
revising everything to be infinitely current.

After all, look how much better the computer science purity of X.400 was, 
over the clearly atrocious -- and by the way, I'm not kidding -- 
RFC822/RFC821 suite.

>You should be ashamed of yourself.

Dan, please stop talking to yourself outloud.  Try listening and you might 
learn something.

d/

----------
Dave Crocker   <mailto:[email-masked]>
Brandenburg InternetWorking   <http://www.brandenburg.com>
tel: +1.408.246.8253;   fax: +1.408.273.6464

---

From: "D. J. Bernstein" <[email-masked]>
Date: 30 May 2001 17:18:09 -0000

> but I wouldn't bet the farm on it.

Ever heard of TCP? Byte streams aren't going away.

> code for being shortsighted and brain-damaged.

It has been around for more than 5 years. It hasn't needed any syntax
extensions except to work around bugs in other software (space after
MAIL FROM, for example). If it turns out someday that I haven't planned
ahead far enough, I'll make whatever changes are necessary.

For comparison: In February 1999, when it became clear that the revised
USENET message header field format was going to allow unencoded UTF-8, I
explained to Eric Allman how to make sendmail 8-bit-clean. (USENET
messages are often forwarded through mail by 8-bit-clean gateways.) A
few days ago, he released another security-fix release of sendmail. It
still isn't 8-bit-clean.

---Dan

---

From: "D. J. Bernstein" <[email-masked]>
Date: 30 May 2001 18:42:00 -0000

Dave Crocker writes:
> In 1992 we were assured that Unicode was instantly going to be in mass
> use, for example.

Really? Precise reference and exact quote, please. Who told you that
Unicode was ``instantly'' going to be in mass use? (It's widely used
now, though it isn't universal yet.)

When I say that many people proposed deprecating 7-bit SMTP a decade
ago, I can back up my statements. For example, Andre Pirard said ``I
find Robert Ullmann's proposition excellent. Require that relays pass 8
bits ...'' See http://www.imc.org/ietf-smtp/old-archive/0326.html.

The only objection to that requirement was the claim that 8-bit support
would take a long time to be deployed. Vixie said that he had some
seven-year-old sendmail binaries, for example, and concluded ``with
near-certainty'' that ``any changes to the SMTP spec will take at least
a decade to reach 90% of the critical server population.''

Well, it's now a decade later, and every critical server supports 8 bits
in the message body EVEN THOUGH YOU IETF IDIOTS STILL ALLOW 7-BIT MTAS.
If the SMTP spec had changed ten years ago to require transparent 8-bit
handling in both the header and the body, we wouldn't have sendmail's
UTF-8 failures today.

I DO NOT WANT TO BE FACED WITH 8-BIT PROBLEMS IN ANOTHER TEN YEARS. I am
astounded that anyone would object to requiring 8-bit support. Even in
the case of a horrible piece of code like sendmail, handling 8 bits is a
simple matter of converting dangerous characters to two-byte sequences
on input and doing the opposite conversion on output.

> After all, we all have such a good track record when predicting the
> future, don't we. 

Obviously _you_ don't.

---Dan

---

From: [email-masked]
Date: Wed, 30 May 2001 16:28:34 -0400 (EDT)

My biggest concern is supporting IDN.
Writing code, and upgrading servers are one time costs.
Support is unfortunatly reoccuring

I am happy to upgrade production nameservers,
I am wont mind upgrading production webservers,
I am willing to upgrade production mailservers, pop servers and etc.

I want to be able to tell users that they must upgrade to
Windows XP service pack 20 or MacOS X.1.2.3 or whatever
to use IDN or we wont support them.

I dont want to have to spend 4 hours on a support call
trying to explain what ACE means or explaining that
UTF-8 is a character set when the problem is that their
capslock key is stuck and mistyped their password.

IDNs should only displayed to users who userstand them.

The only way I see to do this is mapping them at the
application layer.

example: (assume ~i18n~ is any IDN name)

~i18n~.com <-mapping-> myi18n.com

Sender
--------
Date: Wed, 30 May 2001 12:42:06 -0700 (PDT)
From: user <user@~i18n~.com>
To: Joe <[email-masked]>
Subject: ...

test

On-The-Wire
--------
Date: Wed, 30 May 2001 12:42:06 -0700 (PDT)
From: user <[email-masked]>
To: Joe <[email-masked]>
Subject: ...

test

Receive with IDN
--------
Date: Wed, 30 May 2001 12:42:06 -0700 (PDT)
From: user <user@~i18n~.com>
To: Joe <[email-masked]>
Subject: ...

test

Receive without IDN
--------
Date: Wed, 30 May 2001 12:42:06 -0700 (PDT)
From: user <[email-masked]>
To: Joe <[email-masked]>
Subject: ...

test

The IDN application could do a lookup to convert myi18n.com to ~i18n~.com
It is possible that locale style support could be built-in so that
myi18n.com is diaplayed as;
ru_RU gets ~i18n~ru~.com
en_US gets myi18n.com
ja_JP gets ~i18n~ja~.com
etc...

All of these would get converted back before being put back on the wire.

I dont know if this is functional or usefull but I am up for anything
that cuts down on support.

-Bill

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 30 May 2001 23:01:43 +0200

--On 01-05-30 18.42 +0000 "D. J. Bernstein" <[email-masked]> wrote:

> EVEN THOUGH YOU IETF IDIOTS STILL ALLOW 7-BIT MTAS

That was pretty constructive, and I guess everyone in this working group
which feel they participate in the IETF now understand exactly how you feel
about them.

At least it was explicit.

Thank's for sharing.

   paf

---

From: "D. J. Bernstein" <[email-masked]>
Date: 30 May 2001 22:49:36 -0000

Keith Moore writes:
> You stubbornly refuse to fix qmail to comply with the standards, 

Wrong as usual. Please read RFC 2821, section 2.4: ``In general, a relay
SMTP SHOULD assume that the message content it has received is valid
and, assuming that the envelope permits doing so, relay it without
inspecting that content.''

You may reasonably argue that message creators today shouldn't use 8-bit
bytes in header fields. But you are being a fool when you claim that
MTAs must, or even should, reject 8-bit bytes.

> Fortunately, most deployed MTAs block 8-bit characters in message 
> headers, which keeps people from depending on them.

Wrong on both counts. Europeans rely heavily on ISO 8859-1 in Subject
lines, for example. The only reason that we're suffering through the
incompatible change from 8859-1 to UTF-8 is that 8859-1 is missing the
characters that billions of people want.

---Dan

---

From: Keith Moore <[email-masked]>
Date: Wed, 30 May 2001 18:08:39 -0400

> Ever heard of TCP? Byte streams aren't going away.

okay, well perhaps I should have asked:

"do you really think we'll be using user interfaces designed for a
paper-tape I/O model 10 years from now?"

> ahead far enough, I'll make whatever changes are necessary.

unlike sendmail, you had the luxury of implementing qmail after the 
mail standards had stabilized, when pretty much everyone was using
DNS with MX records and SMTP over TCP.

> few days ago, he released another security-fix release of sendmail. It

As a matter of fact, message headers are still required to be in ASCII.
An MTA that accepts or relays broken message headers is arguably broken.
Fortunately, most deployed MTAs block 8-bit characters in message 
headers, which keeps people from depending on them. This is a good thing 
because their behavior isn't defined they wouldn't be reliable even if
they were passed cleanly.  So when are you going to fix qmail?

(you don't have to answer that, it's a rhetorical question)

You stubbornly refuse to fix qmail to comply with the standards, 
because you think that you know better than anyone else how things 
should work - which characteristics of software or a protocol are 
bugs and which are features, who should deploy what new servers at
what timeframe, etc.  It's your right and privilege to think that, and 
you are free to implement your own software however you wish.  But you 
shouldn't be surprised to find out that others have different opinions 
about how things should work.  

For a protocol to be interoperable in the real world it must be able to
tolerate some variation in implementation, user interface, and deployment.
People everywhere will not follow your game plan (or anyone else's) just
because you claim it's a good idea.  Different people in different 
situations will inevitably favor different solution paths. 

Keith

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 1 Jun 2001 01:50:14 +0000

I've had some further thoughts regarding that model I proposed, some
of which are similar to thoughts that have been posted in this thread.
First here's a quick recap of the model:

IDNs can be represented in a multitude of ways, but there are two
especially customary representations: ACE and UTF-8.  Old protocols
use ACE on the wire; new protocols use UTF-8 on the wire.  Evolving
protocols need to either stick with ACE or handle the transition
cleanly, perhaps by negotiating capabilities.  DNS must accept both ACE
queries (returning ACE responses) and UTF-8 queries (returning UTF-8
responses).

It's the DNS part that I've been thinking about.  The ACE
queries/responses are obviously needed for backward compatibility, and
they require no changes to the infrastructure.  But how are the UTF-8
queries going to work?

I can think of two approaches:  Either segregate UTF-8 queries into a
new kind of request (using a protocol extension or a new class or new
resource records or whatever), or allow UTF-8 in regular DNS queries.

If UTF-8 queries use a new type of request, then they can be used
only if both the client's resolver and the local DNS server have been
upgraded to support the new query format.  If you're upgrading the
resolver, you might as well include an ACE encoder/decoder, because
ACE queries will always work, even before the local DNS server has
been upgraded.  And once you have an ACE encoder/decoder, why bother
implementing the new query format?  Applications will never know or care
which format the resolver is using.  Even if every other protocol and
application migrates to UTF-8 and forgets how to encode and decode ACE,
the resolvers and DNS servers could still be speaking ACE to each other
and the other protocols/applications would be oblivious.

The other approach, allowing UTF-8 in regular DNS queries, has
advantages and disadvantages.  The advantage is that some existing
applications will work to some extent by accident.  In particular, if an
application uses UTF-8 natively, and its resolver is 8-bit clean, and
its DNS servers are 8-bit clean, and the UTF-8 encoding of an IDN fits
in 63 bytes (which won't always happen--UTF-8 is about 50% larger than
ACE for some scripts), then a lookup of the IDN will work by accident.
Of course, if this application talks to other applications that don't
use UTF-8 natively, there may be other problems, but let's focus on
DNS for the moment.  One disadvantage of this approach is an increased
opportunity for spoofing:  If some applications accidentally send
UTF-8 in DNS queries, then surely other applications will accidentally
send other 8-bit encodings, some of which are coincidentally valid
UTF-8.  But it could be argued that any existing application that does a
lookup on an 8-bit domain name is asking for trouble, and gets what it
deserves, except that UTF-8 applications will be pardoned (sometimes).

Does anyone find this accidentally-works phenomenon to be a compelling
reason to allow UTF-8 in regular DNS queries?

It looks to me like allowing UTF-8 in regular DNS queries may be more
trouble than it's worth, and creating a new query format just for UTF-8
is certainly more trouble than it's worth.  If someday there is a
pressing need for a new query format for some other reason, we might as
well make it use UTF-8, but UTF-8 itself is not a sufficient reason to
introduce a new format now.

Please remember that even if DNS sticks with ACE for now, that doesn't
impede any other application/protocol from using UTF-8.  It's quite
orthogonal.

AMC

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 31 May 2001 20:19:13 -0700

"Adam M. Costello" wrote:

> implementing the new query format?  Applications will never know or
> care which format the resolver is using.  Even if every other protocol
> and application migrates to UTF-8 and forgets how to encode and decode
> ACE, the resolvers and DNS servers could still be speaking ACE to each
> other and the other protocols/applications would be oblivious.

This is missing a component, which is the resolver-application interface.
Applications store/display names that they are given by DNS, so DNS must
provide them with the data in the appropriate format. Although the
on-the-wire protocol can be ACE as you suggest, the resolver still needs a
way to remember the desired disposition format of the query results.

For example, assume that an ESMTP extension for UTF8 in Received headers
comes to pass, and everybody agrees that displaying the headers in UTF8 is
beneficial to the user community, blah blah blah. When the MTA issues a
reverse lookup for the domain name associated with the IP addr, it doesn't
matter what format DNS uses, but the local resolver still needs to know
that the answer data should be returned in UTF8.

I suppose that it's technically feasible to ask resolvers to maintain
state, but the latest round of proposals have been to use UTF8 as the
default encoding on the wire so that it's preserved end-to-end, with the
expectation that UTF8 will be the preferred default encoding for many
years to come.

> trouble than it's worth,

Yes

> and creating a new query format just for UTF-8 is certainly more
> trouble than it's worth.

No. The EDNS message is already defined and is required/used for things
like A6 and DNAME already. This would just be another usage of that
message. It also allows us to resolve the sizing problems.

-- 
Eric A. Hall                                        http://www.ehsco.com/
Internet Core Protocols          http://www.oreilly.com/catalog/coreprot/