Subject: [idn] Some comments

From: Dan <[email-masked]>
Date: Sat, 13 Jan 2001 15:07:59 +0100 (MET)

Having no time to daily comment on the flow of comments on the list I
here give some comments on things I have read this week:

- There was some comments on 8-bit text i Subject and how it quite
often was mangled beyond recognition.
This does show some of the problems that an ACE can result in.
To get non-ASCII into the Subject-line you today need a quite
complex handling where some parts are encoded and some are not, 
length constraints on the encoded form and a mixture of character sets.
It is even worse with the From:-line. Here only some parts may be
encoded. I have written software to handle the MIME-encoding of header
lines. If all header lines had been in UTF-8 (or UCS-2) it would have been
much, much simpler and much less error prone.

- There have been several references to a "Latin alphabet". As there are
many alphabets that can be called a "Latin alphabet" (for example
the classical Latin alphabet (no G,J or W), the English alphabet or
the Swedish alphabet). While I can guess it is mostly a reference to
the 26 letters of the English alphabet, the English alphabet is not the only
Latin alphabet. Please be more clear.

- There have been talk about URIs and how they will use non-ASCII and
if Unix users are willing to edit ACE or use a new tool.
As a person needing more than ASCII (currently using ISO 8859-1) typing
in URLs using %-escapes or editing domain names in ACE is unacceptable.
If I edit domain names in a zone file to be loaded by my DNS server,
I will edit them using my local character set. I will expect my DNS server
to convert them into whatever format is used in the DNS protocol.
If I can get an editor which can support ISO 8859-1 and UTF-8 is a
friendly way, I might accept to edit my zone files using UTF-8.
And when I enter URLs I enter them using ISO 8859-1. I am not going to
%-encode them. And I cannot really see any reason to have them
%-encoded when being transmitted using HTTP. HTTP is fully capable to
send 8-bit characters. It is OK for them to be converted into UTF-8 for
transmission, but any displyed to me must not be in UTF-8 or %-encoding,
if the letters can be displyed using my local character set.

In short: during user interaction all data should use the local character set.
As a local character set cannot always display all characters in a text,
there should be ONE standard way to encode and display the characters not
supported by the local character set. I do not want many like today
with: %-encoding, ACE, quoted-printable or base64.
For every new encodning for it gets more and more complex to handle text.
Just look at an URL:  Is the host part in ACE-encoding or %-encoding?
Or UTF-8? If a URL is embedded in text, what format is it in then?
The same character set as the text? Or is it in ACE? How is the tool
that is displaying the text to know that an URL is embedded and
decode just that part so that also the URL can be displayed
in a user friendly manner?

   Dan

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 13 Jan 2001 19:20:34 +0100

>This does show some of the problems that an ACE can result in.

The errors for 8bit in email can be classified in three groups:

- Information loss
- Charset weirdness
- Ascii encoding is not decoded

The main problem is the information loss or charset weirdness 
(unknown charset for 8bit data) that happens when people send mail 
wrongly tagged/encoded. This is the major problem just because the 
receiver can not know reconstruct the mangled data regardless of what 
his software is doing.

Non-decoded data is not as problematic, because then information has 
not been lost during encoding and transport.

An ACE encoding is in the category of "non-decoded" data, and not 
"information loss", while UTF-8 on the wire is in the category of 
information loss -- if it is mangled.

>length constraints on the encoded form and a mixture of character sets.

There are length constraints on what you send over the wire. What I 
see above is a layering violation. You have to keep things apart, and 
remember that with the introduction of MIME, SMTP is used as a 
transport mechanism, and whatever you want to send with the help of 
SMTP have to be encoded correctly.

>much, much simpler and much less error prone.

It all depends on what you mean by "simpler". Of course an n:m 
mapping is harder than n:1 mapping (i.e. if every email had to be in 
Unicode, people only had to know how to map between the local 
characters and Unicode).

That is a different thing though than the question whether ACE or 
UTF-8 should be used.

Keep the layers separated please!

>if the letters can be displyed using my local character set.

Just too many things are mixed up here to have this paragraph make sense:

  - You edit the zone file using your local charset
  - The content of the zonefile have to be converted to Unicode
  - The content of the zonefile have to be nameprepped
  - The content of the zonefile have to be encoded
  - You enter URL's in your local charset in your browser
  - The browser convert into Unicode
  - The browser do nameprep
  - The nameprepped data might be %-encoded for HTTP
  - The domain part have to be encoded

Note that only where we have "have to be encoded" above, we have the 
problem of UTF-8 versus ACE.

Now, you Dan, also think that nameprep/equivalence can be done in the 
DNS server, so the DNS query packets can include whatever octets one 
want. My position is that that is just out of scope, and just not 
acceptable solution as the domainname have to be "reused" in the 
actual protocol aswell, in the example above HTTP. So, NOT doing the 
full nameprep/encoding in the client is just wrong, and a mistake 
architecturally.

Also, for the n:time, for UTF-8 to work we need to change a large 
number of protocols, intermediaries/middleware boxes, firewalls etc 
because the change to the protocols is quite large. Not as bad as a 
change to USC-2 or some other "more" binary encoding, but still a 
large change.

With an ACE encoding, we don't have to change the protocols at all.

Do people understand this?

Also, I saw someone saying "the rest of us like UTF-8" (or something 
like that). I would say that there is definitly not consensus for use 
of UTF-8 instead of an ACE. The contrary.

>In short: during user interaction all data should use the local character set.

Yes.

>in a user friendly manner?

All good questions, and information on why the IDN problem is not 
only a problem with DNS, but also a problem with the various 
applications which uses domains.

    paf

---

From: "D. J. Bernstein" <[email-masked]>
Date: 13 Jan 2001 21:37:27 -0000

> of UTF-8 instead of an ACE. The contrary.

What I actually said was ``The rest of us, the UTF-8 people and the
ACE-now-UTF-8-later people, all agree that some programs are going to
have to be fixed.''

What's your position, Patrik? Do you want to look around in a future
UTF-8 world and see that the Internet is using some clumsy ACE? Do you
think that Sendmail will never have to be fixed? Do you object to the
IDN WG warning implementors that programs should be 8-bit-clean?

> not been lost during encoding and transport.

That's what Keith said for years about Quoted-Printable. He ignored the
screams of the users who, in fact, DID NOT HAVE THE INFORMATION.

(Keith's favorite airline: ``Yes, sir, we know we shredded your luggage,
but it's all here! Take some time and sew it back together. By the way,
we don't shred English luggage.'')

The goal is not merely to ensure that, in theory, all the information
has been preserved. The goal is to actually _provide_ the information to
the user.

> because the change to the protocols is quite large.

What matters is the software. The protocols can change for free if the
software already supports the new protocols.

UTF-8, with nameprep handled close to the user as I described, works
with many existing 8-bit-clean pieces of software. ACE-now-and-forever
and ACE-now-UTF-8-later would impose much larger software upgrade costs.

---Dan

---

From: Patrik Fältström <[email-masked]>
Date: Sat, 13 Jan 2001 23:19:58 +0100

>IDN WG warning implementors that programs should be 8-bit-clean?

My position is that:

   - A solution to the IDN problem have to be solved _now_,
     and we don't have time to update all protocols to use
     anything else than US-ASCII before that point in time
   - This means that the immediate solution must be ACE
     based

In the longer term if course a more efficient encoding of Unicode can 
be used, but that is much further ahead.

I would say that the warning message should not be to be 8-bit-clean, 
but that protocols (and therefore applications) should be prepared 
that not only the version of Unicode will change, but also the 
prefered encoding mechanism. I don't want 8-bit clean protocols, and 
UTF-8. I want protocols which can handle UCS-2, UCS-4 or UTF-16 (or 
whatever non-encoded version of Unicode). Why have an encoding at all 
if we are taking the step of changing the protocols?

What I don't like in your arguments is the focus on "applications" 
instead of looking at what is specified in the protocols.

>screams of the users who, in fact, DID NOT HAVE THE INFORMATION.

I don't understand what you say. Note that I had several different 
problems, and that the problem "non-decoded-data" was when the 
recipient did get all the information in the form of a charset 
argument to the text/plain content type, while the other problems 
where when you didn't have this information.

My point was that when we have non-encoded data, the recipient could 
by changing the application see all the information without any loss, 
but he could not if there were some loss somewhere (8th bit removed 
or charset argument missing to 8bit clean data).

What you seem to talk about was how bad it was when the user had an 
application which could not decode the quoted-printable given the 
encoding was correct and charset parameter was correct?

If that is the case, then you don't understand what I talk about.

>we don't shred English luggage.'')

You talk about quoted-printable encoding and charset parameter as it 
was something which destroys the content as shredding of luggage 
would do.

Sorry, that doesn't work.

Shredding the luggage is when the user can not reassemble the luggage 
by using the correct tools. Such things like getting 8bit data 
without charset parameter, or getting 8bit data where the 8th bit is 
set to zero.

>has been preserved. The goal is to actually _provide_ the information to

My argument was that IF the whole encoded message, as with quoted 
printable and a correct charset parameter is coming to the recipient 
of the email, then he can himself by changing application get the 
information without noticing that the data have been encoded at all.

>and ACE-now-UTF-8-later would impose much larger software upgrade costs.

But, software doesn't handle 8 bit stuff correctly today even though 
we have in the IETF been talking about it for a very long time.

Even worse, software doesn't handle the charset parameter correctly 
either (mapping between transport charset and local charset) even 
though MIME has now existed for quite some number of years.

Software is not at all as good as you seem to belive.

A couple of questions:

  - Do you use an operating system which doesn't use ISO 8859-1 as
    native charset for input and output? What charset?
  - What is the minimal set of characters you need
    in your native language, and what is that language?
  - What percent of email you receive and send each day is not english,
    i.e. need characters which doesn't exist in the A-Z set?

Note that I am talking about your own personal experience and not 
what your customers/users of your software report.

    paf

---

From: "D. J. Bernstein" <[email-masked]>
Date: 13 Jan 2001 23:57:01 -0000

Patrik writes:
> I don't want 8-bit clean protocols, and UTF-8. I want protocols which
> can handle UCS-2, UCS-4 or UTF-16

UTF-8 is compatible with ASCII. UCS-4 is not.

Switching a protocol to UTF-8 preserves compatibility; ASCII data is
unaffected. Switching a protocol to UCS-4 destroys compatibility; ASCII
bytes turn into 4-byte sequences.

Do you seriously believe that the Internet is going to move to UCS-4
rather than UTF-8? Exactly what benefits is anyone supposed to see in
this? Occasionally I hear people claiming that a string of Unicode
numbers is convenient because the displayed width of the string is
proportional to the number of bytes; but they're wrong, thanks to
combining characters and double-width characters.

> instead of looking at what is specified in the protocols.

What I don't like in your arguments is the focus on protocol specs as
religious objects, rather than as tools to help implementors provide
working software to system administrators and users.

> was something which destroys the content as shredding of luggage 

``The luggage isn't destroyed, sir. It's just shredded. As I said, you
can take some time and sew it back together. You haven't lost anything;
it's all here! By the way, would you like to buy a sewing machine?''

> we have in the IETF been talking about it for a very long time.

The IETF should have required 8-bit-clean mail software in 1982. Allman
would have fixed his software eventually, certainly before version 6.57
in 1993. UTF-8 header fields should have been allowed in 1996. Everyone
would have been happily using UTF-8 mail in 2001.

Instead, the IETF mail standards _still_ allow MTAs to drop 8-bit bytes,
even in message bodies. See http://cr.yp.to/docs/8bit/06.txt.

---Dan

---

From: Keith Moore <[email-masked]>
Date: Sat, 13 Jan 2001 20:41:55 -0500

> working software to system administrators and users.

and apparently your personal definition of "working" is one that 
requires nearly everyone on the network to upgrade their software
(and perhaps even hardware), at your whim, and on your schedule - 
or risk breaking critical functionality not only for themselves but 
for others as well.

it's a rather strange use of the word.

Keith

---

From: Patrik Fältström <[email-masked]>
Date: Sun, 14 Jan 2001 09:23:51 +0100

>UTF-8 is compatible with ASCII. UCS-4 is not.

Depends all on how you define "compatible", and no, you don't have to 
explain because I know what you are thinking of. We don't agree here. 
It's as simple as that.

>bytes turn into 4-byte sequences.

And?

Was it not a long-term, "correct", solution you wanted?

>combining characters and double-width characters.

If nothing else because of political reasons. The grade of opposition 
to UTF-8 is about the same as the number of bits you need to 
represend words in the language the person opposing the language 
speaks.

I see a big risk the solution will _NOT_ be Unicode if UTF-8 is 
pushed, and that just because of the way a character is encoded. Only 
for political reasons.

>working software to system administrators and users.

Ok.

>it's all here! By the way, would you like to buy a sewing machine?''

Sigh...

>would have been happily using UTF-8 mail in 2001.

People would not have been happy at all. See above.

>even in message bodies. See http://cr.yp.to/docs/8bit/06.txt.

But, you say that the standards is only an aid for implementors. How 
come still I get broken email even though I have a sewing machine?

You didn't respond to my questions regarding what language etc you use, Dan.

   paf

---

From: Dave Crocker <[email-masked]>
Date: Sun, 14 Jan 2001 22:32:44 -0800

Folks,

We went been through this paradigm debate almost 10 years ago.

It was MIME vs. "just do 8-bit SMTP".

MIME was the right answer.  "just do 8-bit SMTP" was not.

The ACE approach is the same paradigm as MIME.

It is the right way to lay an encoding enhancement on an installed base, 
with the smallest amount of disruption.  It permits incremental adoption, 
with immediate benefit to participants in the enhancement, and no 
critical-path changes required of the infrastructure.  Changes to the 
infrastructure can then happen later and more liesurely.

d/

At 09:23 AM 1/14/2001 +0100, Patrik Fältström wrote:
>explain because I know what you are thinking of. We don't agree here. It's 
>as simple as that.
>

=-=-=-=-=
Dave Crocker  <[email-masked]>
Brandenburg Consulting  <www.brandenburg.com>
Tel: +1.408.246.8253,  Fax: +1.408.273.6464

---

From: William Morris <[email-masked]>
Date: Mon, 15 Jan 2001 02:23:59 -0500 (EST)

> The ACE approach is the same paradigm as MIME.

10 years ago, 7-bit with 1 bit parity network connections were 
more common. I dont think that is the case any longer.

MIME was not just a solution for 8-bit data, it also solved the
issue of multipart data.

> infrastructure can then happen later and more liesurely.

I think the end solution should be unicode (utf/ucs/whatever)
ACE is not part of a migration, it is an alternative.
If ACE laid the groundwork for moving to unicode I would
be much more inclined to accept it for the short term.

Infrastructure changes will only happen if there is a need for them.
ACE will meet the developers and consumers needs, so changes will
not happen if it is implemented. 

-Bill

---

From: Dave Crocker <[email-masked]>
Date: Sun, 14 Jan 2001 23:55:48 -0800

>more common. I dont think that is the case any longer.

You missed my point.

The paradigm pertains to making changes with an installed base.

>MIME was not just a solution for 8-bit data, it also solved the

Again, that is not relevant to the question of transition paradigm for an 
installed base.

>ACE is not part of a migration, it is an alternative.

no, it is not.  it is a transfer encoding of unicode.

>not happen if it is implemented.

The "pure" mode will take longer, yes.  That is different from never.

Transition of a large installed base requires attention to maintenance of 
current operation.  That's a necessary expense.

d/

=-=-=-=-=

---

From: "D. J. Bernstein" <[email-masked]>
Date: 15 Jan 2001 08:42:30 -0000

Sorry to bother you with facts, Dave, but Sendmail has done just-send-8
by default for 8-bit non-MIME messages since version 6.57 in 1993. This
blatantly violates RFC 1652 and the 7-bit restriction in RFC 821.

As for MIME messages, current versions of Sendmail can convert 8-bit to
Quoted-Printable, but there's a popular ``8 flag'' that turns this off,
again in violation of RFC 1652 and RFC 821.

Of course, qmail has always done just-send-8 for all messages. It also
deliberately announces 8BITMIME to subvert Sendmail's Quoted-Printable
conversion. Yes, this violates RFC 1652, but it's exactly the same as
the behavior of many Sendmail installations. (It's what the users want!)

Similarly, Exim does just-send-8 for all messages. It announces 8BITMIME
if you set the accept_8bitmime option.

---Dan

---

From: [email-masked]
Date: Mon, 15 Jan 2001 07:38:03 -0700 (PDT)

> more common. I dont think that is the case any longer.

Actually, while this is perhaps true it is largely irrelevant. The issue 10
years ago wasn't what network connections supported, since IP running over such
things did what was necessary to implement an 8-bit path even back then.

What was relevant is that a significant amount of deployed messaging software
reacted badly to 8-bit data in messages. And although many of the problems with
8-bit data handling were fixed in many software packages around that time, it
took many years for the updated versions of that software to actually deploy.
And when it did deploy, it was largely for other reasons, such as to address
security issues.

It seems like a bad idea to assume that another wave of software
vulnerabilities will once again spur deployment of revised sofware when we need
it...

> MIME was not just a solution for 8-bit data, it also solved the

And labelling of data and labelling of charsets. Although it sure would  have
been nice if we'd had a universal charset to specify back then -- most of the
problems I see these days have to do with missing or improper labelling, not
encoding, and quite a bit of this could have been avoided by having a universal
charset. And now that we do have such a thing, we're again having considerable
trouble getting meaningful support for it deployed in a timely fashion. This
does not bode well for IDN done initially as infrastructure enhancements.

> ACE is not part of a migration, it is an alternative.

Not necessarily. ACE is a first step, and I believe it is a necessary first
step. The question of whether or not it is the last step isn't settled at all.
But one of the advantages of ACE is that we don't have to make this choice in
the short term.

> be much more inclined to accept it for the short term.

Um, careful with your terms here. I think you meant UTF-8 or UTF-16 or some
other charset, not Unicode. ACE does encode the Unicode CCS.

> Infrastructure changes will only happen if there is a need for them.

Actually, they often don't happen even if there's a need for them. What's
hnecessary is for there to be a pressing need that cannot be ignored. And it
isn't clear that IDN represents such a need in many quarters, the demand for it
in others notwithstanding.

> not happen if it is implemented.

A very telling statement, if I may say so... and one that argues for ACE
being the last step!

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Mon, 15 Jan 2001 09:11:17 -0800

>blatantly violates RFC 1652 and the 7-bit restriction in RFC 821.

"Just do 8-bit" doesn't and didn't work.  Those who forced 8-bit onto the 
operational framework for all their recipients caused quite a bit of trouble.

That was why an ESMTP option was created, so the receiving SMTP server 
could tell the sending server that 8-bit was supported.

d/

=-=-=-=-=

---

From: GIM Gyeongseog-KIM Kyongsok <[email-masked]>
Date: Tue, 16 Jan 2001 21:01:25 +0900 (KST)

> ACE-now-and-forever
> ACE-now-UTF-8-later
> UTF-8-now-and-forever

  since there seems to be no consensus as to the popularity of these,
can we vote?   
  of course, the populairty should not necessarily be the basis 
for adoption of one of these within IDN WG.

김 경석, 부산대 정보 컴퓨터 공학부; 
  KIM Kyongsok/GIM Gyeongseog, Busan National Univ.
  [email-masked], http://HANGEUL.cs.pusan.ac.kr/hangeul/
  Ph: (051) 510-2292, Fax: (051) 515-2208

---

From: "James Seng/Personal" <[email-masked]>
Date: Tue, 16 Jan 2001 20:33:58 +0800

This is a good idea but not in this sense.

At the last San Diego meeting, there is a strawpoll on the Protocol
Recommendation on "ACE on Application now but do not reject the possibility of
longer term solution in future".

It is important to note that "longer-term solution" does not imply it is
"UTF-8" or any solution in particular.

It is only a strawpoll so a consensus on the WG (ie. this mailing list) is
still needed.

Comments please.

-James Seng