Subject: Re: [idn] I-D ACTION:draft-ietf-idn-uri-00.txt

From: "D. J. Bernstein" <[email-masked]>
Date: 11 Jan 2001 01:07:41 -0000

Adam M. Costello writes:
> UTF-8 gets less than 6 bits per octet on average for the non-ASCII
> octets, and the ACEs generally use base-32, which is 5 bits per
> character, so that's not much different.

ACEs also include several bytes to distinguish themselves from existing
ASCII names.

If you subscribe to my publicfile mailing list, you'll receive email
asking you to confirm the subscription. The Reply-To in the email will
look like this:

publicfile-sc.942884323.agghmppopdgakhmfhbaf-amc=[email-masked]

Now, what happens if your software rejects mail for usernames longer
than 64 bytes? publicfile-sc.942884323.agghmppopdgakhmfhbaf-amc=.xxx is
already 53 bytes, and the number is a timestamp that will soon increase
by another digit, so you have only 10 bytes for the rest of the name.

If you wanted an IDN, what would it be? Would it fit into 10 bytes with
ACE, not counting the .com?

> ASCII names are allowed to use mixed case.  It would be unfair to deny
> the same flexibility to non-ASCII names.

I'd be happy to get rid of uppercase ASCII too. However, that would
require a transition period. In contrast, we aren't saddled with
existing uppercase IDNs.

---Dan

---

From: "Maynard Kang" <[email-masked]>
Date: Thu, 11 Jan 2001 13:36:56 +0800

Hi Dan,

> ACE, not counting the .com?

The 10 byte constraint you are referring to is entirely an *application*
problem rather than an *application layer* problem.

If you use UTF-8 (or any 8-bit CES) in e-mail addresses, SMTP servers which
adhere strictly to RFC 821 will blow up due to the US-ASCII restriction. If
you use ACE, perhaps only your mailing list software will blow up. I think
there are more systems which deal with SMTP than with mailing lists.

The basic premise is this: any solution to an internetworking problem should
first deal with layers as low as possible before moving upwards, as problems
at lower layers tend to affect a greater number of systems.

maynard

---

From: "D. J. Bernstein" <[email-masked]>
Date: 11 Jan 2001 07:59:38 -0000

The general question here is ``What will happen if we start using IDNs
before we do the software upgrades necessary to make IDNs work?''

UTF-8 IDNs will cause trouble in this case. The point I'm making is that
ACE IDNs will cause trouble too.

UTF-8 IDNs will lead to additional non-821-compliant addresses in SMTP
and additional non-822-compliant addresses in message header fields. The
resulting 8-bit addresses are known to cause trouble for some programs.

ACE IDNs will lead to additional non-821-compliant addresses in SMTP and
additional non-822-compliant addresses in message header fields. The
resulting long addresses are known to cause trouble for some programs.

Maynard Kang writes:
> you use ACE, perhaps only your mailing list software will blow up.

My software doesn't have any trouble. Please learn to read. What matters
is that it effectively lowers any limits that your software might place
on address length.

> I think there are more systems which deal with SMTP than with mailing lists.

ezmlm and its variants send a huge amount of mail every day. Keep in
mind that (1) qmail is running on a very active 10% of the Internet's
SMTP servers and (2) automatic bounce handling, via these long return
addresses, is one of the top ten reasons that people install qmail.

> problems at lower layers tend to affect a greater number of systems

It would help to skip the ``layer'' commentary and focus on the effects.
The effects are what users care about.

---Dan

---

From: "Maynard Kang" <[email-masked]>
Date: Thu, 11 Jan 2001 20:21:53 +0800

> resulting long addresses are known to cause trouble for some programs.

Right, I agree both solutions are potential troublemakers. However I still
think that the negative implications of using UTF-8 IDNs will be more
far-reaching than using ACE IDNs.

Long addresses appear more tolerable than 8-bit addresses; as 821 speaks of
"to the maximum possible, implementation techniques which impose no limits
on the length of these objects should be used" in spite of stating a 64-byte
limit on local-parts and 64-byte limit on domain.

(An interesting side note: the latest incarnation of
draft-ietf-drums-smtpupd seems to provide for a new 255-byte limit on domain
but still keeps the 64-byte limit on local-parts. Maybe that should be
changed too. Anyway, this is a drums issue)

822 imposes no limits, I believe. There is adequate provision for the
folding of long header fields.

> is that it effectively lowers any limits that your software might place

My software doesn't either. I think we both mean "your" in a collective
sense, right?

> addresses, is one of the top ten reasons that people install qmail.

How about servers which are running sendmail and will fall over and die when
they see UTF-8? According to their site, Sendmail "handles an estimated 75
percent of the Internet's email traffic".

Actually I switched to qmail for a different reason, but anyway...

maynard

---

From: Keith Moore <[email-masked]>
Date: Thu, 11 Jan 2001 09:46:17 -0500

> ACE IDNs will cause trouble too.

I doubt that anyone disagrees with you on that, but the presumption is
that ACE IDNs will cause less trouble than UTF-8 IDNs.  

Every UTF-8 IDN (except for IDNs that consist of only ASCII characters)
will cause trouble for a significant number of UAs and MTAs.  

But (assuming careful choice of the ACE) few ACE IDNs are likely to be 
significantly longer than UTF-8 IDNs, and the number of MTAs and UAs
that have trouble with long addresses is significantly less than the 
number of MTAs and UAs that have trouble with  8-bit characters.
Indeed, one of the complaints that has been frequently made about UTF-8 
is that it's a very pessimal encoding for some languages; hence, some 
of the ACE schemes are designed to be more compact than UTF-8.
 
> My software doesn't have any trouble. 

You posted an example where you claimed that your software would have 
trouble, because that software generates unusually long local-parts,
leaving very little room for domain names.

You claimed that ACE IDNs would cause those addresses to become even
longer, thus preventing your software from functioning.  But you could
just as easily find examples where UTF-8 IDNs would cause those 
addresses to become longer.   And depending on your choice of ACE
scheme and your choice of IDN, you can find examples to support an
argument that either representation is longer than the other.

It's almost inevitable that representation of strings with a character 
repertoire greater than 2**16 in size will take up more space than a 
representation of strings with a character repertoire less than 2**8 
in size.  Thus, longer names is part of the price of having IDNs,
no matter what encoding scheme is used.  Of course, we do want to find
a compact encoding for IDNs, but the right answer seems to be a tradeoff
between maximizing space efficiency, minimizing implementation complexity, 
and minimizing impact on existing software that expects ASCII.

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 11 Jan 2001 19:09:29 -0000

Keith Moore writes:
> trouble,

False.

> because that software generates unusually long local-parts,

My mailing-list software often generates local-parts that include the
_subscriber's_ local-part and domain name. The result may exceed length
limits in the _subscriber's_ mail software, or in relays.

> the presumption is that ACE IDNs will cause less trouble than UTF-8
> IDNs.  

Actually, people have been claiming that ACE IDNs will cause no trouble
at all, and suggesting this as a litmus test for IDN solutions.

RFC 2825, for example, says that _some_ of the proposals will cause
failures if software isn't upgraded first. In fact, _all_ of the
proposals will cause failures if software isn't upgraded first.

As for the frequency and severity of failures: What matters is the
software involved in deliveries _where IDNs appear_. People likely to
use IDNs are also people likely to have already moved to 8-bit-clean
software---because 8-bit data shows up all the time in Subject lines,
despite Keith's efforts to force people to encode it.

> significantly longer than UTF-8 IDNs,

That simply isn't true. A 10-byte limit causes much more trouble for the
ACE proposals than for UTF-8. Try some examples!

---Dan

---

From: "D. J. Bernstein" <[email-masked]>
Date: 11 Jan 2001 19:47:18 -0000

Maynard Kang writes:
> How about servers which are running sendmail and will fall over and
> die when they see UTF-8? According to their site, Sendmail "handles an
> estimated 75 percent of the Internet's email traffic".

That estimate is completely bogus. It's a wild misrepresentation of some
obsolete information that was once on the Sendmail Inc. web pages: viz.,
that more than 75% of the Internet's SMTP servers were running Sendmail.

(The current web pages also have obsolete information, but I don't see
the wild misrepresentation. Where exactly did you get the quote?)

See http://cr.yp.to/surveys/sendmail.html for the complete story.

---Dan

---

From: Keith Moore <[email-masked]>
Date: Thu, 11 Jan 2001 15:05:50 -0500

> > You posted an example where you claimed that your software would have

The fact that the bugs are detected elsewhere is hardly a justification
for poor design choices in your software, especially when those choices
exceed the design limitations that are documented in the mail protocol
specifications. 

> at all, and suggesting this as a litmus test for IDN solutions.

Such claims are obviously exaggerated.  Any IDN scheme will 
cause some degree of disruption, the question is how much.  

> software---because 8-bit data shows up all the time in Subject lines,

8-bit data does show up all the time in Subject lines, and in other
fields.  It is also quite often mangled beyond recognition, due to
a combination of software that is unable to cope with 8 bit text,
software that attempts to convert 8-bit text to RFC 2047 form, and
software that tries to perform character set conversions on such
text without knowing the original character set.

> ACE proposals than for UTF-8. Try some examples!

A 10-byte limit causes trouble for lots of ordinary domain names, so
I fail to see how this is terribly significant.

Keith

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 11 Jan 2001 21:57:01 +0100

At 15.05 -0500 01-01-11, Keith Moore wrote:
>It is also quite often mangled beyond recognition, due to

Most problematic are software which send ISO 8859-1 untagged in 
various places. I get about 10% of my messages in that form (2-5/day 
out of about 20-50 which are in Swedish). Only visible if you run a 
mail client which doesn't use ISO 8859-1 as native charset of course.

   paf

---

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 12 Jan 2001 09:58:05 +0800

Without going to the details, it is possible to design an ACE which has almost
the resultant length as UTF-8 if "10-byte limit" is your concern.

-James Seng

---

From: "D. J. Bernstein" <[email-masked]>
Date: 12 Jan 2001 08:11:56 -0000

James Seng/Personal writes:
> Without going to the details, it is possible to design an ACE which
> has almost the resultant length as UTF-8

Without gigantic tables of common character sequences? Without using
punctuation or control characters that people can't type? Without a
noticeable risk of misinterpreting existing ASCII names?

If it's possible to do this, go ahead! In the meantime, I'm going to
focus on the proposals that have actually been worked out.

> if "10-byte limit" is your concern.

There's nothing special about 10. My examples show that every extra byte
poses a risk; we don't have the 63-byte safe harbor that some people
have imagined. A user's 5-byte or 10-byte or 15-byte domain name can
indirectly trip his software's 64-byte local-part length limit.

---Dan

---

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 12 Jan 2001 17:20:06 +0800

You said there is a length problem of ACE. I am saying that ACE could be
design with the almost the same length as UTF-8 so it is not a concern.

> punctuation or control characters that people can't type?

Why would ACE have these problems?

> Without a

With a prefix as most ACE proposed, I say it is a "small risk".

> focus on the proposals that have actually been worked out.

A lot of proposals have been proven to work to certain extend under certain
constraint. This includes ACE and UTF-8 and other even more horrible ones like
legacy encodings which is better not discussed etc. I have done all of them
and I still there is a better solution out there (e.g. John Klesin's I-D).

So the issue here is not "which works?", but "which works best?" and "what
constraint?".

> indirectly trip his software's 64-byte local-part length limit.

You are missing the point. See my first sentence.

-James Seng