Subject: Re: Just send UTF-8 with nameprep (was: RE: [idn] Reality Check)

From: Dan Oscarsson <[email-masked]>
Date: Tue, 24 Jul 2001 08:20:43 +0200 (MEST)

John wrote:

>>> 
>>> This is just spreading FUD. No halfway serious online trader
>>> types in the broker address every time he trades. For that,
>>> there are things such as bookmarks and autocompletion.
>> 
>> Also it does not matter if nameprep is not applied at the
>> client end, the DNS server will do nameprep before comparing
>> names - so it will work anyway.

>The above sounds to me as if we are back to requiring that all
>servers need to be upgraded in order to have things work
>properly.  Certainly no DNS server deployed today does nameprep
>before comparing names.  That assumption of massive, more or
>less concurrent, upgrades, or its alternative, that we are
>willing to tolerate a long-ish period in which things don't work
>predictably, are the ones I though the WG concluded it could not
>make.

Only the authorative DNS servers handling non-ASCII names need to
be upgraded. Caching and non-authorative servers can do binary matching.
It will result in those servers seeing each different from of
the same name as different domains, but that does not matter as it will
work anyway.

   Dan

---

From: Martin Duerst <[email-masked]>
Date: Tue, 24 Jul 2001 15:40:01 +0900

>the same name as different domains, but that does not matter as it will

Doing nameprep during operation on the server side is still a bad idea.
If we really need the original spelling of the name (which I'm not sure
we do), a new RR may be quite a bit better.

Regards,  Martin.

---

From: John C Klensin <[email-masked]>
Date: Tue, 24 Jul 2001 04:55:55 -0400

--On Tuesday, 24 July, 2001 15:40 +0900 Martin Duerst
<[email-masked]> wrote:

> Doing nameprep during operation on the server side is still a
> bad idea. If we really need the original spelling of the name
> (which I'm not sure we do), a new RR may be quite a bit better.

Martin,

It was trying to think through an approach based on a new RR
that brought me to the conclusion that, if one went down that
path, a whole new class would be needed.   Please work the cases
yourself, remembering the various issues raised by CNAME, MX,
and PTR RRs that might or might not have IDN values (targets)
and see if you come to the same conclusion.

    john

---

From: Martin Duerst <[email-masked]>
Date: Wed, 25 Jul 2001 18:18:06 +0900

Hello John,

I read your Internet Draft proposing a new class, and
it looked very good to me. I'll send you a few detail comments
separately later.

I don't want to disagree with your conclusion, but I'm not
sure about the premise (that we need the original spelling),
so I didn't get into the details of what would follow.

Regards,   Martin.

---

From: Roozbeh Pournader <[email-masked]>
Date: Wed, 25 Jul 2001 19:33:48 +0430 (IRST)

> so I didn't get into the details of what would follow.

We may need original spelling, for example if we decide that we need to
stop Arabic letters from joining, by using a ZWNJ between them (since a
space is not available). Multiword Arabic domain names will be unreadable
if the words join. But since ZNWJ is stripped at nameprep, we will
sometimes need to get to the original. You get the idea.

roozbeh

---

From: Dan Ebert <[email-masked]>
Date: Wed, 25 Jul 2001 08:48:58 -0700

> sometimes need to get to the original. You get the idea.

It is possible to prevent Arabic letters from joining by separating
words by hyphens.  While I don't think this isn't something which should
be incoporated into any kind of proposal, it may be a useful convention
for the registration of Arabic names. I am not an Arabic speaker so I
don't know how acceptable this would be, but it is a possible solution.

-- 
Dan Ebert      <[email-masked]>       eNIC Corporation
------------------------------------------------------------

---

From: David Hopwood <[email-masked]>
Date: Wed, 25 Jul 2001 20:56:46 +0100

-----BEGIN PGP SIGNED MESSAGE-----

Roozbeh Pournader wrote:
> if the words join.

What's wrong with using hyphens to separate words?

(Remember that English domain names don't look like English sentences,
either.)

- -- 
David Hopwood <[email-masked]>

Home page & PGP public key: http://www.users.zetnet.co.uk/hopwood/
RSA 2048-bit; fingerprint 71 8E A6 23 0E D3 4C E5  0F 69 8C D4 FA 66 15 01
Nothing in this message is intended to be legally binding. If I revoke a
public key but refuse to specify why, it is because the private key has been
seized under the Regulation of Investigatory Powers Act; see www.fipr.org/rip

---

From: Martin Duerst <[email-masked]>
Date: Thu, 26 Jul 2001 10:25:29 +0900

>sometimes need to get to the original. You get the idea.

Hello Roozbeh,

Many thanks for your comment. As far as I remember, currently ZNWJ is
dropped by nameprep without complaining. I think that if that leads to
problems as you describe above, we should very carefully reexamine
this decision. I have already told the nameprep design team that
I think that this should be moved to 'disallowed' to avoid surprises
like the above. But I'm not sure moving it to 'disallowed' would
solve all problems (assuming that the hyphen is used for concatenating
words).

Regards,   Martin.

---

From: "Soobok Lee" <[email-masked]>
Date: Thu, 26 Jul 2001 12:16:31 +0900

Arabic glyph U+0640 (Arabic TATWEEL) looks like Latin hyphen(-).

Arabic FULLSTOP U+06D4  looks like Latin 'underline' (or 'hyphen' ).

Using hyphen instead of punctuation char NWNJ(U+200C,No-width No joiner)
 may not help but cause confusion to native-Arabic people.

Soobok Lee, [email-masked]

----- Original Message -----
From: "Martin Duerst" <[email-masked]>
To: "Roozbeh Pournader" <[email-masked]>
Cc: "John C Klensin" <[email-masked]>; "Dan Oscarsson" <[email-masked]>;
<[email-masked]>; <[email-masked]>
Sent: Thursday, July 26, 2001 10:25 AM
Subject: [idn] ZWNJ (was: Re: Just send UTF-8 with nameprep (was: RE:
[idn]Reality Check))

---

From: "Soobok Lee" <[email-masked]>
Date: Thu, 26 Jul 2001 19:50:58 +0900

Using hyphen in place of NWNJ in Arabic multi-word domains
may cause confusion because arabic script already contains
hyphen-like letters already. that's the point.
That's not related directly to nameprep, but to IDN security.

I hope those small ratholes may not become security holes in
IDN.

Soobok Lee

----- Original Message -----
From: "James Seng/Personal" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>;
<[email-masked]>
Sent: Thursday, July 26, 2001 6:55 PM
Subject: [idn] Re: [idn-nameprep] Arabic hyphen-like glyph

> I think this discussion take a strange twist.
>
> I am not sure where the discussion about "folding look-alive characters"
> but that is certainly not the intention of nameprep, nor jpchar nor
> tsconv.
>
> Please keep the discussion within the focus of the core interest, namely
> nameprep, jpchar or tsconv for "matching problems".
>
> If there is intention to talk about "folding look-alive characters" then
> please write an I-D, get your 7 core interest then come back to the wg.
> Personally, I do not suggest anyone going into this rathole...
>
> -James Seng
>
> From: "Soobok Lee" <[email-masked]>
> To: <[email-masked]>; <[email-masked]>
> Sent: Thursday, July 26, 2001 11:16 AM
> Subject: [idn-nameprep] Arabic hyphen-like glyph
>
>

---

From: "James Seng/Personal" <[email-masked]>
Date: Thu, 26 Jul 2001 17:55:19 +0800

-James Seng

---

From: John C Klensin <[email-masked]>
Date: Thu, 26 Jul 2001 16:25:05 -0400

> I hope those small ratholes may not become security holes in

It seems to me, regardless of other IDN work and its outcomes,
that an analysis that identifies these "small ratholes" in
matching and character identification and their security
implications would be a very useful contribution.  Its text might
ultimately become part of something else, or it might be
published separately.  And I would hope that the WG Co-chairs
would not impose an "interest level" requirement on drafts or
components of such an analysis, treating it instead as part of
the general requirement on the WG to produce a "security
considerations" analysis of any proposed standards-track
materials it produces.

    john

---

From: Patrik Fältström <[email-masked]>
Date: Fri, 27 Jul 2001 00:12:08 +0200

> Personally, I do not suggest anyone going into this rathole...

Before you do this, think about what this means:

- In the IETF, we have been extremely careful NOT talking about
  individual characters, codepoints or such, but instead all the time
  referenced other standards bodies when talking about such details.
- In various protocols, we have been pointing to different bodies,
  depending on mood, time of day or even technical merits(!).

Now, IF you are unhappy with nameprep of today, I want you to be extremely
clear whether:

[A] You think the Unicode tables, unification rules etc are wrong, and you
want to change those and instead do the work in the IETF (basically, scrap
what is defined in the Unicode Consortium or ISO because you belive IETF
can do a better job).

[B] You think a different organization than Unicode Consortium is better to
reference than the Unicode Technical Reports (and we still will not do
character by character definitions in the IETF).

Is it [A] or [B] you are after?

If [A], why would IETF be better on this, and don't you think yet another
standard body doing character things would be bad for the world?

If [B], which body are you thinking of?

My point is, if you are unhappy with work done by an organization which we
reference we in the IETF should either bring the issues over there, or
reference some other organization. And, we need to make up our mind on
which one of these two paths we want to follow. Doing both is often a bad
idea (and a waste of time and resources).

     paf

---

From: Martin Duerst <[email-masked]>
Date: Fri, 27 Jul 2001 00:01:09 +0900

>  may not help but cause confusion to native-Arabic people.

Hello Soobok,

Glyphs in Arabic are not isolated. The hyphen would be isolated,
but tatweel/kashida is always connected. The hyphen would be higher
than both tatweel and Arabic full stop (used in Urdu).

So rather than just looking at the glyph charts, please get an
understanding of how Arabic formatting works before trying to
comment on it.

Regards,   Martin.

---

From: "Soobok Lee" <[email-masked]>
Date: Fri, 27 Jul 2001 11:23:52 +0900

----- Original Message -----
From: "Martin Duerst" <[email-masked]>
To: "Soobok Lee" <[email-masked]>; <[email-masked]>;
<[email-masked]>
Sent: Friday, July 27, 2001 12:01 AM
Subject: Re: [idn] Arabic hyphen-like glyph

> than both tatweel and Arabic full stop (used in Urdu).

Your are correct. the hyphen is slightly higher than the arabic
script baseline.
BTW, when a hyphen is put between two arabic words,
the hyphen look disconnected from the right word, but looks
connected to the left word in cases with some arabic leading consonants.
I am not sure that problem come from the used arabic font.

> understanding of how Arabic formatting works before trying to

Anyway, thanks for your corrections.

Regards, Soobok

---

From: "James Seng/Personal" <[email-masked]>
Date: Fri, 27 Jul 2001 14:43:52 +0800

John,

I do agree an analysis of the security implications are useful at least
as informational.

But I need to stress the wg need to proceed with internet drafts. We can
talk endlessly on ideas in emails but it will remains as what it is, ie,
email. This is the message I am trying to bring across to Lee: Write
draft first, then we can discuss based upon it.

As wg co-chair, I will make no judgement about what goes into the pool
or what's not so long there is sufficient interest and request (and of
course within our charter).

But on a personal note, I think Patrik hits the nails on the head. I am
not sure if IDN WG is the right forum to discuss 'look-alive characters'
issues. Remember we not only talking about codepoints, reportairs and
also fonts rendering and font variants...

-James Seng

---

From: Roozbeh Pournader <[email-masked]>
Date: Fri, 27 Jul 2001 14:39:13 +0430 (IRDT)

> What's wrong with using hyphens to separate words?

Nothing's wrong. I only see using ZWNJ as an alternative. With English
domain names, you can have both 'damnthisworld.com' and
'damn-this-world.com'. With Arabic, you will have only the second choice,
since the first one will not be readable. I think that using ZWNJ may help
having more choices...

---

From: Roozbeh Pournader <[email-masked]>
Date: Fri, 27 Jul 2001 15:13:46 +0430 (IRDT)

On Thu, 26 Jul 2001, Martin Duerst wrote:

> As far as I remember, currently ZNWJ is dropped by nameprep without
> complaining. I think that if that leads to problems as you describe
> above, we should very carefully reexamine this decision. I have
> already told the nameprep design team that I think that this should be
> moved to 'disallowed' to avoid surprises like the above. But I'm not
> sure moving it to 'disallowed' would solve all problems (assuming that
> the hyphen is used for concatenating words).

I think dropping it without complain should be much better than making it
disallowed. I really like to be able to display my Arabic domain name with
proper joining. Example: I register an Arabic equivalent of
RoozbehPournader.com, which will be:

	Reh Waw Zain Beh Heh Peh Waw Reh Noon Alef Dal Reh .something

This will create problems when displayed: Heh and Peh, while belonging to
different words, will join, which is a bad thing.

With the current nameprep rules (good), I can put a ZWNJ between Heh and
Peh, so they get separated, and nameprep will rip it out when resolving
the name, which is also fine.

Other solutions:

1. Disallow ZWNJ: bad. I should then use a hyphen between Roozbeh and
Pournader, which I don't like.

2. Allow ZWNJ in names: worse. Someone will put a ZWNJ between two letters
that do not join (like Reh and Alef), leading to no difference in shape,
by a difference in resolving.

3. Rip ZWNJ in nameprep, but also have some DNS (or WHOIS) mechanism to
get to the non-nameprepped name: great. This way any protocol can get to
the original name if it wants.

I want to recommend the third option. How this may happen, I don't know.
But this can be considered a reason for having the reverse mechanism in
place, which was the subject of the thread I replied to.

roozbeh

---

From: Roozbeh Pournader <[email-masked]>
Date: Fri, 27 Jul 2001 16:00:07 +0430 (IRST)

> hyphen-like letters already. that's the point.

No, it won't cause confusion. Feel free. You can take my word as a native
writer of Persian (which uses the Arabic script). I can be proven wrong,
but only by a native writer of Urdu for example, or someone providing good
enough details.

The point I raised about ZWNJ usage, is not that hyphens are bad, just
that ZWNJ may be considered better by many. I like to see both supported.

roozbeh

---

From: Roozbeh Pournader <[email-masked]>
Date: Fri, 27 Jul 2001 15:51:37 +0430 (IRST)

> Arabic glyph U+0640 (Arabic TATWEEL) looks like Latin hyphen(-).

But this is a joining character (something like a letter). We need
something non-joining.

> Arabic FULLSTOP U+06D4  looks like Latin 'underline' (or 'hyphen' ).

This is a very rare character, only used in Urdu.

>  may not help but cause confusion to native-Arabic people.

I can't get what you really mean. Using hyphen will not cause confusion of
Arabic users, be sure about it. Hyphen is used in many languages that use
the Arabic script.

roozbeh

---

From: John C Klensin <[email-masked]>
Date: Fri, 27 Jul 2001 08:54:23 -0400

--On Friday, 27 July, 2001 14:43 +0800 "James Seng/Personal"
<[email-masked]> wrote:

> John,
> 
> I do agree an analysis of the security implications are useful
> at least as informational.
> 
> But I need to stress the wg need to proceed with internet
> drafts. We can talk endlessly on ideas in emails but it will
> remains as what it is, ie, email. This is the message I am
> trying to bring across to Lee: Write draft first, then we can
> discuss based upon it.
>...

I hope my note was not taken to imply anything else.  I meant to
suggest that a security-focused draft might receive somewhat
different handling than the "need supporters to post" model for
protocol-focused or process-focused drafts.  I share your view
that we should stop having discussions about things that people
reference, but don't write drafts about.

     john

---

From: John C Klensin <[email-masked]>
Date: Fri, 27 Jul 2001 09:30:10 -0400

--On Friday, 27 July, 2001 15:13 +0430 Roozbeh Pournader
<[email-masked]> wrote:

> Other solutions:
> 
> 1. Disallow ZWNJ: bad. I should then use a hyphen between
> Roozbeh and Pournader, which I don't like.

Of course, what seems to me, not knowing Arabic, to be a near
equivalent, a space between words, is disallowed in the current
naming rules and always has been.  Hence, if "Joes Pizza" goes
into the DNS, it goes in either as "joespizza" or "joes-pizza".
No other options.  I think this situation should be improved for
both Arabic and English, and all other languages where it is
relevant, but it seems to me that the perceived requirement makes
another "not in the DNS" argument.

> 3. Rip ZWNJ in nameprep, but also have some DNS (or WHOIS)
> mechanism to get to the non-nameprepped name: great. This way
> any protocol can get to the original name if it wants.
> 2. Allow ZWNJ in names: worse. Someone will put a ZWNJ between
> two letters that do not join (like Reh and Alef), leading to no
> difference in shape, by a difference in resolving.

> I want to recommend the third option. How this may happen, I
> don't know. But this can be considered a reason for having the
> reverse mechanism in place, which was the subject of the thread
> I replied to.

Well, to use my silly example again, we have had mechamisms for
putting "Joes Pizza" in whois since the beginning, even if it
points to a domain containing joespizza.  If that meets your
needs, I suggest you want to cover the whois BOF at IETF, but
that this is not an IDN problem.   If you want to be able to
access the information automatically, when names are displayed to
users, whois is too slow and you have a number of interesting
database and coordination problems, none of which appear to be in
the scope of the IDN WG.

    john

---

From: John C Klensin <[email-masked]>
Date: Sat, 28 Jul 2001 12:02:30 -0400

--On Saturday, 28 July, 2001 17:31 +0430 Roozbeh Pournader
<[email-masked]> wrote:

>> Arabic alphabets put together may have different rendering
>> forms depends whether there is a break between the alphabets
>> or not. But it is not neccessary a 'space' because there is
>> no gap. Someone told me that ZWNJ was suggested as a way to
>> insert this break without adding a space altho that was not
>> the reason ZWNJ was there in the first place.
> 
> You have some idea. The ZWNJ is actually used in languages
> like Persian, which are not Arabic (language) based, but only
> use the Arabic script. Since in this languages the words are
> longer than Arabic words, and there is a need to separate
> parts of the word. Consider the case of German: those long
> words are really hard to read. Add the Arabic joinging to this,
> you will surely have problems finding subword boundaries.
> Persian typography then introduced ZWNJ (which is called
> semi-space in Persian traditional typography) to break the
> words so they become readable.

Roozbeh,

My apologies for my lack of understanding and poor statement of
what I did understand.  Let me try again..

DNS names are identifiers and, as identifiers, are subject to
certain restrictions.  Length is one of them -- we even have
words in English that are over 63 character long, although there
aren't many of them, and they can't be used in domain names.  A
second is that, as identifiers, labels are what computer languge
syntax folks call "atoms" or "atomic".  That typically means
"one label, one "word".  There is a long history of pushing
words together to make one DNS labels and using either the one
joining character available (hyphen) or just catenating them.
In the latter case, we just hope the user will figure out what
is going on to preserve whatever mneumonic value we intend.  

Parenthetically, this is one place where our colleagues with
ideographic languages have a huge advantage: they can actually
write multi-word phrases into DNS labels/identifiers without
doing violence to the natural rules of the writing system.

So I don't know quite what to do with ZWNJ and other separators
or near-separators without opening the door to other characters
normally used in other languages as near-separators or
punctuation or near-punctuation, e.g., ":", "'", "!", or "&",
which have been used, normally or artistically, in Indo-European
languages using Roman-derived character sets for many years, and
even recognizing distinct interpretations for some of the
distinct spacing characters and hyphenation ones in Unicode.

For example, while I would _strongly_ not recommend going down
this path, we could, in principle, adopt presentation and coding
rules that would permit, e.g., 

   "O'Reilly & Associates"

to use the domain name

   www."O'Reilly & Associates".com

by coding the key second label as
   O(U+0027)Reilly(U+00A0)(U+0026)(U+00A0)Associates

Again, I don't suggest doing this, but, ultimately, the DNS
itself would have no problems with it and, if one starts
introducing near-space characters from other scripts, then there
is little justification for prohibiting this type of usage in
Roman-based ones.

     john

---

From: Roozbeh Pournader <[email-masked]>
Date: Sat, 28 Jul 2001 17:31:03 +0430 (IRST)

On Sat, 28 Jul 2001, James Seng/Personal wrote:

> Arabic alphabets put together may have different rendering forms depends
> whether there is a break between the alphabets or not. But it is not
> neccessary a 'space' because there is no gap. Someone told me that ZWNJ
> was suggested as a way to insert this break without adding a space altho
> that was not the reason ZWNJ was there in the first place.

You have some idea. The ZWNJ is actually used in languages like Persian,
which are not Arabic (language) based, but only use the Arabic script.
Since in this languages the words are longer than Arabic words, and there
is a need to separate parts of the word. Consider the case of German:
those long words are really hard to read. Add the Arabic joinging to this,
you will surely have problems finding subword boundaries. Persian
typography then introduced ZWNJ (which is called semi-space in Persian
traditional typography) to break the words so they become readable.

<off-topic>
There are still a bunch of Nastaliq calligraphers, who do not
use the ZWNJ, to be able to create long long words which gives the
calligrapher the ability to play with the slanted baseline. People
consider this to be artistic.
</off-topic>

> btw, can arabic keyboard actually enter ZWNJ? I am asking because I
> could not find a way to do so.

On Persian keyboards, this is usally on Shift+Space (or Shift+B when the
OS can't handle the Shift+Space combination). Microsoft keyboards do not
support it, but if you're using Windows 2000, all the Arabic keyboards and
the Persian keyboard have it on Ctrl+Shift+2 (why there, don't ask me).

roozbeh

---

From: "James Seng/Personal" <[email-masked]>
Date: Sat, 28 Jul 2001 11:50:59 +0800

-James Seng

---

From: Roozbeh Pournader <[email-masked]>
Date: Sun, 29 Jul 2001 15:46:50 +0430 (IRST)

John,

You're right about the identifier nature of DNS names. Being brought up in
such a world, I'm already well familiar with the way this impacts the
language. For a good example, see Arthur C Clarke's The Light of Other
Days, ISBN 0812576403, where words like SearchEngine are common. The DNS
and other identifier restrictions have changed the shape of English
language, for sure.

Getting back to the thread, Arabic lacks many of the possiblities of the
Latin script, for getting a distinguished sense out of a sequence of
letters (which we will call identifiers). I consider the use of ZWNJ to be
equivalent to the use of inter-identifier captialization. Just like that,
it should be ignored, just like that, it will help the reader, and just
like that, the original should be retreivable in some way.

Please note that even in single words, ZWNJ is used. In many single words
like the Persian words for "houses", "circular", "eraser", "compatriot",
and "synonymous", or single-word names of places, it may not be dropped in
any way, or the word becomes completely unreadable.

Arabic is connected, unlike Latin where the letters are separate enough
that you can sometimes omit the space (like in domain names, or German).
It's also unlike Han, where there is a good boundary between the words,
without even the need for spaces. So it should use spaces and ZWNJ heavily
to stop joining where it will ruin the meaning or readablity of the
phrases. Please note that ZWNJ is somehow considered a *nothing* in the
Unicode recommendation. It should only affect contextual shaping, and
nothing else...

While I see the use of space-like characters in Latin problematic (mainly
because of indistinguishablity of the written word), the case is
difference with ZWNJ. It is not a space character.

BTW, there are also many other needs for being able to retreive the
original non-nameprepped name. Have you thought about national digit
shapes (as used in Arabic and Indic scripts), for example? Many countries
do not use European digits (which Europeans call Arabic).

roozbeh

---

From: "Adam M. Costello" <[email-masked]>
Date: Mon, 30 Jul 2001 04:38:07 +0000

Roozbeh Pournader <[email-masked]> wrote:

> original non-nameprepped name.

It will be nice to be able to use non-nameprepped names, but what do
you mean by "retrieve"?  If you mean recover them from ACE, that's too
much to hope for; ACE can store the nameprepped name and the mixed-case
info, but anything more would be difficult and inefficient.  If you mean
retrieve from DNS, I don't think that's helpful, because names retrieved
from DNS are very rarely seen by typical users.

I think what you really want is simply to be able to use non-nameprepped
names in application protocols, so that users can see them.  That will
be possible as soon the protocol adds explicit support for IDNs, so that
it no longer relies on ACE as the sole means of representing IDNs.

Maybe you were imagining that an application, after receiving an ACE
name in an application message, would do a DNS lookup to retrieve a
non-nameprepped version of the name.  That would not only increase the
load on the DNS, but might also return the wrong version of the name,
from the sender's point of view.  For example, if mail user agents did
that, everyone would see my email address as "[email-masked]",
which I think is ugly.  If we want to allow users to see non-nameprepped
names, we should aim for the spelling chosen by the sender, not the
spelling chosen by the registrant.  That means the non-nameprepped
spelling has to be taken from the application message, not from the DNS.

AMC

---

From: John C Klensin <[email-masked]>
Date: Mon, 30 Jul 2001 08:35:14 -0400

--On Sunday, 29 July, 2001 15:46 +0430 Roozbeh Pournader
<[email-masked]> wrote:

> You're right about the identifier nature of DNS names. Being
> brought up in such a world, I'm already well familiar with the
> way this impacts the language. For a good example, see Arthur
> C Clarke's The Light of Other Days, ISBN 0812576403, where
> words like SearchEngine are common. The DNS and other
> identifier restrictions have changed the shape of English
> language, for sure.

Conceptually, however, these are _new_ words and new
identifiers, albeit with funny spellings.  The association
between "SearchEngine" and "search engine" is mostly in the mind
of the observer.  It is certainly not in the DNS.

But, to the extent that preservation of those forms is an
advantage, an external-to-the-database process such as nameprep
throws them away.  Either nameprep would have to be reversible
(which seems impossible) or we would need to store the "raw"
names in the DNS, put nameprep in the servers, and make matching
comparisons based on 
  (nameprep (query)) = nameprep (stored-raw-value)
or, worse, I suspect
  encoding(nameprep(query)) =
nameprep(encoding(stored-raw-value))

Those approaches don't seem likely to be workable and certainly
would require massive protocol and server changes -- changes
which the WG has been trying to avoid and which it has been told
are out of its scope.

Perhaps that is part of the point you are trying to make.
 
> Getting back to the thread, Arabic lacks many of the
> possiblities of the Latin script, for getting a distinguished
> sense out of a sequence of letters (which we will call
> identifiers). I consider the use of ZWNJ to be equivalent to
> the use of inter-identifier captialization. Just like that, it
> should be ignored, just like that, it will help the reader,
> and just like that, the original should be retreivable in some
> way.

I think I have understood this, although I haven't been
expressing it very well.  I think that almost every language has
one or another of these issues.  The FooBar convention, for
which believe C and UNIX get the credit (or blame), rather than
the DNS, is really not satisfactory for English, although it is
a hack that sort of works there and not for Arabic.  The basic
Han-folding decisions of Unicode submerge distinctions among the
several languages using derivations of those characters.  And so
on.

> Please note that even in single words, ZWNJ is used. In many
> single words like the Persian words for "houses", "circular",
> "eraser", "compatriot", and "synonymous", or single-word names
> of places, it may not be dropped in any way, or the word
> becomes completely unreadable.

If it is an integral part of the word, such that the word
changes or loses meaning without it, then you, indeed, have a
strong case for its preservation.  I would think that what would
then be needed would be a simple rule (i.e., one that does not
require a program to understand Arabic or consult dictionaries)
to know when it can (and should) be removed and when not.

> Arabic is connected, unlike Latin where the letters are
> separate enough that you can sometimes omit the space (like in
> domain names, or German). It's also unlike Han, where there is
> a good boundary between the words, without even the need for
> spaces. So it should use spaces and ZWNJ heavily to stop
> joining where it will ruin the meaning or readablity of the
> phrases. Please note that ZWNJ is somehow considered a
> *nothing* in the Unicode recommendation. It should only affect
> contextual shaping, and nothing else...

There is a now-traditional nasty comment about standards that
have their origins in printer drivers, but we are stuck with the
thing.

> While I see the use of space-like characters in Latin
> problematic (mainly because of indistinguishablity of the
> written word), the case is difference with ZWNJ. It is not a
> space character.

But it seems to me that, despite the fact (if I have understood
you) that the same character is used in both situations, ZWNJ is
used both within words (your examples from Persian above) and to
separate words.  The "identifier" notion says that we don't get
phrases in identifiers in alphabetic languages.  Non-alphabetic
languages and character sets get lucky in this case, since there
doesn't seem to be any point in confining them to single "words"
rather than short phrases.

> BTW, there are also many other needs for being able to
> retreive the original non-nameprepped name. Have you thought
> about national digit shapes (as used in Arabic and Indic
> scripts), for example? Many countries do not use European
> digits (which Europeans call Arabic).

Of course.  The many examples of this sort of thing, and more
flexible treatment for characters and separators that are not
part of the basic DNS model, are among the things that have
driven me to conclude that we need something with search-like,
rather than lookup, properties to handle international names
without doing violence to the underlying languages.  Any
canonicalization scheme, and any scheme that maps one character
into another, will lose information that would be desirable in
some context, and that isn't a good thing.

But, in the more limited situations you identify, it seems to me
that it would be possible to do a bit of new protocol work and
then adopt some conventions that would be of help.  Suppose, for
example, that we created a new RR type, specifically for
non-"Hosttable"-format labels, for which the "data" (target) was
a raw (no nameprep or other canonicalization or mappings)
Unicode string.  And suppose a convention were adopted such that
any non-traditional label was associated with one of these
records as well as whatever A, MX, AAAA, or equivalent RR it
pointed to (this might force us to make an exception to the "one
record only" rule for CNAME, but maybe that is just a minor
details).

Now, the IDN WG could clearly not do this -- we would need to
get DNSEXT or some other group involved.  But, if the goal is to
be able to retrieve and examine the original name, then, it
seems to me that the way to do that is to store the original
name somewhere, rather than hoping that nameprep can both do its
job and be reversible.

   john

---

From: "Mark Davis" <[email-masked]>
Date: Mon, 30 Jul 2001 20:40:59 -0700

After giving this some thought, I agree with Roozbeh that the ZWNJ should be
removed from the prohibited list, and added to the Mapped Out list.

There is a strong difference between this and the BIDI codes. Those could be
used to provide a display that was very much different than the default
ordering, and could lead to confusion or be used for spoofing. ZWNJ on the
other hand, is harmless. The only effect it would have is to break a normal
cursive join, which would be easily recognized for what it is.

Mark

—————

πάντων μέτρον ἄνθρωπος — Πρωταγόρας
[http://www.macchiato.com]

---

From: David Hopwood <[email-masked]>
Date: Tue, 31 Jul 2001 12:23:15 +0100

-----BEGIN PGP SIGNED MESSAGE-----

> removed from the prohibited list, and added to the Mapped Out list.

ZWNJ already is in the Mapped Out list
(draft-ietf-idn-nameprep-05.txt section 3.3).

A related question is whether names that are returned in reverse lookups have
to be nameprepped. If they don't, then they can include ZWNJ, uppercase,
national digit shapes, and similar presentation differences. If they do, then
such differences would be lost. (I'm talking here about queries by "new"
applications that can handle an arbitrary Unicode response to a reverse lookup.
For "old" applications that get an ACE response, it's desirable that there
should only be one form of the ACE name for a given equivalence class of
Unicode names.)

I don't think it's a big deal because the results of reverse lookups aren't
seen very often by users, but I'm coming round to the view that they shouldn't
need to be nameprepped (although they must still satisfy the validity rules,
obviously). This is still far less important than other considerations when
choosing a proposal, though.

- --

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 31 Jul 2001 21:20:48 +0000

David Hopwood <[email-masked]> wrote:

> A related question is whether names that are returned in reverse
> lookups have to be nameprepped.

In my opinion, nameprep is for defining the equivalence between names,
and therefore it needs to be invoked only when comparing names.  I
see no advantage to perform nameprep before sending a name, and the
disadvantage is that it loses information that might be helpful to
humans.

There are two exceptions to that rule:  First, ACEs must always be
nameprepped, because the whole point of ACE is to be compatible with
old applications, which don't know about nameprep.  Second, non-ACE
DNS queries (not responses) should be nameprepped on the client as a
performance optimization, because that would always be the first thing
the server would do anyway (there is no human user on the server side to
lament the loss of information, and the client can retain the original
form).  If there are other protocols in which a server would always
apply nameprep to all incoming names and discard the original form, the
same argument could apply to them.  But in general, for non-ACE names, I
think nameprepped names should not be shipped around; nameprep should be
performed as part of the comparison operation.

AMC

P.S. Actually, the easiest way to compare two labels is to convert both
to ASCII (using nameprep, and ACE if necessary) and then compare using
the customary case-insensitive ASCII comparison.  This not only catches
the equivalence between two old-style labels, two ACE labels, and two
non-ASCII labels, but also handles the case where one label is non-ASCII
and the other is the corresponding ACE form, and the case where one
label is simple non-ACE ASCII and the other is a non-ASCII equivalent
form (using full-width Latin letters, for example).