Subject: [idn] Few comments on idna

From: Marc Blanchet <[email-masked]>
Date: Mon, 11 Feb 2002 23:40:20 -0500

Hi,
 here is few comments on idna.

Marc.

- p2. "To allow such a label to be handled by existing applications, an 
"ACE label" is defined ..."
 + ACE should be spelled: ASCII Compatible Encoding.

- 3. 2). "When requirements 1 and 2 both apply, requirement 1 takes 
precedence".
 + My understanding is the inverse: i.e. requirement 2 takes precedence. I 
might be wrong.

- 4.1 ToASCII and 4.2 ToUnicode
 + the algorithm should more clearly state what happens when the "Verify 
actions" result to fail.

- 5 ACE prefix.
 "The ACE prefix, used in the conversion operations (section 4), will be 
specified in a future revision of this document.
 It will be two alphanumeric ASCII characters followed by two 
hyphen-minuses. ... "

 + I think IANA should specify the prefix in the IANA registry. Also the 
specification of the prefix (i.e. xx--) should not
 be included in this document, but is left to IANA. Or, it could be instead 
put in an IANA section in the document.

- 5 ACE prefix.
 + At the end of this section, there should be some words about the fact 
that in the long term, we might have a different prefix.
 The intent of this is only to tell developers that one way to change this 
"protocol" is through a new prefix. So implementations
 will be prepared to behave. Suggestion of wording:
 "Future revisions of this specification may involve using a different ACE 
prefix."

- 6.1 Entry and display in applications.
 "if it does, rendering the ACE SHOULD NOT be the default."
 + we should make it clear that the prefix MUST be shown if the ACE version 
is shown.  This is probably obvious for us, but the
 document is not clear about the fact that when "ACE" is used, it means 
with or without the prefix.  This comment is actually for
 most of the document. 6.4 second paragraph has the same ambiguity.

- after 8, should have an IANA considerations section. Here is a suggestion.
 IANA Considerations
 This specification requires the IANA to define a prefix that would be used 
for identifying an ACE label in a domain name.
 The prefix should be chosen to be not used at the time of the choice by 
any label in any depth of the dns tree. This might
 be impractical to find. The IANA should consult the IESG and the wg chairs 
to define such a prefix.

- 9. Security Considerations
 s/Much of the security of the Internet relies/Security on the Internet 
partly relies/

- General comment about normative references:
 + IDNA is the core/root of the documents.  It is right now not mentioning 
stringprep at all.
 It MUST clearly have a paragraph on stringprep and clearly reference it, 
with a clear "Mandatory to implement" statement.
  This is because stringprep contains important data for developers (for 
example, how to deal with versioning).

------------------------------------------
Marc Blanchet
Viagénie
tel: +1-418-656-9254x225

------------------------------------------
http://www.freenet6.net: IPv6 connectivity
------------------------------------------
http://www.normos.org: IETF(RFC,draft),
  IANA,W3C,... standards.
------------------------------------------

---

From: Marc Blanchet <[email-masked]>
Date: Mon, 11 Feb 2002 23:41:32 -0500

Few comments on stringprep

3. Mapping.
 + typo: s/database should used to/database should be used to/

4. Normalizatoin
 + there is an inline comment to be taken out. "[[[[[ Patrick asks: ..."

6. Unassigned code points
 + typo: s/repertoire named in the stringprep/repertoire named in a 
stringprep/

 6. 3rd paragraph.
 " prevents the need for versioning in protocols...".
 + I think we should put some reference to either 
draft-ietf-idn-versioning-00.txt
 or to some idn meeting minutes to refer the reader to the discussion on 
this topic.

6.1
+ typo:  on MN line: s/because they are never appear/because they never 
appear/

Annex B. IANA Considerations
+ this section should be completed and the in-line comment removed.

---

From: Marc Blanchet <[email-masked]>
Date: Mon, 11 Feb 2002 23:41:12 -0500

Few comments on punycode

- title: Punycode version 0.3.3.
 + the title should be changed to something like punycode version 1.0. 
version 0.3.3 does not look a stable specification.
  I would also argue to have a more descriptive name in the title, 
something like: "Punycode: an encoding designed for use iwth
  Internationalized Domain Names (idn)."

- 3.2 typo:  s/but should insead/but should instead/

- Annex B.
 + to me, this is not clear if it is mandatory to implement or not. We 
should put a clear statement about this.

= Annex D.
 + I would erase the first 8 lines in comments. The author is already 
identified in the draft (author section). Also
 the numbering of the versions is inconsistent!!!.

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 12 Feb 2002 06:47:24 +0000

> + ACE should be spelled: ASCII Compatible Encoding.

Good catch, "ASCII Compatible Encoding" never appears anywhere in the
IDNA draft.  It's awkward to insert in the middle of "ACE label", but we
could add "(ACE stands for ASCII Compatible Encoding)" at the end of the
sentence.

> + My understanding is the inverse: i.e. requirement 2 takes precedence. I 

You are wrong.  :)

The ASCII form might be ugly, but software won't choke on it.  The
non-ASCII form is pretty, but some software will choke on it, and that's
more important.  So requirement 1 takes precedence.  It basically says
don't pass non-ASCII domain names to software that might choke on it.

> + the algorithm should more clearly state what happens when the "Verify 

Section 4.1 says "ToASCII fails if any step of it fails.  Failure
means that the original sequence cannot be used as a label in an IDN."
Section 4.2 says "ToUnicode never fails.  If any step fails, then the
original input sequence is returned immediately in that step."  How can
we be any clearer?

> + we should make it clear that the prefix MUST be shown if the ACE
> version is shown.  This is probably obvious for us, but the document
> is not clear about the fact that when "ACE" is used, it means with
> or without the prefix.

Section 2 says "The 'ACE prefix' is defined in this document to be
a string of ASCII characters that appears at the beginning of every
ACE label." and "The conversion of labels to and from the ACE form is
specified in section 4.", and in section 4 ToASCII is defined in such a
way that ACE labels always begin with the ACE prefix.  Section 5 "ACE prefix"
even gives an example:

    For example, the eventual ACE prefix might be the string "jk--".
    In this case, an ACE label might be "jk--r3c2a-qc902xs", where
    "r3c2a-qc902xs" is the part of the ACE label that is generated by
    the encoding steps in [PUNYCODE].

I have a hard time imagining what would lead someone to think that "ACE"
refers to the part after the prefix.  But if Patrik or Paul wants to
make an editorial change emphasizing that, I don't see what harm it
would cause.

> + the title should be changed to something like punycode version 1.0. 

Agreed.  In the next revision it will be 1.0.0.

> Internationalized Domain Names (idn)."

Agreed.  How about "Punycode: An encoding of Unicode for use with IDNA"?
(I think it's significant that it's for use with IDNA as opposed to IDN.
I could expand IDNA if you like, but that would get very long.)

> - 3.2 typo:  s/but should insead/but should instead/

Thanks.

> should put a clear statement about this.

The introduction says "Punycode can support an optional feature
described in appendix B", and the third (== last) paragraph of appendix
B is "Punycode encoders and decoders are not required to support these
annotations, and higher layers need not use them."  Is that not clear
enough?  Should I add a third such statement at the beginning of the
appendix?  Or move the last paragraph to the beginning of the appendix
(which means it would contain forward references)?

> identified in the draft (author section).

You would, but I wouldn't. :) The first thing most people are going to
do with the sample code is save it to a .c file, and I want the version
information to be kept with it, along with the URL pointing at possibly
newer versions of the sample code.

> Also the numbering of the versions is inconsistent!!!.

The sample code has its own version number.  The version numbers of both
the spec and the sample code will become 1.0.0 in the next revision,
but they could diverge after that.  Editorial changes to the spec don't
cause changes in the sample code.  Also, an interface change in the
sample code could cause a large jump in the sample code version (to
1.1.0 or 2.0.0) while causing only a small jump in the spec version (to
1.0.1).

AMC

---

From: "Kent Karlsson" <[email-masked]>
Date: Tue, 12 Feb 2002 12:05:51 +0100

Some smaller comments (most, but not all, editorial):

IDNA:
  clause 1:
	"src attribute in an HTML <IMG> tag" --> "'src' attribute in an XHTML 'img' tag" 
		(promote XHTML instad of HTML, and quoting attribute and tag names)

  clause 6.1:
	"and can display domain names in any charset"; apart from that "any"
	here is a very strong assumption, I cannot make sense of that sentence.

  clause 6.2:
	"not only domain names in Unicode, but also in local script"; that
	does not make sense

  clause 6.3:
	"this memo", memo?

  clause 6.4:
	"displaying the name with the replacement character"?  The replacement
	character does not figure in displaying any other character (except
	in Linux xterm, but that's twisted); a (or or one of the) replacement
	glyph(s) is used however, when a proper glyph cannot be found.

	"...SHOULD show the name in ACE format... This is to make is to make it
	easier to transfer the name correctly to other programs"; cut and paste
	works very well also for strings that contain characters not in the
	font used at the moment, using an ACE display will be more of problem
	than help.  Please remove that UNHELPFUL recommendation.

	"show...replacement character"; same error; display is done via glyphs
	not via characters (the mapping is non-trivial)

  clause 6.4: "all Unicode text is stored in logical order".  I wish,
	but this does not hold for the "logical_order_exception" characters,
	nor for Khmer ROBAT, and need not hold for the Arabic presentation
	forms.

  General: There should be opening for transition to a long-term non-ACE
	solution (probably based on UTF-8).  The ACE based solution should
	be explicitly declared to be short-term.

stringprep:
  clause 1.1: "aepfel" is a *fallback* for "äpfel", they are not equivalent
	at all.  They sometimes (only sometimes) collate the same in German.
	(One could argue that combining diaeresis and combining latin small
	letter e are "more equivalent" (than the given example); but small
	letter e as a diacritic above has fallen out of use.)

  clause 3: [case] "mapping"; nameprep does not do case mapping, it does
	case folding (not quite the same).

  clause 4: "for KC" --> "form KC"

nameprep:
  clause 1.2: "SYMBOLS" does not occur in the tables; though it should
	[as I mentioned in another comment]...

  clause 3.1: Khmer "INHERENT" vowels are invisible, and should in most
	cases (if not all cases) be ignored, in IDNs in particular;
	there may be a few mappings that should be done for Khmer for
	mistaken encodings as characters)

  clause 5.3: the replacement character is not used or display, it works
	on the character level, not at the glyph level

  clause 5.8: 10646 does not deprecate any characters; but Unicode does
	deprecate those characters.

		Kind regards
		/kent k

---

From: Martin Duerst <[email-masked]>
Date: Tue, 12 Feb 2002 21:00:29 +0900

>stringprep:
>   clause 1.1: "aepfel" is a *fallback* for "辰pfel", they are not equivalent

This is definitely true. Some more details:

The text says

     "aepfel" vs. "<U+00E4>pfel.com"

thus equating apples with oranges. It should either be

     "aepfel.com vs. <U+00E4>pfel.com"

or it should be

     "Aepfel" vs. "<U+00C4>pfel"

(nouns are always written in upper case in German).

Regards,    Martin.

---

From: Marc Blanchet <[email-masked]>
Date: Tue, 12 Feb 2002 12:47:09 -0500

-- mardi, février 12, 2002 06:47:24 +0000 "Adam M. Costello" 
<[email-masked]> wrote/a écrit:

>> + My understanding is the inverse: i.e. requirement 2 takes precedence.
>> I  might be wrong.
>
> You are wrong.  :)
>

Well, I had a hard time to clearly understand those two paragraphs.
I'll try to come up with a suggestion on writing.

> original input sequence is returned immediately in that step."  How can

I read those statements. But the way the steps are written, it is not clear 
what to do if verification fails. There is an implicit rule in what you 
wrote about the fact that you continue to the next step. I would add at the 
end of all verify steps a simple sentence to make it clearer. Something 
like: "If verification fails, exit".

> way that ACE labels always begin with the ACE prefix.  Section 5 "ACE
> prefix" even gives an example:
>

I tried to "forget" about my "involvement" in idn and then read the drafts 
with an implementor perspective new to idn. With this twist, I found not 
clear where the prefix was implied, when it was not.

> I could expand IDNA if you like, but that would get very long.)

That would be much better at least than the current title.
I'm fine with your proposal.
However, I would expand the IDNA word (even if it costs extra bits...). 
Make things more clearer.

> (which means it would contain forward references)?

I don't know. I would probably add a sentence just after "Punycode can 
support ...", something like: "However, it MUST NOT be implemented for 
IDNA."

> newer versions of the sample code.

The version will be freezed with the RFC number. i.e. the version of the 
code is the RFC number.

> The sample code has its own version number.

well, the draft was more interesting than that: See:

title of the draft: "Punycode version 0.3.3"
then annex D:

/*
punycode.c 0.4.0 (2001-Nov-17-Sat)

/* This is ANSI C code (C89) implementing Punycode version 0.3.x. */

How relevent to a specification is the date you made the code?

> The version numbers of both

Again, the version now will be the RFC number.

I think your version numbering management is all fine for yourself. I don't 
think it is appropriate for an RFC.

I would suggest that you refer between your new versions of the code to the 
"Punycode version RFCXXXX". And leave the RFC "clean" of your internal 
numbering schemes.  For example, we don't put CVS tags in RFCs, even if 
they are useful for document management.

Marc.

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 12 Feb 2002 12:10:58 -0800

>The sample code has its own version number.

Then you should say next to the version number what that version 
number is for. "This is code version foo, which matches Punycode 
specification version bar."

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Adam M. Costello" <[email-masked]>
Date: Tue, 12 Feb 2002 22:59:13 +0000

[This message contains responses to Kent Karlsson, Marc Blanchet, and
Paul Hoffman.]

Kent Karlsson <[email-masked]> wrote:

>  "...SHOULD show the name in ACE format... This is to make is to make
>  it easier to transfer the name correctly to other programs"; cut and
>  paste works very well also for strings that contain characters not
>  in the font used at the moment, using an ACE display will be more of
>  problem than help.

It depends on the platform and the GUI.  Sometimes copy/paste retrieves
a string from the application, but sometimes it just retrieves a string
from the GUI corresponding to exactly what is displayed.  Maybe the
language in the draft should be sensitive to this.

> There should be opening for transition to a long-term non-ACE solution
> (probably based on UTF-8).

The draft already says:

    If a signalling system which makes negotiation possible between old
    and new DNS clients and servers is standardized in the future, the
    encoding of the query in the DNS protocol itself can be changed from
    ACE to something else, such as UTF-8.

> The ACE based solution should be explicitly declared to be short-term.

No, because the draft then says:

    The question whether or not this should be used is, however, a
    separate problem and is not discussed in this memo.

Marc Blanchet <[email-masked]> wrote:

> > How about "Punycode: An encoding of Unicode for use with IDNA"?
> 
> I would expand the IDNA word (even if it costs extra bits...). 

Okay.

> I would probably add a sentence just after "Punycode can support ...",
> something like: "However, it MUST NOT be implemented for IDNA."

A Punycode encoder/decoder that implements mixed-case annotation will
behave exactly the same as a Punycode encoder/decoder that does not
implement mixed-case annotation, except that the encoder can accept
flags and the decoder can return flags.  But those flags don't alter
the output of the encoder except for the case of ASCII letters (which
doesn't matter for domain labels), and they don't alter the sequence of
code points returned by the decoder at all.  So there is no reason to
forbid the Punycode encoder/decoder from implementing this feature.

I will, however, add explicit language to that effect:

    Mixed-case annotation MUST NOT alter the encoder output as defined
    in section 6.3 except for affecting the case of ASCII letters, and
    mixed-case annotation MUST NOT alter the sequence of code points
    returned by the decoder as defined in section 6.2.

> The version will be freezed with the RFC number. i.e. the version of
> the code is the RFC number.

It's possible that I might continue to develop the code after the RFC is
published.  For example, I might make stylistic changes or add comments
or slightly change the interface.  Those later versions of the sample
code won't appear in an RFC, so they'll need their own version number,
and it will be nice if the sample code in the RFC has a version number
that can be compared with the version number in the more recent sample
code.

> /* This is ANSI C code (C89) implementing Punycode version 0.3.x. */

What's wrong with that?  Version 0.4.0 of punycode.c implements version
0.3.anything of Punycode.  I can use a wildcard like that because all
0.3.x versions are equivalent, with only editorial differences.

> How relevent to a specification is the date you made the code?

It's not, but the the date is relevant to the code itself.  We are
embedding a file inside a document, and that file contains its version
number and date stamp.

> I think your version numbering management is all fine for yourself. I
> don't think it is appropriate for an RFC.

Lots of RFCs have version numbers:

RFC-1950 ZLIB compressed data format specification version 3.3
RFC-1951 DEFLATE compressed data format specification version 1.3
RFC-1952 GZIP file format specification version 4.3
RFC-2060 Internet Message Access Protocol - version 4rev1
RFC-2246 The TLS protocol version 1.0
RFC-2437 PKCS #1: RSA cryptography specifications version 2.0
RFC-2616 Hypertext Transfer Protocol -- HTTP/1.1

> specification version bar."

Okay, I'll make that more clear.

AMC

---

From: "Kent Karlsson" <[email-masked]>
Date: Wed, 13 Feb 2002 11:35:12 +0100

> a string from the application, 

Correct behaviour.

> but sometimes it just retrieves a string
> from the GUI corresponding to exactly what is displayed.

Any such system/application is SEVERELY FLAWED.  What is displayed
is glyphs.  In many cases there is no (uniquely) corresponding character.
For example (taking a Latin script example) an fj-ligature glyph,
or a Brahmic script ligature glyph, or a sequence of (positioned,
as all glyphs) subglyphs.  Or consider a BIDI string, correctly displayed.

I'd suggest NOT taking such severely flawed implementations into account.

		Kind regards
		/kent k

---

From: "Kent Karlsson" <[email-masked]>
Date: Wed, 13 Feb 2002 11:43:21 +0100

> > Agreed.  How about "Punycode: An encoding of Unicode for use with IDNA"?

In the UTR 17 model, Punycode is a "Transfer Encoding Syntax" (TES),

A few other things are TESes: UTF-7, MIME/QP, MIME/BASE64, GB/HZ,
and also the syntaxes \uhhhh of Java/C and &#xhhhhh; of XML.

I would not simply refer to Punycode as an "encoding", as to many
that would suggest that it was a Character Encoding Scheme or
Character Encoding Form.

		Kind regards
		/kent k

PS
This comment does NOT mean that I endorse or even in the tiniest way
support Punycode, QP, or HZ.  For IDNs, I still favour using UTF-8.

---

From: "Adam M. Costello" <[email-masked]>
Date: Wed, 13 Feb 2002 21:24:27 +0000

Kent Karlsson <[email-masked]> wrote:

> A few other things are TESes: UTF-7, MIME/QP, MIME/BASE64, GB/HZ, and
> also the syntaxes \uhhhh of Java/C and &#xhhhhh; of XML.
>
> I would not simply refer to Punycode as an "encoding", as to many that
> would suggest that it was a Character Encoding Scheme or Character
> Encoding Form.

Would other Unicode experts concur that the title of the Punycode draft
should call it a transfer encoding syntax?  Or if it doesn't really fit
into the model at all, should it use a vague term like "transformation"
rather than "encoding"?

> > sometimes [copy/paste] just retrieves a string from the GUI
> > corresponding to exactly what is displayed.
>
> Any such system/application is SEVERELY FLAWED.

Probably true, but unfortunately it's also quite common.  It's the norm
for X Windows applications.  X is severely flawed in many ways.

AMC

---

From: Marc Blanchet <[email-masked]>
Date: Wed, 13 Feb 2002 16:47:03 -0500

> forbid the Punycode encoder/decoder from implementing this feature.

To me, we want a spec which defines clearly what has to be implemented and 
not.
This option is an optional behavior that should not be implied as to be 
implemented in order to be conformant.

I would argue again to say clearly that this option should not be 
implemented in IDNA.

> published.

And others will implement the spec. So does the specific versioning of the 
code of a specific author make any sense in the context of multiple source 
codes. No.

>For example, I might make stylistic changes or add comments
> or slightly change the interface.

sure. great.

>  Those later versions of the sample

again. version number of the code will be the RFC number.

> 0.3.x versions are equivalent, with only editorial differences.

I understood that you had internal versioning of your algorithm and this is 
ok.
I don't think it will be of any relevance inside an RFC.

If I implement your spec, I'll say it is compliant with RFC XXXX, not to 
the x.y.z version of your code. So your own versioning is irrelevent to the 
RFC.

> embedding a file inside a document, and that file contains its version

again. it is not relevent. We are publishing a specification with sample 
code in it.

> RFC-2616 Hypertext Transfer Protocol -- HTTP/1.1

yes. it is exactly my point. You could say in the title "Punycode v1.0 ...".
But the code inside it does not need to have a version number.

Marc.

---

From: "D. J. Bernstein" <[email-masked]>
Date: 13 Feb 2002 16:30:57 -0000

Kent Karlsson writes:
> What is displayed is glyphs.  In many cases there is no (uniquely)
> corresponding character.

You misunderstand.

Look at the copy-and-paste supported by the UNIX xterm program. Does it
read dots off the display and attempt to convert them to characters? Of
course not. It obtains characters directly from xterm.

This does _not_ mean, however, that it obtains characters directly from
``the application,'' such as the MH/NMH ``show'' program that Keith uses
to read his mail. The system has been engineered so that

   * programs such as ``show'' simply print characters, without worrying
     about dots on the display, copy-and-paste, etc.;

   * programs such as ``xterm'' read characters and manage the display,
     without worrying about where the characters came from.

This modularity speeds software development; let me again recommend
Gancarz's ``UNIX Philosophy'' book. The system doesn't have, and doesn't
need, copy-and-paste in programs such as ``show''.

Web-mail systems are engineered the same way:

   * The web server converts your mail to HTML, without worrying about
     your display.

   * The browser displays the HTML, without worrying about how the HTML
     was generated.

This modularity again speeds software development---but it again means
that there's no copy-and-paste support in the web-mail system.

We seem to have many novice programmers here, people under the delusion
that copy-and-paste is always handled by ``the application,'' which can
provide characters in some dorky 7-bit encoding if it wants to. But the
real world has, in many cases, advanced beyond that primitive model.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago

---

From: "Kent Karlsson" <[email-masked]>
Date: Thu, 14 Feb 2002 12:45:02 +0100

Dan,

	I hesitate to say, but you misunderstand.  Grossly.

	For "copy and paste" involving a terminal emulator, the
"application" of interest is the terminal emulator itself.  

	Now, converting from a stream of characters to a stream of
glyphs is in general a non-trivial step.  For a very restricted
subset of strings, the mapping **MAY** be (but usually isn't,
since it is a bad idea and does not generalise well) made by
mapping from the stream of input characters to a stream of
characters representing the glyphs that are to be displayed.
Sometimes, in these restricted cases, the mapping may be the
identity mapping, but even in these restricted cases, the mapping
may take context into account (for contextual shaping of Arabic
for instance) or do BiDi mapping to "visual order" (i.e. left-to-right
order, misnamed).

	But in general this approach does not work: ligatures that don't
have corresponding characters, contextual shapes that don't have
corresponding characters, etc. All-in-all this depends much on
the font chosen in the "front UI application".

	In either case, it is *wrong* to take the sequence of "glyph
indicies" (which is the result of these processes, whether these
correspond to allocated characters or not) as the object to "cut and
paste".  It must be the sequence of characters that was the input
to that process that is the object to do "cut-and-paste" on.  

	Now if a background application does the mapping from characters
to glyph indices (maskerading as characters), cutting and pasting
at the terminal emulator application level can't be expected to
work.  But this is a degenerate situation; the background application
should not attempt to do this mapping.  A properly constructed
"front UI application" should get a stream of characters from 
the "background application"; the mapping to glyph indices (and
where to place the glyphs) should be done by the "front UI
application" (normally via some system library; and the actual
mapping often depends on the font used); cutting (and pasting)
should involve only the "front UI application" and should work on
the characters received, not on the glyph indices (even if they
are mascarading as characters).

	The glyphs, or rather their indices, in a step further may
be turned into a set of pixels on a screen (the true glyphs, if
you like, but it is tedious to say "glyph indices" all the time,
so they are informally referred to as "glyphs"), a step which I
haven't talked about, and is not expected to be reversed by
some kind of OCR process for cut-and-paste (where did you get
that idea from?); in short: the pixels are irrelevant here.

	And yes, I do copy-and-paste from web pages quite often,
I think I know how it works, in principle at least... On rare
occations I also use terminal emulators, and copy-and-paste
there too.  I think I know how that works, or should work, in
principle too...  Xterm, under Linux, does some odd things where
it maps characters to characters (glyph indices really). I'm not
sure if, or from which version, it does correct copy-and-paste.

	Whichever side ("background" or "front") that gets
a Punycoded string, to be able to convert to a Unicode
string, must "know" that it is Punycoded.  That is very
hard to do if *both* sides are "unaware" of Punycode, and
where it may be applied and were it may not be applied...

		Kind regards
		/kent k

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]]On Behalf Of
> D. J. Bernstein
> Sent: den 13 februari 2002 17:31
> To: [email-masked]
> Subject: [idn] Why IDNA breaks copy-and-paste
> 
> 
> Kent Karlsson writes:

---

From: "Mark Davis" <[email-masked]>
Date: Thu, 14 Feb 2002 06:55:23 -0800

transfer encoding syntax would be the right term. See
<http://www.unicode.org/unicode/reports/tr17/, section 7.

Mark
—————

Γνῶθι σαυτόν — Θαλῆς
[For transliteration, see http://oss.software.ibm.com/cgi-bin/icu/tr]

http://www.macchiato.com

----- Original Message -----
From: "Adam M. Costello" <[email-masked]>
To: "'IETF idn working group'" <[email-masked]>
Sent: Wednesday, February 13, 2002 13:24
Subject: Re: [idn] a few more comments on the last call documents

> > A few other things are TESes: UTF-7, MIME/QP, MIME/BASE64, GB/HZ,
and
> > I would not simply refer to Punycode as an "encoding", as to many
that

---

From: "D. J. Bernstein" <[email-masked]>
Date: 14 Feb 2002 18:14:41 -0000

In July 2001, when this failure pattern was pointed out, Costello
admitted that ``getting an IDN for my domain'' would cause failures with
IDNA during his proposed (multi-year) software upgrade. He admitted that
his previous ``nothing will actually break'' comment was ``overstated.''

He claimed that IDNA ``fails in fewer instances'' than other proposals.
So what? IDNA indisputably fails in more instances than the status quo.

Now, several months later, many IDNA proponents once again seem to
believe that they can deploy IDNA right now without breaking anything.
But that belief doesn't stand up to an examination of the facts.

> "application" of interest is the terminal emulator itself.  

This ``application,'' like a browser looking at the output of a web-mail
server, doesn't know where the domain names are, so it can't convert
those names to PunyCode for copy-and-paste.

Bottom line: If the user sees (glyphs for) non-ASCII characters, then
other programs will receive non-ASCII characters (in the local character
encoding). Many of those programs won't know anything about IDNA, so
their domain-name lookups will fail.

How are we supposed to avoid these failures? What are software authors
supposed to do? Why do the IDNA proponents keep dodging this fundamental
question?

> glyphs is in general a non-trivial step.

Irrelevant. Copy-and-paste grabs characters, not glyphs. You are putting
a huge amount of effort into attacking a stupid copy-and-paste design
that no sane programmer would use.

The current X release has full UTF-8 support; you simply set a UTF-8
locale such as LANG=en_US.UTF-8. Copy-and-paste handles full Unicode.

---

From: "Marco d'Itri" <[email-masked]>
Date: Fri, 15 Feb 2002 00:06:19 +0100

>their domain-name lookups will fail.
And if the user types non-ASCII characters with his own keyboard then
programs which do not know about IDN will receive non-ASCII characters
and their domain-name lookups will fail. So what?
Users will learn that entering IDN in applications not IDN-enabled does
not work and will update them or use a X applet with an embedde punycode
converter.

 >supposed to do?
If you really want to know, while waiting for all software to be updated
to the IDN API (which I hope that in simple cases like ping or telnet
would amount to adding a flag to getaddrinfo(3) or something like this)
my plan is to LD_PRELOAD a simple library which will wrap name
resolution library calls and call the new API.
This may not work with more complex programs like a MUA which puts a
domain in a RFC 2822 header, but would fix all simple programs like
ping and telnet (and I think your tcpclient).
And "just entering UTF-8" would still not work for my MUA, because UTF-8
characters are not allowed in mail headers until DRUMS says so.

 >locale such as LANG=en_US.UTF-8. Copy-and-paste handles full Unicode.
As I explained, in most countries you cannot "simply set a UTF-8
locale" because that would damage interoperability with other people
using a native charset, so this is not a generally applicable solution.

-- 
ciao,
Marco

---

From: Markus Kuhn <[email-masked]>
Date: Thu, 14 Feb 2002 12:12:53 +0000

We try to make sure that xterm preserves as much of the original UCS
plaintext that was sent to it as feasible for the purpose of cut&paste.
This is not achieved with high perfection yet in the current version,
for example xterm currently still converts tabs into spaces and drops
combining characters if there are more than two per base character. It
also drops most control characters, in particular it currently does not
preserve SGR. This is supposed to improve in the future, but cut&paste
will always remain a compromise hack in terminal emulator applications.
In particular, do not expect cut&paste to work in any satisfactory way
for bidi scripts in the foreseeable, but they are a hopeless can of
worms anyway for terminal emulators and probabaly should better only be
used with GUI applications in general.

http://www.cl.cam.ac.uk/~mgk25/unicode.html

Markus

-- 
Markus G. Kuhn, Computer Laboratory, University of Cambridge, UK
Email: mkuhn at acm.org,  WWW: <http://www.cl.cam.ac.uk/~mgk25/>

---

From: "D. J. Bernstein" <[email-masked]>
Date: 15 Feb 2002 19:53:03 -0000

Markus Kuhn writes:
> xterm currently still converts tabs into spaces and drops
> preserve SGR.

The ability to copy and paste tabs and font selections isn't relevant to
IDNs. However, the limit of two combining characters may interfere with
some languages. Why two?