Subject: Re: [idn] Re: idn-uri document

From: Steve Dyer <[email-masked]>
Date: Fri, 01 Nov 2002 09:40:52 +0000

At 23:40 31/10/2002 +0100, Erik Nordmark wrote:
> > >The defined syntax rules for declare certain ASCII domain names illegal
> > >(such as *.example.org). Where is the check for illedgal names assumed to
> > >be performed? For IDNA it probably makes sense to only apply this types
> > >of checks (setting the UseSTD3ASCIIRules flag) when verifying domain name
> > >registrations and not do such checks in the clients.
> >
> > This is an IDNA question, not a idn-uri question. As far as I remember,
> > the idea was to have the checks done on the clients, too (with some
> > leeway for unassigned characters to stay forward-compatible with
> > new character assignements). The reason for this was to create
> > pressure on registries to follow the rules.

Hi,
I am surprised that you indicate that certain ASCII domains will be 
illegal. I can see why unix people may be unhappy about "*.example.org".
I trust that most high ASCII will be allowed - "é" (e-acute) for example 
since software such as BIND9 handles them.

I am also concerned that a group discussing a protocol is suggesting that 
the protocol is designed to "create pressure on registries". A protocol is 
a stand-alone technical facility, not an instrument of policy.

Regards

Steve Dyer
CentralNic Ltd

---

From: Martin Duerst <[email-masked]>
Date: Sun, 03 Nov 2002 11:47:31 +0900

Hello Steve,

In an URI, *.example.org (e.g. http://*.example.org) is very
clearly illegal.

Also, I don't think it's a bad idea that a group that decides on the
syntax of a new protocol element give some thoughts about where the
rules should be checked, and where not, in order to assure consistent
deployment. The restrictions on domain names given by IDNA/nameprep/
stringprep are not in any sense policy issues, they are very clearly
technical issues.

Regards,    Martin.

At 09:40 02/11/01 +0000, Steve Dyer wrote:
>I trust that most high ASCII will be allowed - "$Bq(B (e-acute) for example

---

From: Martin Duerst <[email-masked]>
Date: Sun, 03 Nov 2002 15:49:18 +0900

At 23:40 02/10/31 +0100, Erik Nordmark wrote:
> > Ah, I see, of course the resolvers pass things through and then
> > pass the negative result back, so they don't actually reject it.
> > So now the sentence reads:
> >
> > However, such syntax should never be used, and will never be
> > resolved because no such domains will be registered.
>
>ok
>
>
>My point is that the idn-uri document is more restrictive in its
>syntax than the IDNA document. I don't know if this is a good idea
>or a bad idea, and we need to understand which type of idea it is.

URIs [RFC 2396] uses the same restrictions as STD3. I think it
therefore makes sense to also use these restrictions in this
upgrade.

On the other hand, while URI syntax contains these checks, URI
implementations are supposed to not check (because they would need
scheme-specific knowledge to do so, which would restrict
deployment of new schemes (even more than it's currently restricted)).

> > >The above statement says that for all domain names (note that the term
> > >"IDN" is defined to include the existing ASCII domain names)
> > >one should apply nameprep. This might be fine but it makes sense
> > >stating this explicitly. The ToASCII in IDNA does not apply nameprep
> > >to all-ASCII labels.
> >
> > The idea was simply to say: We RECOMMEND that you apply the IDNA rules
> > already when you create an URI. What these rules are is up to IDNA.
> > If IDNA says that their preparation of ascii-only labels is the
> > identity operation, then we recommend that you apply that (i.e. do
> > nothing), and not something else. If you see a way to make this clearer,
> > please tell me.
>
>It is the identity operation for ASCII-only labels (plus some checks
>that will reject certain labels).
>
>If you want to do that part, but not apply the Punycode step
>then I think you need to explicitly state that one should
>apply ToASCII without the punycode step (and other steps that don't
>make sense if there are any).

Yes, thanks, fixed.

> > >Which are the "any steps required as part of domain name resolution"
> > >above? I can't figure out to what it might refer.
> >
> > That's the nameprep and related checking that the client has to
> > do when it resolves a domain name. In IDNA terms, 'client' would
> > be easy to understand. But using the word 'client' in an URI context
> > doesn't work, so I tried to word around it. Any improved wording
> > appreciated.
>
>In that case I think refering to ToASCII would be best.

I have changed the parts in question to read:

For domain names containing non-ASCII characters, the legal domain
names are those for which the ToASCII operation
([IDNA], [Nameprep]; using the
unescaped UTF-8 values as input), with the flags "UseSTD3ASCIIRules"
and "AllowUnassigned" set, is successful. The URI resolver MUST
apply any steps required as part of domain name resolution by
[IDNA], in particular the ToASCII operation, with the above-mentioned
flags set.
URIs where the ToASCII operation results in an error should be
treated as unresolvable.

For domain names containing non-ASCII characters, the Nameprep
specification ([Nameprep]) defines some mappings,
which mainly include normalization to NFKC and folding to lower case.
When encoding an internationalized domain name in an
URI, these mappings SHOULD NOT be applied. It should be assumed
that the domain name is already normalized as far as appropriate.

I have explicitly changed it so that encoding does not require any
Nameprep steps, have clearly and repeatedly mentioned ToASCII,
and have added the necessary flags.

> > >Finally, is the intent that nameprep always be applied before characters
> > >are encoded in UTF-8? Then it makes sense stating that in the first real
> > >paragraph on page 4.
> >
> > No. In the context e.g. of IRIs, the conversion from an IRI to an URI
> > would not do nameprep.
>
>OK. Then it makes sense stating that explicitly somewhere.

Done above. I'm submitting a new draft, it should show up in
the directory in a few days (due to backlog).

Regards,    Martin.

---

From: Erik Nordmark <[email-masked]>
Date: Sun, 3 Nov 2002 20:07:10 +0100 (CET)

> deployment of new schemes (even more than it's currently restricted)).

Hmm, is this captured in some specification?

Clearly(?) URI parsers will have to enforce some syntax
checks e.g. in order to tell which part of a URI is
a domain name.

  Erik

---

From: John C Klensin <[email-masked]>
Date: Tue, 05 Nov 2002 13:23:53 -0500

--On Thursday, 31 October, 2002 23:40 +0100 Erik Nordmark
<[email-masked]> wrote:

>> If IDNA says that their preparation of ascii-only labels is
>> the identity operation, then we recommend that you apply that
>> (i.e. do nothing), and not something else. If you see a way
>> to make this clearer, please tell me.
> 
> It is the identity operation for ASCII-only labels (plus some
> checks that will reject certain labels).
> 
> apply ToASCII without the punycode step (and other steps that
> don't make sense if there are any).

Erik,

I tried to raise this argument in some of my remarks about IDNA,
but, as with many other things, it was apparently ignored by the
IESG.  Let me repeat it, in a different way, in this context.
One of the things that many of us feel is key to the success of
the Internet protocol suite is a minimum of profiling and
options for variations within a protocol.  That is obviously a
principle, not a hard rule: the stringprep-nameprep separation
arose as the better choice relative to having different
protocols make up their own mapping variations.  But, even in
that case, we haven't permitted profile variations _within_
IDNA: a user, site, or registry cannot select a different
stringprep profile, and that is a Good, probably necessary,
thing.  

But much the same argument applies to the type of use Martin
contemplates.   IDNA, as written, is _one_ protocol.  It is not
a toolkit for building other protocols, nor is it a a set of
profiles that other protocols can adopt.  Those two may be much
the same thing in practice.  As soon as we say "you should use
that operation from IDNA, but without some particular step" we
head down a slippery slope.  That is especially true because
IDNA contains (or appears to contain) a good deal of normative
text outside the particular of, e.g., ToASCII.  It is not clear
whether "...explicitly state that one should apply ToASCII
without the punycode step (and other steps..." includes some,
all, or none of those textual specifications.  And, whatever
choice is made, "do just what is done over there, except..." is
a poor way to do protocol specification and introduces some of
the worst properties of profiling without any of the benefits of
being explicit about what can, and cannot, be profiled.

      john

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Wed, 06 Nov 2002 00:44:54 +0100

I came in here just to say two things as a "user"
- the punycode thing seems OK but the DNS aspects are
   inadequate"
- the text you produced is not easily understandable.
I was said to be a troll for the second point. May be
could we also look at the second one?

>But, even in that case, we haven't permitted profile variations _within_ 
>IDNA: a user, site, or registry cannot select a different stringprep 
>profile, and that is a Good, probably necessary, thing.

"probably" necessary ???
At this date, this rises a lot of question.
All I found was a very early commitment to that in the process.
I did not read everythng yet.
Any reason why it is "certainly" necessary?
(a part from obvious non technical reasons)

>being explicit about what can, and cannot, be profiled.

You point is accepted. But to say that applying ToASCII to
a valid ASCII name is equivalent to do nothing does not
removes the need to use ToASCII, but helps understanding
that this is what actually happens. The way it is written leaves
many questions open.
jfc

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 6 Nov 2002 15:22:19 +0100 (CET)

> being explicit about what can, and cannot, be profiled.

Which is why I welcome folks that try to use IDN(A) is new protocols
to share their experience. That way we can collectively learn about the 
actual issues that folks run into and use that to improve on the way 
IDNA is specified.

In addition to the URI side there is also the registry side where I
think there is active development today. And I hope that somebody will
try to explore new protocols that will not use an ACE for the IDNs
so that we can collectively understand the issues in that space as well.

  Erik

---

From: "JFC (Jefsey) Morfin" <[email-masked]>
Date: Wed, 06 Nov 2002 17:04:25 +0100

>so that we can collectively understand the issues in that space as well.

Would there be a test nameprep C code to help with test
registry tools? I suppose there are some traps to avoid.
jfc

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Thu, 7 Nov 2002 14:19:09 -0800

In draft -03, it says:

     For domain names containing non-ASCII characters, the Nameprep
     specification ([Nameprep]) defines some mappings, which mainly
     include normalization to NFKC and folding to lower case.  When
     encoding an internationalized domain name in an URI, these mappings
     SHOULD NOT be applied.  It should be assumed that the domain name is
     already normalized as far as appropriate.

Why the "SHOULD NOT"? What's wrong with a system that creates URIs doing that?

An alternate wording for the last two sentences would be:

     When encoding an internationalized domain name in an URI,
     these mappings do not need to be applied if the domain name is
     already normalized as far as appropriate.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: "Adam M. Costello" <[email-masked]>
Date: Fri, 8 Nov 2002 06:25:09 +0000

Paul Hoffman / IMC <[email-masked]> wrote:

>     encoding an internationalized domain name in an URI, these
>     mappings SHOULD NOT be applied.  It should be assumed that the
>     domain name is already normalized as far as appropriate.
>
> Why the "SHOULD NOT"?

Indeed, I also see no need for that recommendation.

>     When encoding an internationalized domain name in an URI, these
>     mappings do not need to be applied if the domain name is already
>     normalized as far as appropriate.

Am I supposed to perform some test to determine whether the domain name
is already normalized as far as appropriate?  No.  I find that clause
confusing and unnecessary.

I think the only point of this paragraph is that domain names in URIs
are not necessarily Nameprepped.  That's all it needs to say.

Draft -03 says:

> For domain names containing non-ASCII characters, the legal
> domain names are those for which the ToASCII operation ([IDNA],
> [Nameprep]; using the unescaped UTF-8 values as input), with the flags
> "UseSTD3ASCIIRules" and "AllowUnassigned" set, is successful.  The
> URI resolver MUST apply any steps required as part of domain name
> resolution by [IDNA], in particular the ToASCII operation, with the
> above-mentioned flags set.

URI resolvers should indeed set AllowUnassigned, but URI resolvers
aren't the only things that use URIs.  Consider a program that creates
HTML documents.  The domain names in those URIs are stored strings, and
Stringprep requires that unassigned code points be prohibited in stored
strings, so AllowUnassigned would have to be unset in that situation.

Finally, I'd like to point out an important caveat:  Even if the URI
generic syntax is updated to allow non-ASCII characters (escaped) in
the host field, that doesn't mean you can actually put non-ASCII domain
names into any URI you please.  The IDNA rules still apply.  If you know
that a URI is occupying an IDN-aware slot (for example, if it appears
in a new version of HTML that refers to the IDNA spec, or if it appears
in an old-HTML document but new HTTP features are used to negotiate
IDN-awareness), then you're free to put non-ASCII domain names (escaped)
into the URI.  But otherwise IDNA section 3.1 requirement 2 applies, and
the domain name must contain only ASCII characters.  The rationale is to
prevent non-ASCII domain names from falling into the unwitting hands of
old software that will choke on them.

It might be prudent for the idn-uri document to remind the reader of
this.

AMC

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 8 Nov 2002 14:54:17 +0100

On Wed, Nov 06, 2002 at 05:04:25PM +0100,
 JFC (Jefsey) Morfin <[email-masked]> wrote 

> registry tools? I suppose there are some traps to avoid.

There are some tools <URL:http://www.i-d-n.net/#tools>

---

From: "Adam M. Costello" <[email-masked]>
Date: Sat, 9 Nov 2002 02:20:01 +0000

I wrote:

> If you know that a URI is occupying an IDN-aware slot, then you're
> free to put non-ASCII domain names (escaped) into the URI.  But
> otherwise IDNA section 3.1 requirement 2 applies, and the domain name
> must contain only ASCII characters.

I was assuming that the URI occupies a slot of some sort.  If it is not
in a slot at all (for example, it appears in a plain text message body),
then requirement 2 does not apply, and you are free to put non-ASCII
domain names in it.

I have a question about the rationale behind allowing non-ASCII domain
names in URIs.  A URI containing a non-ASCII domain name seems to be
useful only for software that is IDN-aware but IRI-unaware.  If the
software is IDN-unaware, it will choke on the URI.  If the software is
both IDN-aware and IRI-aware, then you might as well use an IRI rather
than a URI, to avoid the ugly %-escaped UTF-8.

My question is, do we really expect to see much, if any, software that
is IDN-aware and IRI-unaware?  IDNs and IRIs are being introduced at
roughly the same time, and both deal with internationalization.  Isn't
it reasonable to assume that almost all web-related software will either
support both, or support neither?  Why complicate things by introducing
a new IDN-supporting URI type if it's not useful?  URIs with ACE are
ugly, but work with old software.  IRIs with non-ASCII domain names are
pretty, but require new software.  URIs with %-escaped non-ASCII domain
names are both ugly and require new software, so what's the point?

Here's another model to consider:

All generic URIs (URIs beginning with scheme://) continue to be
IDN-unaware, and therefore the host field must contain only ASCII
characters.  Generic IRI syntax (any IRI beginning with scheme://) is
IDN-aware, and therefore non-ASCII names are allowed in the host field
(escaped).

I don't think there will be any problem converting generic IRIs to
generic URIs, even if domain names happen to appear in other places
(the path or query-string).  In the generic syntax, the host field is
the only place a domain name can appear that has client-side semantics.
If domain names happen to appear anywhere else in the URI, they must
be interpreted by the server, right?  If non-ASCII domain names appear
in a generic IRI outside the host field, that must mean the server is
IDN-aware, and therefore those domain names don't need to be converted
to ACE if the IRI is converted to a URI.  The server will still be
IDN-aware regardless of whether it is accessed via a URI or an IRI.
The host field is the only domain name that needs to be ToASCII'd when
converting a generic IRI to a URI.

For converting non-generic IRIs to URIs, if you know the scheme, you
can extract the domain names and apply ToASCII to them.  If you don't
know the scheme, then you have no way of knowing if the URI you produce
might contain non-ASCII domain names that old software will choke on.
Whether we call the URI valid or invalid doesn't change the fact of
whether it breaks old software.  Calling the URI valid is like blaming
the old software for conforming to yesterday's standard.  Calling the
URI invalid is like blaming the convertor for performing a conversion
without enough knowledge to do it safely.  The latter makes more sense
to me.

How many non-generic URI schemes containing domain names are there?
Suppose we require that after the introduction of IRIs all new
non-generic schemes containing domain names must either be IDN-aware
for both URIs and IRIs, or IDN-unaware for both URIs and IRIs.  Then
IRI-to-URI convertors won't need to do anything special for those
schemes.  The number of schemes that convertors need to know about will
never increase.

AMC