Subject: RE: [idn] Re: stringprep and unassigned code points

From: Yves Arrouye <[email-masked]>
Date: Thu, 25 Oct 2001 10:23:58 -0700

> I must be extremely bad at expressing myself. This is now the third time
> since monday I explain how things work.

No, no, it's just me.

> > Now
> > let's suppose Unicode encodes something new, for which a new Nameprep is
> > released (because some additions to the mapping table are needed).
> 
> A mapping can only be from an earlier unassigned codepoint.

My case exactly.

> > At
> > this point, the DNS servers may contain the ACE(Nameprep(names in this
> > new thing)).
> 
> ACE have absolutely nothing to do with this. This is a stringprep issue.

I did not say it had anything to do with ACE. I used DNS and IDNA as an
example of the use of Nameprep, a stringprep profile.

> Only nameservers which do understand the new version of stringprep can
> have
> these names.
> 
> > And applications get these new codepoints,
> 
> What do you mean by "get these new codepoints"?

I mean that my mail client, for example, or my Web browser, after an OS
update, can now see these new code points, from an input method, for
example, whilst they were not generated before.

> > but because they
> > do not have the appropriate mapping table, they will pass ACE(Old
> > Nameprep(names in this new thing))
> 
> No, because the codepoint is unassigned in old version of stringsprep, and
> because of that MUST NOT be used.

See, that where there is where I get confused (must be me). The stringprep
says:

"Applications creating queries MUST treat U code points as if they were
AO when preparing the query to be entered in the process described by a
profile of stringprep. Those applications MAY optionally have a
preprocessor that provide stricter checks: treating unassigned code
points in the input as errors, or warning the user about the fact that
the code point is unassigned in the version of a profile that the
software is based on; such a choice is a local matter for the software."

And

" All code points not assigned in the character repertoire named in the
stringprep profile are called "unassigned code points". Stored strings
using the profile MUST NOT contain any unassigned code points. Queries
for matching strings MAY contain unassigned code points. Note that this
is the only part of this document where the requirements for queries
differs from the requirements for stored strings."

I take it from "MUST treat U code points as if they were AO when preparing
the query" that these code points MAY be used.

What is wrong (since you say I am wrong) with my reading of these two
paragraphs?

YA

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Tue, 30 Oct 2001 18:25:59 -0800

At 4:34 PM -0800 10/30/01, Yves Arrouye wrote:
>I will be glad to understand that I am wrong in saying that this scenario
>shows that some queries from older applications won't have the expected
>result against more recent servers (older/recent as far as Stringprep goes).
>But can you please point out where my reasoning is then?

"won't have the expected result" is quite different than "breaks". In 
your scenario, the application has no idea what the expected result 
is. The application simply knows that it is entering a character that 
is (according to the version of nameprep it is using) unassigned. 
Thus, a response of "no such name" makes perfect sense to the 
application. Remember, the client might be talking to a server that 
is using the same version of nameprep, and such a server will 
certainly generate "no such name" for any name that contains an 
unassigned character.

There is no expectation that entering random unassigned characters 
will work. As you point out, those characters may be mapped or they 
may be prohibited in the version of nameprep that is running on the 
server. Further, even though the server might be running newer 
software, the character might still be unassigned.

In short, a client has no particular "expected result" for a query 
with unassigned characters in it.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Yves Arrouye <[email-masked]>
Date: Tue, 30 Oct 2001 22:11:08 -0800

> > No. I don't think so. Remember this section 6 of stringprep, that says
> > that for queries (which is what I believe you are talking about now)
> > applications should treat U code points the same as AO? Well U+20000
> > being U, it will be passed on, since this second application is making
> > queries.
> 
> Ok. I understand where the text is unclear. What I am talking about is a
> query, yes. The idea is that the application which is generating/sending
> the query is doing the stringprep, has the old version, and reject the
> codepoint. So the codepoint is not even sent to the server (which _DO_
> understand the new codepoint).

That is actually against Stringprep section 6.1 which states (note the
MUST):

profile of stringprep."

Which makes sense since it provides for a very good chance that the
application will be able to cope with later revisions of Unicode (modulo the
caveats that started this whole thread).

> The important thing is that nothing unassigned end up in the zonefile.

Agreed. Stringprep is clear on that, no ambiguity there. I think it is also
clear on the query (the part I quoted above). The ambiguities are in the
assumptions that are then made and that can be broken (eg: passing
unassigned code points is a 100% guarantee of working with future
registrations; it is not if mapping tables change but is in the other
cases). That is what will benefit from some clarifications.

YA