Subject: [weirds] Unicode search - alternative approach

From: Guillaume Leclanche <[email-masked]>
Date: Mon, 25 Nov 2013 22:26:52 -0500

2013/11/25 Hollenbeck, Scott <[email-masked]>

> > So my proposal is :
>
> > 1. we document the different search variants, and maybe write them in
> some formal language
>
> > 2. with each search, return the i18n rules that were used to do the
> search in the response (using the formal language ?)
>
> > 3. define a segment that can be used by the client to request the
> default/available search rules on the server
>
> > 4. allow the client to tune the way the server is doing its search
>
> > I don't think steps 2-3 are too difficult to implement. Step 4 looks
> definitely more challenging as there would be sanity/security/performance
> implication.
>
> >
>
> > What do you think ?
>
>
>
> This sounds more complicated than the text I proposed in response to Ernie
> and James, and if people have an issue with my proposal I can’t see this
> being supported, either.
>

It is a different approach. It does not mandate any searching rules to the
server. It simply requires the server to document and provide its rules.

The point is that it looks difficult for the working group to converge on
defining the MUSTs or SHOULDs that would be written down for i18n matching.
It looks to me more reasonable to let every server do whatever it wants and
to make it mandatory to return how it has done the matching for a given
search request.

Guillaume

---

From: "John Levine" <[email-masked]>
Date: 26 Nov 2013 03:53:33 -0000

>server. It simply requires the server to document and provide its rules.

Could you check with some TLD operators, such as the ones on this
list, and see if they'd be willing to do that?  I'd be surprised.

R's,
John

---

From: Andy Newton <[email-masked]>
Date: Tue, 26 Nov 2013 13:34:11 +0000

From: Guillaume Leclanche <[email-masked]<mailto:[email-masked]>>
Date: Monday, November 25, 2013 10:26 PM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: [weirds] Unicode search - alternative approach

It is a different approach. It does not mandate any searching rules to the server. It simply requires the server to document and provide its rules.

Is there a formal syntax for such rules definitions?

-andy

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 26 Nov 2013 08:47:50 -0500

> > search in the response (using the formal language ?)

[…]

> server. It simply requires the server to document and provide its rules.

Well, you could use the mechanism that Kim Davies has proposed for
capturing label generation rules that apply to a zone.  You could
also, I think, provide a link to the relevant repository.  I don't
think you could practically isolate the rule in question and transmit
it. 

(4) is, I think, unwise.  You're trying to look things up in the
repository.  The repository's generation rules are the only search
rules that ought to apply, I think.  I'd like a counter-argument if
that's to be pursued.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Simon Perreault <[email-masked]>
Date: Tue, 26 Nov 2013 10:24:20 -0500

> list, and see if they'd be willing to do that?  I'd be surprised.

If it can be "implemented" with a URL pointing to an informal 
description of how search is done, then I don't see why not.

"Do whatever you want server-side, but at least document it somehow."

We could informatively point to the LGR draft as a good starting point.

Simon
-- 
DTN made easy, lean, and smart --> http://postellation.viagenie.ca
NAT64/DNS64 open-source        --> http://ecdysis.viagenie.ca
STUN/TURN server               --> http://numb.viagenie.ca

---

From: Ernie Dainow <[email-masked]>
Date: Tue, 26 Nov 2013 11:29:52 -0500

I'm not sure I understand the objection to requiring normalization for
lookup and search. Registration of a domain requires it (RFC 5891), plus a
lot more onerous checking down to the character (RFC 5892).

Organizations that are responsible for domain registrations are generally
the same ones that will be providing RDAP servers. A requirement that RDAP
lookup and search must normalize to NFC seems simple by comparison.

-Ernie

On Tue, Nov 26, 2013 at 10:24 AM, Simon Perreault <
[email-masked]> wrote:

> If it can be "implemented" with a URL pointing to an informal description
> of how search is done, then I don't see why not.
>
> _______________________________________________
> weirds mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/weirds
>

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 26 Nov 2013 12:07:24 -0500

> lot more onerous checking down to the character (RFC 5892).

I don't think anyone is objecting to restricting the searchable space
to U-labels (if they are, then I'm misreading).  I agree that the only
thing to be looked up are U-labels.

But that's not what "variants" are about (despite the confusion of
some whenever that word appears).  If "variants" means anything, it's
a case where there are two different U-labels (valid ones) that are,
according to registry policy, "the same" in some sense.

The usual example that gets trotted out at this point is Chinese.
Traditional and Simplified Chinese characters are sometimes but not
always different.  They are conceptually the same character, however,
and I have been repeatedly assured that many if not most literate
Chinese readers can work out what is meant by one of these writing
systems even if they're normally users of the other.  In any case,
this is the reason why there are two "IDN" top-level domain labels for
"China": 中国 (xn--fiqs8s) and 中國 (xn--fiqz9s).  The idea of
variants handling is that there needs to be some way to tell that
these names are linked in some important way.  (This sort of thing has
interesting side effects.  For instance, today in .ca hoe.ca and hœ.ca
turn out to be variants, because of a policy that CIRA has in an
effort to accommodate the use of œ.  This produces occasionally odd
results like hœ, and çîrà.ca, and so on, that are consistent with their
policy.)

The proposal that started this was that the RDAP server return the
additional record (or a see also reference?  Or something?) along with
the rules by which the variant is generated.  I was arguing (and I
think Simon is arguing also) that you couldn't include those rules in
the response, at least not practically, but you could return a pointer
to the policy that determined them to be variants.  This has the nice
advantage of forcing the RDAP server operator to determine the meaning
of "variant" for itself; for our purposes, two names are variants of
one another just in case the server operator says so and can hand you
a URL from which you can learn how that determination was made.  (Note
that the URL could point to a web page that says, "We will call any
two names variants if you pay us $1 million and your first born son."
We don't care for the purposes of the protocol.)

I think the target policy should probably be expressed using
http://tools.ietf.org/html/draft-davies-idntables-05, but I don't
think that ought to be a MUST.  How come 2119 doesn't have IT WOULD BE
NICE IF?

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Ernie Dainow <[email-masked]>
Date: Tue, 26 Nov 2013 14:49:28 -0500

> thing to be looked up are U-labels.

I was responding to Scott's last proposed wording for search,

"In order to provide more consistent pattern matching behavior servers SHOULD normalize Unicode client inputs using the same normalization form the server uses to store Unicode data."

> according to registry policy, "the same" in some sense.
At the beginning of this "alternative approach" sub-thread, I 
interpreted "different search variants" to be about the  variations of 
search processing, i.e. what type of normalization is done by the 
server. I thought we had arrived at a conclusion that we would not 
provide a way to search for variant names, at least in this version of RDAP.

If search processing requirements are should rather than must, 
documentation of what the server does should be provided. But it is 
awkward to require a client to perform a search and then look for a 
pointer to documentation in the response. Any limitations or options in 
general beyond what is in the RDAP spec should just be clearly listed on 
the Help page so it is all in one place, rather than having pointers to 
different documentation scattered across various responses.

-Ernie

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Tue, 26 Nov 2013 20:04:25 +0000

> -----Original Message-----
> From: weirds [mailto:[email-masked]] On Behalf Of Ernie Dainow
> Sent: Tuesday, November 26, 2013 2:49 PM
> To: Andrew Sullivan; [email-masked]
> Subject: Re: [weirds] Unicode search - alternative approach
> 
> 
> "In order to provide more consistent pattern matching behavior servers
> SHOULD normalize Unicode client inputs using the same normalization
> form the server uses to store Unicode data."

My concern was that the rules for domain names might not apply to other data (like entity names) stored at the registry. I tried to generalize the text to allow whatever normalization form is appropriate for the context.

Scott

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 26 Nov 2013 15:18:31 -0500

> My concern was that the rules for domain names might not apply to other data (like entity names) stored at the registry. I tried to generalize the text to allow whatever normalization form is appro…

Oh, I get it.  I think that's too complicated.  The precis WG is in
any case recommending NFC everywhere, and that seems like extremely
good advice to me.  We don't want the KC forms because of ambiguity.
And NFC is more compact for network transmission than NFD.  So why not
just require all queries to be NFC normalized and all servers to
normalize to NFC before matching on normalized-to-NFC data?  

(The actual effect of this of course will be that people normalize
their databases with NFC before storing.)

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 26 Nov 2013 16:13:38 -0500

On Tue, Nov 26, 2013 at 02:49:28PM -0500, Ernie Dainow wrote:
> interpreted "different search variants" to be about the  variations
> of search processing, i.e. what type of normalization is done by the

Yeah, I thought so too, but the discussion terms confused me.  Anyway,
if we're talking about different kinds of normalization then we're
going to plunge right down this rathole.  

The only possible reason I can think of for not requiring
NFC-normalized UTF-8 for every term (on the network -- what a client
does with preprocessing mapping and so on is up to the client) is if
someone claims they need to publish data using scripts not available
in Unicode.  As soon as the Ancient Mayan-using registry shows up (and
there's an argument to be made that it is possible, though unlikely),
I'll cheerfully give in.  Until then, I can think of no reason to
enable anything else, and plenty of reasons not to.

If we _really_ want to explore this swamp, then I suggest we have a
look at the current precis drafts on framework and mapping before we
go futher.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: "John Levine" <[email-masked]>
Date: 26 Nov 2013 22:52:39 -0000

>My concern was that the rules for domain names might not apply to other data (like entity names) stored at the
>registry. I tried to generalize the text to allow whatever normalization form is appropriate for the context.

They probably won't.  The PRECIS WG is developing profiles for things
like handles, which are unlikely to be identical to the rules for
U-labels.

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 27 Nov 2013 06:55:28 -0500

> like handles, which are unlikely to be identical to the rules for

Local mappings are unlikely to be the same, sure.  For instance, there
might be handling for i and dotless-i that is peculiar to Turkish.
You might be able to do something about that in, say, address searches
if you somehow knew that the registrant country was Turkey (perhaps by
a previous lookup).

In general, RFC 5198 recommends NFC.  There is a width-mapping issue,
however, with NFC, and it might be necessary to deal with that.
Indeed, the more I think about this the less I can see why people
interested in all of this shouldn't go read
ools.ietf.org/html/draft-ietf-precis-framework-12 and probably the
mappings draft too.  This work should probably be a profile of that stuff.

If the concern is for the case where the data in the repository is
unnormalized, then I suggest the effort is probably doomed unless the
repository's RDAP server normalizes everything as part of its search
strategy (in pseudo-database language, WHERE nfc(submit_term) =
nfc(database_column)).  I _suppose_ you could make an argument for
compatibility normalization under some circumstances there, but I
think it'd be much more reliable to follow the strategy offered by
precis: mappings on storage and before lookup, and use a consistent
profile (probably of IdentifierClass) for everything.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Wed, 27 Nov 2013 12:12:23 +0000

> From: weirds [mailto:[email-masked]] On Behalf Of Andrew
> Sullivan
> Sent: Tuesday, November 26, 2013 3:19 PM
> To: [email-masked]
> > My concern was that the rules for domain names might not apply to
> other data (like entity names) stored at the registry. I tried to
> generalize the text to allow whatever normalization form is appropriate
> for the context.
> >
> 

Have a look at the text from rdap-query-07:

"Because Unicode characters may be combined with another Unicode character or characters, partial matching requires that a server maintain a list of valid character combinations to be considered a match.  When comparing DNS U-labels, servers SHOULD use the code points specified in [RFC5892] to determine partial matches.  When comparing entity names, servers SHOULD use the normalization rules and code points specified by [I-D.ietf-precis-nickname] to determine partial matches"

This was all changed in -08 as a result of the Vancouver in-room discussion. No one expressed support for NFC. So back to the list, and we're again ping-ponging between "normalize" and "don't normalize". How could the text above be changed to work for you?

I do have a concern about adding a normative dependency to a draft that isn't ours. Is the nicknames document one of the "profile documents" mentioned in the precis charter? If so, how close is it to completion?

Scott

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 27 Nov 2013 07:20:17 -0500

On Tue, Nov 26, 2013 at 04:13:38PM -0500, Andrew Sullivan wrote:
> Yeah, I thought so too, but the discussion terms confused me. 

I just reviewed the minutes from Vancouver, and we did not agree to
that.  (I was late for that meeting because of another obligation, so
that is part of my confusion.  Apologies.)

The minutes say that John Klensin was going to send some text.  Is
that happening?  

In any case, I have some quibbles with
draft-hollenbeck-weirds-rdap-search-02 section 4.  First, and most
strongly, I do not believe that the text "When comparing DNS U-labels,
servers SHOULD use the code points specified in [RFC5892] to determine
partial matches," should be left.  People will take that to mean "the
list of code points in the appendix of RFC 5892", and that can't be
right because (1) that appendix isn't normative and (2) it's for a
specific version of Unicode.  I think what is wanted instead is
"servers MUST use only those code points consistent with the rules
specified in [RFC5892]," or something like that.  (The more elegant
way in English to say this is "servers may only use. . ." but RFC 2119
doesn't differentiate MAY and MAY ONLY, so we're stuck with
"MUST. . .only".)  Second, I don't understand why this is using the
rules in precis-nickname.  Among other things, nickname works on
FreeformClass, which is already completely unsuitable for U-labels.

I am prepared to build a precis profile for all this if it would help.

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Wed, 27 Nov 2013 14:12:55 +0000

> Sent: Wednesday, November 27, 2013 7:20 AM
> To: [email-masked]

Done and included in rdap-query-08.

> I am prepared to build a precis profile for all this if it would help.

Text from my individual submission was been pulled into rdap-query-08 post-Berlin. Look at query-08, please.

Scott

---

From: Marc Blanchet <[email-masked]>
Date: Wed, 27 Nov 2013 10:09:24 -0500

> mappings draft too.  This work should probably be a profile of that stuff.

During the very first weirds BOF, I said that a precis profile might be needed. ;-)

But:
a) there has been some voiced concerns that we should not specify anything on the backend side of the servers.
b) while we can specify client behavior (such as do normalization before sending), I put the point that all the guys in the wild doing javascript and else, will most likely do not prepare anything and will just spit out the bytes they receive from the user. 
c) because of the "reality check" of b) and that there is no appetite for a), then I said on the mic during last ietf and on the mailing list that we should then provide some "hints" for the server implementations, not normative, but info and pointers.
d) the corrolary being that the queries (i.e. the protocol part, which is in our scope) would be non-prepared strings.

I would love to be more prescriptive but I think it is a lost battle.

Marc.

---

From: Andy Newton <[email-masked]>
Date: Wed, 27 Nov 2013 19:40:06 +0000

On 11/27/13 10:09 AM, "Marc Blanchet" <[email-masked]> wrote:

>a) there has been some voiced concerns that we should not specify
>anything on the backend side of the servers.
>b) while we can specify client behavior (such as do normalization before
>sending), I put the point that all the guys in the wild doing javascript
>and else, will most likely do not prepare anything and will just spit out
>the bytes they receive from the user.
>c) because of the "reality check" of b) and that there is no appetite for
>a), then I said on the mic during last ietf and on the mailing list that
>we should then provide some "hints" for the server implementations, not
>normative, but info and pointers.
>d) the corrolary being that the queries (i.e. the protocol part, which is
>in our scope) would be non-prepared strings.
>

I agree with this.

-andy

---

From: Joseph Yee <[email-masked]>
Date: Thu, 28 Nov 2013 09:26:51 -0500

> >I would love to be more prescriptive but I think it is a lost battle.
I agreed with this too.

There are few more 'reality checks' beyond users' query and search
processing.

(1) There are existing data in various form.  Some kept the original form,
some lost the accent mark, some were translated/transliterated.
(2) The name registries do not collect user data directly, users and/or
registrars may 'alter' the data before it reached registries.

It's easy to detect if the data was normalized or not, but very hard to
tell if it's the 'true presentation'.

So search on entity names other than domain will always be fuzzy and
inconsistent across.  It is easier for new registries to implement and
enforce what Andrew mentioned in earlier post.  We should make
recommendation in the draft, but making it MUST would not improve the
situation. It might even hurt adoption, IMO.

>I would love to be more prescriptive but I think it is a lost battle.
One would also need to be prescriptive on collect-audit-revise on data
store and collection to fix (1) and (2) to 'win'.  And that is beyond
WEIRDS' control.

Joseph

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 28 Nov 2013 10:22:54 -0500

> tell if it's the 'true presentation'.

This thread (and the above in particular) makes me think we are
conflating more than one thing.  Or anyway, that I am confused, which
I guess is normal.

The _normalization_ question to me is just this: given a particular
search string, how is it handled prior to transmission and when it is
received by the server?  If we do not specify some normalization
treatment of that, then this search functionality is going to be
totally useless.  Different systems normalize in different ways, and
if there isn't a standard way to normalize that everyone's going to
have to guess.  That's what the normalization forms are for.

That is completely unrelated to the question of whether the data is
exactly "correct" for some value thereof.  

In other words, it's none of our buisness whether the registry has
stored école or ecole.  And I do not think we should try to specify
rules that make école and ecole match (or worse, Goethe and Göthe).
It is certainly going to make for a poor protocol, however, if people
have to guess whether to send "école" as NFC or NFD or NFKC or NFKD.
Or, for that matter, Win-1252 or UTF-8.  I think we should write down
what the wire behaviour is there.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Marc Blanchet <[email-masked]>
Date: Thu, 28 Nov 2013 10:30:22 -0500

> have to guess.  That's what the normalization forms are for.

I agree with that. And we have been saying it.

> what the wire behaviour is there.

right. my initial proposal a while ago was that the client do normalization, or more generally, prepare the string. But given the use case of weirds/whois, which does include people doing javascript web apps, we ought to care about the fact that the strings will be sometimes unprepared at all.  That is where I think we should not be blind.  So we could prescribe something for the client, but if the client does nothing, what happens?  The simple answer may be "results are unpredictable".  That is why I was proposing the idea of telling the server implementations about what they can expect and what could they do to mitigate the problem.

Marc.