Subject: [weirds] Comments on draft-ietf-weirds-json-response-05

From: Ernie Dainow <[email-masked]>
Date: Tue, 20 Aug 2013 12:09:50 -0400

9.  Responding To Searches

The domains array returned from a search

"domains" :
      [
        {
          "handle" : "1-XXXX",
          "name" : "1.example.com",
          ...
        },

seems inconsistent with the request in draft-ietf-weirds-rdap-query-06 
and with domain responses already defined. It should have "ldhName" or 
"unicodeName" (or both) instead of "name".

This response would match the search query if the latter is changed to 
domains/?name=<search pattern>

Domains is a new array, not used elsewhere. But "nameservers" and 
"entities" arrays returned from search are already defined, as quite 
large structures. Does this mean the server can return any subset of 
information from these structures in a search response? This will result 
in a rather lengthy response for an entities search by name, since the 
name ("fn") is embedded inside the vcardArray. Some examples here would 
help. I think simple new arrays specific for "nameservers" and 
"entities" search results might be preferable.

-Ernie

---

From: MICHAEL YOUNG <[email-masked]>
Date: Tue, 20 Aug 2013 13:56:09 -0400

Hi Ernie,

Do you have any examples of what you propose for a new array specific to nameservers and entities?

-M

On 2013-08-20, at 12:09 PM, Ernie Dainow <[email-masked]> wrote:

> seems inconsistent with the request in draft-ietf-weirds-rdap-query-06 and with domain responses already defined. It should have "ldhName" or "unicodeName" (or both) instead of "name".
> 
> This response would match the search query if the latter is changed to domains/?name=<search pattern>
> 
> Domains is a new array, not used elsewhere. But "nameservers" and "entities" arrays returned from search are already defined, as quite large structures. Does this mean the server can return any subset of information from these structures in a search response? This will result in a rather lengthy response for an entities search by name, since the name ("fn") is embedded inside the vcardArray. Some examples here would help. I think simple new arrays specific for "nameservers" and "entities" search results might be preferable.
> 
> -Ernie
> 
> _______________________________________________
> weirds mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/weirds

MICHAEL YOUNG
[email-masked]

---

From: Andy Newton <[email-masked]>
Date: Tue, 20 Aug 2013 18:00:24 +0000

>"unicodeName" (or both) instead of "name".

This was a cut&paste error. Thanks.

>"entities" search results might be preferable.

Would 'entitySearchResults', 'nameserverSearchResults', and
'domainSearchResults' work better?

-andy

---

From: "Linlin Zhou" <[email-masked]>
Date: Wed, 21 Aug 2013 17:48:47 +0800

section 4 search processing

I don't quite understand " Servers SHOULD NOT partially match combinations
of Unicode characters where a Unicode character may be legally combined with
another Unicode character or characters. " What's the problem if a server
handles this search? Can someone give me an example?

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Ernie Dainow
> Sent: Wednesday, August 21, 2013 12:10 AM
> To: [email-masked]
> Subject: [weirds] Comments on draft-ietf-weirds-json-response-05
> 
> seems inconsistent with the request in draft-ietf-weirds-rdap-query-06 and
> with domain responses already defined. It should have "ldhName" or
"entities"
> arrays returned from search are already defined, as quite large
structures.
> Does this mean the server can return any subset of information from these
> structures in a search response? This will result in a rather lengthy
response
> for an entities search by name, since the name ("fn") is embedded inside
the
> vcardArray. Some examples here would help. I think simple new arrays
specific
> for "nameservers" and "entities" search results might be preferable.
> 
> -Ernie
>

---

From: Ernie Dainow <[email-masked]>
Date: Wed, 21 Aug 2013 10:31:37 -0400

> 'domainSearchResults' work better?
Using existing objects is generally a good idea, but the search results are
sufficiently different that using new names such as this would make the
distinction clear.

-Ernie

---

From: Andy Newton <[email-masked]>
Date: Wed, 21 Aug 2013 15:33:14 +0000

From: Ernie Dainow <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, August 21, 2013 10:31 AM
To: Andrew Newton <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [weirds] Comments on draft-ietf-weirds-json-response-05

Using existing objects is generally a good idea, but the search results are sufficiently different that using new names such as this would make the distinction clear.

I agree. Some software frameworks might work better with different names. Thanks for pointing this out.

-andy

---

From: Andy Newton <[email-masked]>
Date: Wed, 21 Aug 2013 15:39:07 +0000

>handles this search? Can someone give me an example?

Linlin,

Let me start by saying that I'm not an I18N expert. I have talked to
several IETFers with more experience in internationalization issues, and
the advice given was to avoid interoperability issues where combining
characters can be confused with character of single code points but with
the same semantic meaning.

Honestly, I don't know what harm would come from just letting the server
respond with its own rules. Nor do I know if the rules change depending on
registry policy. This may be the case that the server responds with what
the server has in a similar way we have done with lookups.

-andy

---

From: "Linlin Zhou" <[email-masked]>
Date: Thu, 22 Aug 2013 15:42:31 +0800

> From: Andy Newton [mailto:[email-masked]]
> Sent: Wednesday, August 21, 2013 11:39 PM
> To: Linlin Zhou; 'Ernie Dainow'; [email-masked]
> >I don't quite understand " Servers SHOULD NOT partially match
> >combinations of Unicode characters where a Unicode character may be
> >legally combined with another Unicode character or characters. " What's
> >the problem if a server handles this search? Can someone give me an
> >example?
> 
> Linlin,
> 
several
> IETFers with more experience in internationalization issues, and the
advice
> given was to avoid interoperability issues where combining characters can
be
> confused with character of single code points but with the same semantic
> meaning.

I asked someone about the combining Unicode characters, he gave me an
example such as "ç" (Latin small letter c with cedilla). It can be encoded
as U+00E7 or U+0063 U+0327 (a regular "c" followed by a combining cedilla).
I don't know whether this is the right scenario discussed in this section.

If I have a domain contains this letter, for example façade.com. A user may
partially search faç*.com, the database find the record that Unicode name
starts with faç, as you said in a similar way like lookups. While the text
in this section makes me feel that a user may possibly search U+0066 U+0061
U+0063*.com or U+0066 U+0061 U+0063 U+0327*.com. Since Chinese Unicode
characters don't have this situation, I want to know more about this issue
and how to handle it by the server.

Anyway, thanks Andy.

the
> server has in a similar way we have done with lookups.
> 

Regards,
Linlin

---

From: Pieter Vandepitte <[email-masked]>
Date: Thu, 22 Aug 2013 14:44:51 +0200 (CEST)

Linlin and Andy, 

Isn't the solution to simply perform some normalization before searching (see https://en.wikipedia.org/wiki/Canonical_equivalence ) and make sure the data in your database is normalized the same way (e.g. NFC, NFD, NFKC or NFKD)? 

kr, 

Pieter 

Pieter Vandepitte 
Software Engineer 
+32 16 29 89 27 
www.dnsbelgium.be 

----- Original Message -----

> From: "Linlin Zhou" <[email-masked]>
> To: "Andy Newton" <[email-masked]>, "Ernie Dainow" <[email-masked]>,
> [email-masked]
> Sent: Thursday, August 22, 2013 9:42:31 AM

---

From: Ernie Dainow <[email-masked]>
Date: Thu, 22 Aug 2013 17:01:17 -0400

Here is an example of doing a search 'abc*' against 'abçd' using each 
normalization type:
NFC  -1
NFKC -1
NFD   0
NFKD  0

The first two fail because ç is normalized to a single Unicode point, 
whereas the last two (Decomposed forms) succeed because ç is normalized 
to two Unicode points, c followed by cedilla.

In terms of usefulness to an RDAP client, I think a search result of 
'abçd' is more useful than no result. I believe that taking this 
approach would mean that we support partial match searches.

It is important that the normalization form used by the server is 
specified as part of the spec. Otherwise, clients will get different 
results depending on the server that is queried for the search.

(Python code to do this test is very simple. Note that it doesn't matter 
in what form the server has stored the data.)
import unicodedata
a = u'abçd'
for normtype in ('NFC', 'NFKC', 'NFD','NFKD'):
     anorm = unicodedata.normalize(normtype, a)
     print normtype, anorm.find('abc')

-Ernie

On 8/22/2013 8:44 AM, Pieter Vandepitte wrote:
> Linlin and Andy,
>
> Isn't the solution to simply perform some normalization before 
> searching (see 
> https://en.wikipedia.org/wiki/Canonical_equivalence) and make sure the 
> data in your database is normalized the same way (e.g. NFC, NFD, NFKC 
> or NFKD)?
>
> kr,
>
> Pieter
>
> *Pieter Vandepitte*
> *Software Engineer*
> +32 16 29 89 27
> *www.dnsbelgium.be <http://www.dnsbelgium.be>*
>
>
> ------------------------------------------------------------------------
>
>     *From: *"Linlin Zhou" <[email-masked]>
>     *To: *"Andy Newton" <[email-masked]>, "Ernie Dainow"
>     <[email-masked]>, [email-masked]
>     *Sent: *Thursday, August 22, 2013 9:42:31 AM
>     *Subject: *Re: [weirds] Comments on draft-ietf-weirds-json-response-05
>
>
>

---

From: Andy Newton <[email-masked]>
Date: Thu, 22 Aug 2013 21:26:13 +0000

From: Ernie Dainow <[email-masked]<mailto:[email-masked]>>
Date: Thursday, August 22, 2013 5:01 PM
To: Pieter Vandepitte <[email-masked]<mailto:[email-masked]>>
Cc: Linlin Zhou <[email-masked]<mailto:[email-masked]>>, Andrew Newton <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [weirds] Comments on draft-ietf-weirds-json-response-05

It is important that the normalization form used by the server is specified as part of the spec. Otherwise, clients will get different results depending on the server that is queried for the search.

I like this approach. Which form do you recommend?

-andy

---

From: "John Levine" <[email-masked]>
Date: 22 Aug 2013 23:08:57 -0000

>It is important that the normalization form used by the server is specified as part of the
>spec. Otherwise, clients will get different results depending on the server that is queried
>for the search.
>

Why would a server use a normalization other than the one for DNS names?

R's,
John

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Fri, 23 Aug 2013 00:57:50 +0000

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of John Levine
> Sent: Thursday, August 22, 2013 7:09 PM
> To: [email-masked]
> specified as part of the

RFC 5890 requires Normalization Form C (NFC)from Unicode Standard Annex #15.

Scott

---

From: Pieter Vandepitte <[email-masked]>
Date: Fri, 23 Aug 2013 08:41:11 +0200 (CEST)

As how i understand it, the NF only depends on how you store your domain names in your database (i.e. normalize your input in the same form as your working data). It's not because RFC 5890 requires NFC for U-label _representation_ that you are not allowed to use any other NF for storage. 

So I would not specify any NF in the specs. It's up to the developer how he will implement search (or will RDAP impose rules on _how_ data is stored? I hope not...) and let's hope the developer takes into account normalization (as any developer should do for any project...) and other stuff like character escaping, character encoding (same thing: it's not because RDAP requires UTF-8, that a developer may not use ISO-8859-1 encoding in the database) etc.... A spec cannot cover all errors a developer could possibly make, or am i so terribly wrong about that? In best case we could add a 'Recommendations' section/document but I would not make any requirements on that... 

kr 

Pieter 

Pieter Vandepitte 
Software Engineer 
+32 16 29 89 27 
www.dnsbelgium.be 

----- Original Message -----

> From: "Scott Hollenbeck" <[email-masked]>
> To: "John Levine" <[email-masked]>, [email-masked]
> Sent: Friday, August 23, 2013 2:57:50 AM

---

From: Ernie Dainow <[email-masked]>
Date: Mon, 26 Aug 2013 17:15:55 -0400

I think the example of searching "abc*" against "abçd" demonstrates that 
normalizing the search string to the same form as the target name is not 
sufficient to return the same results. If one server uses NFC, the 
normalized target is  U+0061, U+0062, U+00E7, U+0064 and a search for 
"abc" returns no result. A server using NFD against the normalized 
target U+0061, U+0062, U+0063, U+0327, U+0064 will return abçd.

If you store domain names in other than NFC, then every time you 
retrieve a name from storage you will have to normalize it to NFC to 
comply with RFC 5890. This would be a lot of overhead in a database search.

It seems to me that use of NFC makes it impossible to search against 
partial characters. Even if the search string is a partial character 
sequence, it will never find a match, since the target in NFC does not 
have decomposed characters from normalization and does not have other 
partial characters, since it is a valid IDN. The text in 
draft-ietf-weirds-rdap-query-06 could be greatly simplified by removing 
the paragraphs about partial matches and specifying that searches must 
be done against NFC normalized names (and recommending that names be 
stored in NFC form).

Note that this does not produce search results that may be more 
intuitive to a user. For example, in a French dictionary, the following 
words are listed in this order:
     semaine
     sémaphore
     semblant
     séminaire
So when people look up words beginning with "se", they expect to see 
alphabetic order regardless of the diacritical marks on the letter e.

I believe most people doing an RDAP search for "se*" would expect to get 
the whole list. With NFC normalization, they will only get
     semaine
     semblant

-Ernie

On 8/23/2013 2:41 AM, Pieter Vandepitte wrote:
> As how i understand it, the NF only depends on how you store your 
> domain names in your database (i.e. normalize your input in the same 
> form as your working data). It's not because RFC 5890 requires NFC for 
> U-label _representation_ that you are not allowed to use any other NF 
> for storage.
>
> So I would not specify any NF in the specs. It's up to the developer 
> how he will implement search (or will RDAP impose rules on _how_ data 
> is stored? I hope not...) and let's hope the developer takes into 
> account normalization (as any developer should do for any project...) 
> and other stuff like character escaping, character encoding (same 
> thing: it's not because RDAP requires UTF-8, that a developer may not 
> use ISO-8859-1 encoding in the database) etc.... A spec cannot cover 
> all errors a developer could possibly make, or am i so terribly wrong 
> about that? In best case we could add a 'Recommendations' 
> section/document but I would not make any requirements on that...
>
> kr
>
> Pieter
>
> *Pieter Vandepitte*
> *Software Engineer*
> +32 16 29 89 27

---

From: "John Levine" <[email-masked]>
Date: 26 Aug 2013 21:34:25 -0000

>the whole list. With NFC normalization, they will only get

Unfortunately, this is a tarpit.  While the French consider e and é to
be approximately the same, the Swedes consider a, å, and ä to be as
different as a, b, and c.

I think the least bad we can do is to say that searches are against
NFC strings, and if you want to find accented characters, you have
to search for them.

R's,
John

---

From: Pieter Vandepitte <[email-masked]>
Date: Tue, 27 Aug 2013 11:16:39 +0200 (CEST)

I see the whole point now. I would rather agree with John's solution which is unambiguous to me... 

kr 

Pieter 

Pieter Vandepitte 
Software Engineer 
+32 16 29 89 27 
www.dnsbelgium.be 

----- Original Message -----

> From: "John Levine" <[email-masked]>
> To: [email-masked]
> Sent: Monday, August 26, 2013 11:34:25 PM

> Unfortunately, this is a tarpit. While the French consider e and � to
> be approximately the same, the Swedes consider a, �, and � to be as

---

From: "Linlin Zhou" <[email-masked]>
Date: Wed, 28 Aug 2013 17:13:29 +0800

Store the data in databases with the NFC normalization form and search for "abc*" or "abç*" also in NFC, I feel this is the most straightforward way for this issue.

Regards,
Linlin

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> John Levine
> Sent: Tuesday, August 27, 2013 5:34 AM
> To: [email-masked]
> >I believe most people doing an RDAP search for "se*" would expect to
> >get the whole list. With NFC normalization, they will only get
> >     semaine
> >     semblant
> 
> Unfortunately, this is a tarpit.  While the French consider e and  to be
> approximately the same, the Swedes consider a, , and  to be as different as a,
> b, and c.
> 
> I think the least bad we can do is to say that searches are against NFC strings,
> and if you want to find accented characters, you have to search for them.
> 
> R's,
> John