Subject: [weirds] Updated Search Processing Text Proposal

From: "Hollenbeck, Scott" <[email-masked]>
Date: Tue, 17 Dec 2013 13:24:59 +0000

I'm going to try to capture and summarize the results of the recent discussion on search processing text. This is what's currently in rdap-query-08:

Servers SHOULD thus accept any search pattern that is otherwise properly encoded.  If the pattern consists entirely of ASCII characters, servers SHOULD use case-insensitive prefix matching against ASCII labels to determine partial matches.  If the text in the pattern is or can be converted to a U-label, servers SHOULD use prefix matching against U-labels to determine partial matches.

Proposed new text:

An RDAP client submitting a query string containing non-US-ASCII characters SHOULD convert such strings into Unicode in UTF-8 encoding. It MAY perform any local case mapping deemed necessary. It SHOULD normalize the strings using Normalization Form C (NFC, [Unicode-UAX15]); note that clients might not be able to do this reliably.

An RDAP server SHOULD treat each query string as Unicode in UTF-8 encoding. If a string is not valid UTF-8, the server MAY immediately stop processing the query and return an HTTP 400 error response code.  Servers SHOULD map fullwidth and halfwidth characters to their decomposition equivalents. Servers SHOULD convert strings to the same coded character set of the target data that is to be looked up or searched and each string SHOULD be normalized using the same normalization that was used on the target data. Normalization Form C (NFC, [Unicode-UAX15]) is RECOMMENDED to be consistent with IDNA [RFC5891] and to maximize predictability and interoperability. Case insensitive pattern matching SHOULD be used to compare US-ASCII characters in query patterns.

DNS names are treated according to the DNS matching rules as described in Section 3.1 of RFC 1035 [RFC1035] for NR-LDH labels and the matching rules described in Section 5.4 of RFC 5891 [RFC5891] for U-labels.  Matching of DNS names proceeds one label at a time, because it is possible for a combination of U-labels and NR-LDH labels to be found in a single domain or host name. The determination of whether a label is a U-label or an NR-LDH label is based on whether the label contains any characters outside of the US-ASCII letters, digits, or hyphen (the so-called LDH rule).

I appreciate the thoughts and text provided by everyone - thanks!

Scott

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 17 Dec 2013 09:42:05 -0500

On Tue, Dec 17, 2013 at 01:24:59PM +0000, Hollenbeck, Scott wrote:

> Case insensitive pattern matching SHOULD be used to compare US-ASCII
> characters in query patterns.

I still very strongly suggest that the behaviour needs to be
string-based, not character-based.  If you're going to do case
insensitive work, you need to use Unicode case folding, not do case
folding for some characters and not others.  The above has the absurd
consequence that école and École don't match, but école and éCole do.

We said we didn't want to get into the complications of case folding
or the disaster that will result from trying to do
decorated-vs-undecorated character matching, which is why not to use
all of case folding.  So I can see an argument for case insensitivity
for ASCII strings, but one cannot do this character by character.  

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Tue, 17 Dec 2013 14:48:59 +0000

> -----Original Message-----
> From: weirds [mailto:[email-masked]] On Behalf Of Andrew
> Sullivan
> Sent: Tuesday, December 17, 2013 9:42 AM
> To: [email-masked]
> Subject: Re: [weirds] Updated Search Processing Text Proposal
> 

So how would you suggest re-wording that sentence?

Scott

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 17 Dec 2013 10:00:35 -0500

> So how would you suggest re-wording that sentence?

    If and only if the entire query pattern uses only Unicode
    characters from the US-ASCII range, then case insensitive pattern
    matching SHOULD be used.  Otherwise, no case folding is performed.
    
I confess that I expect this will be a controversial position.
Certainly, from experience, I expect people from the .gr registry to
react very negatively to such a proposal.

If we're going to deal with case, the only alternative to the above
(which people said they didn't like on grounds of complication) is to
use Unicode case folding all the time.  If you do that, you have to
decide whether to handle SpecialCasing or not.  If you want text for
this approach, I can come up with something but it'll be more
complicated than the above.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Tue, 17 Dec 2013 18:22:33 +0000

> Sent: Tuesday, December 17, 2013 10:01 AM
> To: [email-masked]

Is this not acceptable to anyone?

Scott

---

From: Andy Newton <[email-masked]>
Date: Tue, 17 Dec 2013 18:35:56 +0000

>Is this not acceptable to anyone?

I agree with this in principle, but I guess it depends on the details.

Is special casing with Unicode commonly found in libraries? If not, we
probably ought to punt on this.

-andy

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 17 Dec 2013 14:49:20 -0500

> probably ought to punt on this.

The problem is that special casing is Hard.  It's the thing that, just
for instance, copes with the fact that the character "i" is the lower
case of "I" in every language except Turkish.  In Turkish, it is the
lower case of "İ" (U+0130), and the lower case of "I" is "ı" (U+0131).
What this means is that special casing is locale-sensitive.  But we
don't have a mechanism AFAICT for communicating locale for this sort
of thing.

This makes me feel like special-casing search should not be included,
and there should be text that says "we're not doing any of that".
Therefore, here's another suggestion:

    An RDAP server SHOULD treat each query string as Unicode in UTF-8
    encoding.  If a string is not valid UTF-8, the server MAY
    immediately stop processing the query and return an HTTP 400 error
    response code.  

    When processing queries, there is a difference in handling DNS
    names, including those including putative U-labels, and everything
    else.  DNS names are treated according to the DNS matching rules
    as described in Section 3.1 of RFC 1035 [RFC1035] for NR-LDH
    labels and the matching rules described in Section 5.4 of RFC 5891
    [RFC5891] for U-labels.  Matching of DNS names proceeds one label
    at a time, because it is possible for a combination of U-labels
    and NR-LDH labels to be found in a single domain or host name. The
    determination of whether a label is a U-label or an NR-LDH label
    is based on whether the label contains any characters outside of
    the US-ASCII letters, digits, or hyphen (the so-called LDH rule).

    For everything else, servers SHOULD map fullwidth and halfwidth
    characters to their decomposition equivalents.  Servers SHOULD
    convert strings to the same coded character set of the target data
    that is to be looked up or searched and each string SHOULD be
    normalized using the same normalization that was used on the
    target data.  In general, storage of strings as Unicode is
    RECOMMENDED.  For the purposes of comparison, Normalization Form
    KC (NFKC, [Unicode-UAX15]) with case folding is RECOMMENDED to
    maximize predictability and the number of matches.  Note the use
    of case-folded NFKC as opposed to NFC in this case.

This is still only for search, right, and not for exact matching?
(That's my impression given the text we're adjusting.)  That is, my
intention is that the above apply to search, because I think case
folded format (==NFKC + case fold) is more likely to yield matches;
but I think NFC and unfolded is a better exact-string matching answer,
which is why it's still called for here in the U-label matching.

Sorry for the thrash on this topic.  Internationalization sucks.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 17 Dec 2013 18:08:54 -0800

Participatorially speaking,

On Tue, Dec 17, 2013 at 5:24 AM, Hollenbeck, Scott <[email-masked]
> wrote:

> Proposed new text:
>
> An RDAP client submitting a query string containing non-US-ASCII
> characters SHOULD convert such strings into Unicode in UTF-8 encoding. It
> MAY perform any local case mapping deemed necessary. It SHOULD normalize
> the strings using Normalization Form C (NFC, [Unicode-UAX15]); note that
> clients might not be able to do this reliably.
>

Having been beaten about the head and neck with a thick stick on this
topic, I've become sensitive to it.

The SHOULDs here automatically have me asking questions like "Under what
circumstances would you not do this?  Why isn't it a MUST?  What's the
nature of the discretion left to the implementer?"  I've also been
encouraged recently to, upon using a SHOULD, also add a sentence giving an
example or guidance of a situation in which an implementation would
reasonably opt to do something else, or at least what breaks if the
implementer chooses not to comply.

> encoding. If a string is not valid UTF-8, the server MAY immediately stop
> processing the query and return an HTTP 400 error response code.  Servers
> SHOULD map fullwidth and halfwidth characters to their decomposition
> equivalents. Servers SHOULD convert strings to the same coded character set
> of the target data that is to be looked up or searched and each string
> SHOULD be normalized using the same normalization that was used on the
> target data. Normalization Form C (NFC, [Unicode-UAX15]) is RECOMMENDED to
> be consistent with IDNA [RFC5891] and to maximize predictability and
> interoperability. Case insensitive pattern matching SHOULD be used to
> compare US-ASCII characters in query patterns.
>

Another thing that's been suggested to me, which might be simpler, is to
drop the RFC2119 language altogether.  Instead of saying "X SHOULD/MUST do
Y", simply say "X does Y"; the implication here is that if X doesn't do Y,
it's simply not implementing RDAP.

Also, this is entirely SHOULD and RECOMMENDED stuff.  That means an
implementer is entirely within its rights to do something completely
different than what's specified here, and still claim compliance.  How
interoperable would such a thing really be?

>
> DNS names are treated according to the DNS matching rules as described in
> Section 3.1 of RFC 1035 [RFC1035] for NR-LDH labels and the matching rules
> described in Section 5.4 of RFC 5891 [RFC5891] for U-labels.  Matching of
> DNS names proceeds one label at a time, because it is possible for a
> combination of U-labels and NR-LDH labels to be found in a single domain or
> host name. The determination of whether a label is a U-label or an NR-LDH
> label is based on whether the label contains any characters outside of the
> US-ASCII letters, digits, or hyphen (the so-called LDH rule).
>

Here's an example where "are" is there where in the previous paragraph we
use "SHOULD/MUST be".  It might be just fine to write the earlier paragraph
that way.

I don't feel particularly strongly about this, but I think if we were to
move forward with this as-is, we should expect our ADs to ask these
questions and be prepared to answer them.

-MSK, still nursing past RFC2119 welts

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Wed, 18 Dec 2013 13:09:01 +0000

> Sent: Tuesday, December 17, 2013 2:49 PM
> To: [email-masked]

Let me try to take a crack at addressing Murray's SHOULD/MUST comment [1] starting with Andrew's text above:

An RDAP server treats each query string as Unicode in UTF-8 encoding.  If a string is not valid UTF-8, the server can immediately stop processing the query and return an HTTP 400 error response code.  

When processing queries, there is a difference in handling DNS names, including those including putative U-labels, and everything else.  DNS names are treated according to the DNS matching rules as described in Section 3.1 of RFC 1035 [RFC1035] for NR-LDH labels and the matching rules described in Section 5.4 of RFC 5891 [RFC5891] for U-labels.  Matching of DNS names proceeds one label at a time, because it is possible for a combination of U-labels and NR-LDH labels to be found in a single domain or host name. The determination of whether a label is a U-label or an NR-LDH label is based on whether the label contains any characters outside of the US-ASCII letters, digits, or hyphen (the so-called LDH rule).

For everything else, servers map fullwidth and halfwidth characters to their decomposition equivalents.  Servers convert strings to the same coded character set of the target data that is to be looked up or searched and each string is normalized using the same normalization that was used on the target data.  In general, storage of strings as Unicode is RECOMMENDED.  For the purposes of comparison, Normalization Form KC (NFKC, [Unicode-UAX15]) with case folding is used to maximize predictability and the number of matches.  Note the use of case-folded NFKC as opposed to NFC in this case.

Scott

[1] http://www.ietf.org/mail-archive/web/weirds/current/msg03509.html

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 18 Dec 2013 08:19:56 -0500

This is ok with me.  It strikes me, however, that RFC 2119 is having
pretty perverse effects if we're now rewriting text to get around
using those precise definitions, in order not to get into picky
procedural arguments. :(

-- 
Andrew Sullivan
[email-masked]

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Wed, 18 Dec 2013 13:26:35 +0000

> Sent: Wednesday, December 18, 2013 8:20 AM
> To: [email-masked]
> pretty perverse effects if we're now rewriting text to get around using
> those precise definitions, in order not to get into picky procedural
> arguments. :(

Indeed! Maybe we can just lower-case them... ;)

Scott

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 18 Dec 2013 12:27:44 -0800

> using those precise definitions, in order not to get into picky

I agree, it's become unpleasant space.  I think we might be due for a
revision that provides better guidance (and this comes up from time to
time), but that's not work for this WG.

-MSK