Subject: [weirds] Open Questions on the Search function for RDAP

From: Ning Kong <[email-masked]>
Date: Thu, 18 Jul 2013 08:51:32 +0800

Hi folks,

I try to list all related open questions on the search functions through this new thread. Any comments are appreciated!

#1  Why should we consider the search function?
	1)  Optional search function described by the ICANN AGB 
	2)  ICANN WSWG (Whois Survey Working Group) Survey Results (Requirement 5 for search)
	     Note1: the final report of this survey is not published, but the results can be accessed in the link of https://community.icann.org/display/WSDT/WSWG+Survey+Results
	     Note2: According to the results, over 81% respondents support the search function of domain name, over 53% respondents indicated positive interest in the search function of others (over 31% for NameServers, over 32% for Contact Name, over 34% for Contact Email).

#2  What should be the search function?
	1)  What query types should be searchable
		Domain?
			Handle?
			Name?
			Status?
			Others?
		Nameserver?
			Handle?
			Name?
			IP?
			Others?
		Entity?
			ID?
			Name?
			Email?
			Adr?
			Others?
		IP?
			Handle?
			StartAddress?
			Name?
			Others?
		AS?
			Handle?
			StartAutnum?
			Name?
			Others?

	2)  What kind of search functions should be considered?
		Wildcard? (single character match, multiple characters match, specific number of characters match)
		Boolean? (and, or, not)
		Page?
		Order by? (Date, Name)

	2)  What path segments should be used for search?
		Option1:   domains/?name=exampl*.com      by hollenbeck-weirds-rdap-search-02
	        Option2:   search/?domain=exampl*.com      by zhou-weirds-rdap-restful-search-00
		Option3:   domains/?name=exampl*.com&page=3&orderby=name

	3)  What response format should be used for search?
		Search level ?
		Multiple Results ?
		Page ? (Total Page Number, Current Page Number)
		Oder by ?

	4)  Others?
		I18N consideration
		Security consideration (Privacy Protection)
		Anything else?

#3  How should we document the search function?
	Option1:  one separated search document which covers all the issues of search
	Option2:  Merging all the search function into all related RDAP documents (Query, Response, others?)
	Option3:  Merging the basic search function into all related RDAP documents (Query, Response, others?), drafting one separated search document to cover the advanced search function

#4  Other Open Questions
	1)  How to clearly define the basic search function and the advanced one? 
	2)  Need we to consider the search function across different RDAP servers? (Search + Redirect ?)

Cheers,
Ning

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 18 Jul 2013 00:15:59 -0700

On Wed, Jul 17, 2013 at 5:51 PM, Ning Kong <[email-masked]> wrote:

> I try to list all related open questions on the search functions through
> this new thread. Any comments are appreciated!
>

I'd like to get some on-list discussion of this stuff, please.  We've
allocated only 10 minutes to this in Berlin, and the set of search topics
Ning posted will take far longer than that to cover.  Figuring out what to
discuss in person would be a really good idea; the rest can be handled on
the list.

Speaking as a participant, I like the idea of only doing the lowest common
denominator in the base documents, which is to say the prefix domain name
matching that someone (I think it was John Levine) found several providers
do.  I don't have a current preference in terms of how that would be
specified in the query.

-MSK

---

From: "John Levine" <[email-masked]>
Date: 18 Jul 2013 14:00:44 -0000

>do.  I don't have a current preference in terms of how that would be

Steve Sheng and I didn't find anyone who provided anything other than
a prefix match on specific fields.

Although I understand why consumers might want fancier options, it
seems unlikely that providers would ever implement them, and I don't
see any point in spending time on non-existent features.

The typical index structure that databases use (B-trees, if you know
what they are) make a prefix match cheap and easy.  Anything else
beyond a range match is a lot slower.  So I don't think it's a
coincidence that prefix match is all they offer.

R's,
John

---

From: Andy Newton <[email-masked]>
Date: Thu, 18 Jul 2013 16:25:05 +0000

>coincidence that prefix match is all they offer.

I agree with John.

As to the specific fields, John do you have a list?

-andy

---

From: "John R Levine" <[email-masked]>
Date: 18 Jul 2013 13:15:02 -0400

> As to the specific fields, John do you have a list?

Neustar: domain, nameserver, registrar, contact

Afilias: name, host (I think same as nameserver), contact, registrar

Verisign:  domain, nameserver, registrar

As best I recall, every searchable WHOIS was run by one of these three.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: Ernie Dainow <[email-masked]>
Date: Thu, 18 Jul 2013 14:13:15 -0400

> Afilias: name, host (I think same as nameserver), contact, registrar

host = nameserver
contact is searchable only by handle
registrar is searchable by handle or name
more details:
whois -h whois.afilias.info help

-Ernie

---

From: Marc Blanchet <[email-masked]>
Date: Thu, 18 Jul 2013 20:47:11 -0400

Le 2013-07-18 à 13:15, "John R Levine" <[email-masked]> a écrit :

> As best I recall, every searchable WHOIS

including all cctld? or just the gTLD?

Marc.

> was run by one of these three.
> 
> Regards,
> _______________________________________________
> weirds mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/weirds

---

From: "John R Levine" <[email-masked]>
Date: 18 Jul 2013 20:49:36 -0400

> including all cctld? or just the gTLD?

I didn't check every ccTLD, but the only ones I found that provided any 
sort of search were run by those three big registries.

Regards,

---

From: James Mitchell <[email-masked]>
Date: Fri, 19 Jul 2013 00:57:19 +0000

.au provides partial name match on domain name, host name, and registrar.

Regards,
James Mitchell
AusRegistry

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of John R Levine
> Sent: Friday, 19 July 2013 10:50 AM
> To: Marc Blanchet
> Cc: [email-masked]
> Subject: Re: [weirds] Open Questions on the Search function for RDAP
>

---

From: "John R Levine" <[email-masked]>
Date: 18 Jul 2013 21:08:17 -0400

> .au provides partial name match on domain name, host name, and registrar.

Tnx.  Is it just a prefix match or something more complicated?

>> Please consider the environment before reading this e-mail.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: James Mitchell <[email-masked]>
Date: Fri, 19 Jul 2013 01:10:58 +0000

Prefix-match. There is a limit on the number of results returned (10 of the top of my head), and the prefix must be at least five characters (wildcard from position 6+).

> From: John R Levine [mailto:[email-masked]]
> Sent: Friday, 19 July 2013 11:08 AM
> To: James Mitchell
> Cc: [email-masked]
> Subject: RE: [weirds] Open Questions on the Search function for RDAP
> 
> >> John Levine, [email-masked], Primary Perpetrator of "The Internet
> for
> >> Dummies",

---

From: Ernie Dainow <[email-masked]>
Date: Fri, 19 Jul 2013 11:32:20 -0400

On another thread there was support for including
<draft-hollenbeck-weirds-rdap-search> as the basic search. This would
provide search functionality equivalent to a number of existing whois
servers. Based on this, the following requirements would be met.

yes Domain?
Handle?
yes Name?
Status?
Others?
yes Nameserver?
Handle?
yes Name?
IP?
Others?
yes Entity?
ID?
yes Name?
Email?
Adr?
Others?
no IP?
Handle?
StartAddress?
Name?
Others?
no AS?
Handle?
StartAutnum?
Name?
Others?

2)  What kind of search functions should be considered?
yes Wildcard?
single character match,
yes multiple characters match,
specific number of characters match
Boolean? (and, or, not)
Page?
Order by? (Date, Name)

2)  What path segments should be used for search?
yes Option1:   domains/?name=exampl*.com       by
hollenbeck-weirds-rdap-search-02
        Option2:   search/?domain=exampl*.com       by
zhou-weirds-rdap-restful-search-00
Option3:   domains/?name=exampl*.com&page=3&orderby=name

Search level?
yes Multiple Results?
Oder by?

4)  Others?
yes I18N consideration
yes Security consideration (Privacy Protection)
Anything else?

#3  How should we document the search function?
Option1: one separated search document which covers all the issues of search
Option2: Merging all the search function into all related RDAP documents
yes Option3: Merging the basic search function into all related RDAP
documents
future Drafting one separate search document to cover advanced search

-Ernie

On Wed, Jul 17, 2013 at 8:51 PM, Ning Kong <[email-masked]> wrote:

>              Note1: the final report of this survey is not published, but
> the results can be accessed in the link of
> https://community.icann.org/display/WSDT/WSWG+Survey+Results
>              Note2: According to the results, over 81% respondents support
> the search function of domain name, over 53% respondents indicated positive
> interest in the search function of others (over 31% for NameServers, over
> 32% for Contact Name, over 34% for Contact Email).
>
>                 Wildcard? (single character match, multiple characters
> match, specific number of characters match)
>                 Boolean? (and, or, not)
>                 Page?
>                 Order by? (Date, Name)
>
>         Option3:  Merging the basic search function into all related RDAP
> documents (Query, Response, others?), drafting one separated search
> document to cover the advanced search function
>
> #4  Other Open Questions

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 22 Jul 2013 12:17:47 -0700

Any other opinions on this?  Ning should be able to narrow down his use of
ten minutes based on the feedback if there's obvious consensus.

Ning, can you chip this down into a plan that will fit in the time we have
for face-to-face meeting?

-MSK

On Fri, Jul 19, 2013 at 8:32 AM, Ernie Dainow <[email-masked]> wrote:

>>              Note2: According to the results, over 81% respondents
>> support the search function of domain name, over 53% respondents indicated
>> positive interest in the search function of others (over 31% for
>> NameServers, over 32% for Contact Name, over 34% for Contact Email).
>>

---

From: Joseph Yee <[email-masked]>
Date: Mon, 22 Jul 2013 18:17:28 -0400

please see inline

> #1  Why should we consider the search function?

[snip]

>         1)  What query types should be searchable
[snip]

What should be searchable is more driven by users & providers, IMHO.
If possible, just define a proper general naming convention should do,
then it's just matter of whether server support the term or not.

>                 Option3:   domains/?name=exampl*.com&page=3&orderby=name

Answer both questions above here.
Seems like there's a preference of query string over path, and I think
it's fine, it doesn't matter much.

I'm concern of trying to present complex search in query string
format. The mix of what symbol characters may stored, rdap response,
query string's handling on percent encode and unreserved could give
developers and users enough headaches.  That being said, boolean, page
and order are MAYBE to consider.

>                 Security consideration (Privacy Protection)

yes and yes.

>         Option3:  Merging the basic search function into all related RDAP documents (Query, Response, others?), drafting one separated search document to cover the advanced search function

Leaning to option 3, but only if we found good way to separate/extend
basic to advance.

>         2)  Need we to consider the search function across different RDAP servers? (Search + Redirect ?)

If you mean wildcard on the TLD part, I personally would prefer
smarter client.  Don't expect all can be done under one long query
string.

Best
Joseph

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 23 Jul 2013 09:29:19 -0700

> for face-to-face meeting?
There have only been a handful of replies, but it does seem like we're
trending toward having the truly basic search option added to rdap-query
and json-response.

Is there anyone that feels differently, or should the authors of those
documents proceed in that direction?

And if we do that, what's left for the search timeslot in Berlin?

-MSK

---

From: "John Levine" <[email-masked]>
Date: 23 Jul 2013 17:37:02 -0000

>And if we do that, what's left for the search timeslot in Berlin?

Check that we agree on what the responses should be?

* If there are more answers than a server is willing to return in one
group, is it worth specifying a way to ask for more, i.e., are there
any existing WHOIS servers that will do that?

* What are the possible answers: all the matches, some of the matches, no
matches, invalid patern (wildcard in the middle, for example),
anything else?

R's,
John

---

From: MICHAEL YOUNG <[email-masked]>
Date: Tue, 23 Jul 2013 14:18:32 -0400

I don't really see the point in supporting search if you aren't willing to provision a full answer.

Of course no one wants to answer a 50,000 match query, on a public service, in a single response.  I think the answer lies in being able to set the limit of the response via server policy, BUT you should have a marker that allows you to pick up where the last response left off to allow you to ask for the next group.  So I ask for 50,000 and get back 100 matches with an identifier the server will recognize if I submit it with my next query, AND that gets me the next 100 matches, and so forth and so on.  The server can apply rate-limiting mechanisms against that methodology, such as how often I can query, response size, etc.

-Michael

> matches, invalid patern (wildcard in the middle, for example),

MICHAEL YOUNG
[email-masked]

---

From: "Linlin Zhou" <[email-masked]>
Date: Wed, 24 Jul 2013 11:31:42 +0800

> 	Option1:  one separated search document which covers all the issues
of
> search
I have a question for a long time in my mind. I have no idea how to separate
the basic and advanced search functions. If we pick the option 3, the only
difference I can see is that maybe the basic search function specifies two
kinds of query types, the advanced function defines such as 8 query types
and may also gives the Boolean query path specifications. IMO, we could
define the same rules for query path segment, response format, IDN
specifications and so on, both the basic and advanced search just follow
them. Do we really need to separate them into several drafts? What's others
thinking?

Regards,
Linlin

---

From: Peter Koch <[email-masked]>
Date: Wed, 24 Jul 2013 09:50:12 +0200

> documents proceed in that direction?

I'd like the chairs to understand - and make sure by way of the writeup
that the IESG understands - there's a significant risk of policy washing
with the search capability even if it is declared an optional feature
(where there's an ongoing ambiguity regarding implementation vs. operation
already).

-Peter

---

From: Ning Kong <[email-masked]>
Date: Wed, 24 Jul 2013 16:56:38 +0800

Sure. I'll try to do this work.

It seems that the focus of our discussion has been moved from Question #3 to Question #1. I hope we can get a rough consensus on #1 before the WG meeting in Berlin. So we can mainly focus on discussing about Question #2 and #3.

As for #1, IMO, there are mainly two different ways to answer this question.
     Way 1, We should only consider the search functions which are currently implemented in existing Whois system. That is to say, if current Whois system doesn't (or rarely) support search functions, RDAP shouldn't include search functions.
      Way 2, the potential requirements on the search functions for the RDAP should be the key factor for us to consider. ICANN WSWG survey results may be a useful reference for the WG. (Based on the result, the requirements for search are considerable.)

IMO, I lean on way 2 because I believe the requirements should be the motivation for us to develop RDAP protocols. I'd like to know you guys' clear choices on these two ways.

Cheers,
Ning

> Any other opinions on this?  Ning should be able to narrow down his use of ten minutes based on the feedback if there's obvious consensus.
> 
> Ning, can you chip this down into a plan that will fit in the time we have for face-to-face meeting?
> 
> -MSK
> 
> 
> On another thread there was support for including <draft-hollenbeck-weirds-rdap-search> as the basic search. This would provide search functionality equivalent to a number of existing whois servers. Based on this, the following requirements would be met. 
> 
> yes		Option1:   domains/?name=exampl*.com       by hollenbeck-weirds-rdap-search-02
> yes	Option3: Merging the basic search function into all related RDAP documents

---

From: "John Levine" <[email-masked]>
Date: 24 Jul 2013 15:58:58 -0000

>     Way 1, We should only consider the search functions which are
>currently implemented in existing Whois system. That is to say, if
>current Whois system doesn't (or rarely) support search functions, RDAP
>shouldn't include search functions.
>      Way 2, the potential requirements on the search functions for the
>RDAP should be the key factor for us to consider. ICANN WSWG survey
>results may be a useful reference for the WG. (Based on the result, the
>requirements for search are considerable.)

I can report from sad experience that nothing good will come from
attempting to design complex untested features and hoping that someone
will implement them.

The search requirements are, as far as I can tell, just a wish list of
what WHOIS users would like.  But there are strong reasons, both
technical and policy reasons, that registries and registrars don't
offer complex WHOIS searches now.  Since they don't offer complex
search now, I don't understand why they would offer it with RDAP.
There is nothing in the new gTLD program that requires any search at
all.

Even if we try and limit ourself to the technical issues, complex
searches are a difficult database problem.  Searches that look similar
can require very different amounts of work, e.g., a search for "ab*"
is usually two or three disk operations, while a search for "*ab"
often requires scanning the entire table.  To design something that
had a chance of working, we would need both database experts and
registry/registrar operations experts.

So please, let's specify what people are likely to implement.  If it
later turns out that registries actually plan to do complex search,
then we can work with them to design something they plan to
implement.

R's,
John

---

From: Ning Kong <[email-masked]>
Date: Thu, 25 Jul 2013 16:37:29 +0800

Hi John,

Thanks for your comments. Please see my feedback inline.

> what WHOIS users would like.  
No. Not only WHOIS users but also WHOIS providers joined the ICANN WSWG survey. Based on the result documents, the participants cover Commercial business user, Non-commercial organization user, Individual or end user, Domain name Registrar and/or Registry, Internet access provider or network operator.

> But there are strong reasons, both
"They don't offer it now" doesn't definitely equal to "They won't offer it later".

> registry/registrar operations experts.
I understand. But the above problem can't be the reason that we should not do this work.

> then we can work with them to design something they plan to
Agree. IMO, making a survey is a method to specify what people are likely to implement. I wonder if you think the ICANN WSWG survey is enough or not.

As I mentioned in my last reply, I hope we can firstly make clear on the question #1. Here, I slightly modify it.
Question #1   Should we consider the search function (whether it is simple or complex)?
Answer 1: Yes. Both simple and complex function.
Answer 2: Yes. only simple function. 
Answer 3: No.

So, what is your choice?

Cheers,
Ning

---

From: "John R Levine" <[email-masked]>
Date: 25 Jul 2013 18:09:59 +0200

> Answer 2: Yes. only simple function.

We've seen providers that say they'll do simple search, so that's what we 
should put in the spec.

If we find registries that plan to offer more complex searches, we can add 
that later, but I'd very much prefer to put something simple in the spec 
that we are confident can be implemented.

Regards,

---

From: Andy Newton <[email-masked]>
Date: Thu, 25 Jul 2013 16:30:06 +0000

>that we are confident can be implemented.

+1

-andy

---

From: Frederico A C Neves <[email-masked]>
Date: Thu, 25 Jul 2013 13:55:03 -0300

> >that we are confident can be implemented.

I also support this direction.

Fred

---

From: Ning Kong <[email-masked]>
Date: Fri, 26 Jul 2013 08:49:35 +0800

> I also support this direction.

I can accept this direction, too. Any other different choices?
If this direction is our current consensus, let's move on to discuss the following questions (#2, #3, #4 or others) based on this direction. 

Ning