Subject: Re: [weirds] Open Questions on the Search function for RDAP

From: "John R Levine" <[email-masked]>
Date: 23 Jul 2013 16:43:21 -0400

> I don't really see the point in supporting search if you aren't willing to 
> provision a full answer.
> 
> Of course no one wants to answer a 50,000 match query, on a public service, 
> in a single response.  I think the answer lies in being able to set the limit 
> of the response via server policy, BUT you should have a marker that allows 
> you to pick up where the last response left off to allow you to ask for the 
> next group.  So I ask for 50,000 and get back 100 matches with an identifier 
> the server will recognize if I submit it with my next query, AND that gets me 
> the next 100 matches, and so forth and so on.  The server can apply 
> rate-limiting mechanisms against that methodology, such as how often I can 
> query, response size, etc.

No TLD offers anything like that in their existing WHOIS service, and most go 
to considerable effort to prevent bulk queries.  What leads you to believe they 
would provide extra features to make data mining their RDAP servers easier?

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 23 Jul 2013 17:10:56 -0400

On Tue, Jul 23, 2013 at 04:43:21PM -0400, John R Levine wrote:

> No TLD offers anything like that in their existing WHOIS service,
> and most go to considerable effort to prevent bulk queries.  What
> leads you to believe they would provide extra features to make data
> mining their RDAP servers easier?

That many do all have a bulk whois service, and since RDAP can do
authentication they could then do the bulk whois using the same
protocol?

It's not only WHOIS that we're replacing.  If it were, why do the
work?

A

-- 
Andrew Sullivan
[email-masked]

---

From: MICHAEL YOUNG <[email-masked]>
Date: Tue, 23 Jul 2013 17:48:15 -0400

Hi John,

I see Andrew made this point as well, this is about putting the capability and therefore the choice in place, some TLDs absolutely offer this type of service now and many of the NTLDs have stated they will in their applications. 

I agree some won't want to offer this type of functionality, that's why I said this has to be determined by server policy, but that a standardized way of offering it is important for those who do.

Rate-limiting, authenticated users only being allowed to use the added search functionality, or just not offering the service (server policy) are all ways of controlling potential data mining,......  

-Michael

On 2013-07-23, at 4:43 PM, "John R Levine" <[email-masked]> wrote:

>> I don't really see the point in supporting search if you aren't willing to provision a full answer.
>> Of course no one wants to answer a 50,000 match query, on a public service, in a single response.  I think the answer lies in being able to set the limit of the response via server policy, BUT you should have a marker that allows you to pick up where the last response left off to allow you to ask for the next group.  So I ask for 50,000 and get back 100 matches with an identifier the server will recognize if I submit it with my next query, AND that gets me the next 100 matches, and so forth and so on.  The server can apply rate-limiting mechanisms against that methodology, such as how often I can query, response size, etc.
> 
> No TLD offers anything like that in their existing WHOIS service, and most go to considerable effort to prevent bulk queries.  What leads you to believe they would provide extra features to make data mining their RDAP servers easier?
> 
> Regards,
> _______________________________________________
> weirds mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/weirds

MICHAEL YOUNG
[email-masked]

---

From: "John R Levine" <[email-masked]>
Date: 23 Jul 2013 18:04:19 -0400

> I agree some won't want to offer this type of functionality, that's why 
> I said this has to be determined by server policy, but that a 
> standardized way of offering it is important for those who do.

Having read a lot of the applications and all of the existing gTLD 
agreements, I see no indication that any registry or any applicant will 
support data mining.  On the contrary, they tend to say they will 
specifically forbid it.  For example, I looked at one of the many similar 
applications from Donuts, number 1-1255-37010, which says:

  2.6.3. Security and Stability Considerations

  Abuse of the Whois system through data mining is a concern as it can
  impact system performance and reduce the quality of service to legitimate
  users. The Whois system mitigates this type of abuse by detecting and
  limiting bulk query access from single sources. It does this in two ways:
  1) by rate limiting queries by non-authorized parties; and 2) by ensuring
  all queries result in responses that do not include data sets
  representing significant portions of the registration database. In
  addition, the Whois web interface adds a simple challenge-response CAPCHA
  that requires a user to type in the characters displayed in image format.
  Both systems have blacklist functionality to provide a complete block to
  individual IPs or IP ranges.

Can you point us at a specific registry or applicant that says they will 
support data mining of the kind you're describing?

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: "John Levine" <[email-masked]>
Date: 23 Jul 2013 22:16:19 -0000

>authentication they could then do the bulk whois using the same

It is my impression that to the extent it exists, it's like ZFA, with
a password that the client uses to batch download a file.  The new TLD
application form mentions bulk WHOIS access, and the Verisign
applications say they provide three files for download, of domains,
nameservers, and registrars.

Hence I'd want to hear from a registry that says that they will
implement bulk RDAP before trying to invent a query protocol for it.
What's the point otherwise?

FWIW, I've talked to large registrars who've told me that although in
theory they are supposed to provide bulk access, nobody's ever asked
for it.

R's,
John

---

From: Chris Wright <[email-masked]>
Date: Wed, 24 Jul 2013 08:20:32 +1000

Ill actually affirm the opposite for you John, we at AusRegistry (.au, a number of other ccTLDs, and 167 new gTLDs) will not be offering any sort of search/bulk functionality, thus do not require these features.

Thanks

Chris

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of John Levine
Sent: Wednesday, 24 July 2013 8:16 AM
To: [email-masked]
Subject: Re: [weirds] Open Questions on the Search function for RDAP

>authentication they could then do the bulk whois using the same 

It is my impression that to the extent it exists, it's like ZFA, with a password that the client uses to batch download a file.  The new TLD application form mentions bulk WHOIS access, and the Verisign applications say they provide three files for download, of domains, nameservers, and registrars.

Hence I'd want to hear from a registry that says that they will implement bulk RDAP before trying to invent a query protocol for it.
What's the point otherwise?

FWIW, I've talked to large registrars who've told me that although in theory they are supposed to provide bulk access, nobody's ever asked for it.

R's,
John

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 23 Jul 2013 19:58:52 -0400

> What's the point otherwise?

I don't work there any more, but when I did work at a registry
provider we would have dearly loved to have a real protocol for this,
because the alternative was a major pain when we got these requests.
Still, that doesn't respond to your need.

A

-- 
Andrew Sullivan
[email-masked]

---

From: MICHAEL YOUNG <[email-masked]>
Date: Tue, 23 Jul 2013 20:39:28 -0400

Ok John in all fairness you are cherry-picking, and the AGB Whois question required an answer element on how to control unsanctioned data mining, hence why Donuts has this in there.

That said, there's no reason they may not offer select authenticated users the ability to do wider searches.  Even this answer simply asserts that they are not going to offer a single query response that brings back half the registry database in one go, it doesn't rule out a series of incremental queries with applied rate limiting.

No applicant is offering open public whois data mining as a service, that was against the AGB requirements, but many are offering search functionality as it got them an extra point in the application scores.  

So, given that, I am still making the point that supporting that search functionality in a standardized manner is important.

How you control data mining is another very important question, but you don't do it by a broken search implementation.

All this might end up being a moot point given ICANN's latest direction in centralized next gen Whois.

-M

On 2013-07-23, at 6:04 PM, "John R Levine" <[email-masked]> wrote:

> Having read a lot of the applications and all of the existing gTLD agreements, I see no indication that any registry or any applicant will support data mining.  On the contrary, they tend to say they will specifically forbid it.  For example, I looked at one of the many similar applications from Donuts, number 1-1255-37010, which says:
> 
> Can you point us at a specific registry or applicant that says they will support data mining of the kind you're describing?
> 
> Regards,

MICHAEL YOUNG
[email-masked]

---

From: "John R Levine" <[email-masked]>
Date: 23 Jul 2013 21:27:50 -0400

> Ok John in all fairness you are cherry-picking, and the AGB Whois 
> question required an answer element on how to control unsanctioned data 
> mining, hence why Donuts has this in there.

Page A-19 of the AGB says that WHOIS search is optional.  I've read an 
awful lot of the applications and this answer is very typical.

Unless you can identify a registry that wants to use RDAP to provide bulk 
access, I think this thread is done.

Regards,

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 23 Jul 2013 21:28:31 -0400

On Tue, Jul 23, 2013 at 08:39:28PM -0400, MICHAEL YOUNG wrote:
> 
> No applicant is offering open public whois data mining as a service,
> that was against the AGB requirements, but many are offering search
> functionality as it got them an extra point in the application
> scores.

To be fair to John, he's not asking whether this could be useful.
He's asking whether anyone actually will implement it.  If there were
someone saying, "Yes, we will," then that makes a difference.  In the
absence of such a claim, however, the previous arguments are all in
force: don't make the protocol complicated to solve cases nobody wants
to do.

Despite the fact that, when I worked there, Afilias could have used a
protocol that offered this functionality, we didn't implement IRIS,
and it _did_ offer this ability.  So that's a counter-argument to
adding the feature to the protocol.

> All this might end up being a moot point given ICANN's latest direction in centralized next gen Whois.

If someone wants to make an argument for real that ICANN is going to
pursue that direction, then I suggest we stop working on names
altogether and finish the numbers case the way some people wanted to
do originally.  (For my part, I think that the "latest direction" is
in fact just a bunch of half-informed -- or less -- people whose
answer to all problems is to centralise everything under someone's
control.  I look forward to the RDAP-vs-OSICANNWhois bakeoff.)

A
-- 
Andrew Sullivan
[email-masked]

---

From: Andy Newton <[email-masked]>
Date: Wed, 24 Jul 2013 10:15:38 +0000

>What's the point otherwise?

John,

We (ARIN) offer search and bulk access (two different things), though we
don't offer search access with paging. The search features are somewhat
similar to the current basic features but with combining logic for
multiple fields in some situations. The bulk access is restricted to those
signing an AUP only, and is offered up as ZIP files via HTTP in a
completely different manner than our restful whois service.

All that said, I agree with your design approach. Let's get the basic
stuff out the door and worry about the corner cases and more advanced
features later.

-andy

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Wed, 24 Jul 2013 11:18:22 +0000

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Andrew Sullivan
> Sent: Tuesday, July 23, 2013 9:29 PM
> To: [email-masked]

[snip]

> control.  I look forward to the RDAP-vs-OSICANNWhois bakeoff.)

"half-informed -- or less"? Sigh...

Anything the EWG (of which I am a member) recommends is nothing more than that - a _recommendation_ to be considered by the GNSO. It's not a "direction" at all. I fully expect that whatever eventually comes out of the policy development process will be different, and a single, centralized repository might not be part of the solution because it does present significant challenges. Either way the work we're doing _here_ should be useful (if not a real driver) and will very likely shape the future if/when we deploy RDAP and gain real operational experience with it.

Running code should easily trump theory. I don't see there ever being an "OSICANNWhois".

While we're on the topic of centralization, I do believe there's one place where it could be very helpful for us: user credentials and authentication. I'd much prefer to work with a system where a small number of trusted entities become the "go to" sources for credentials so that people who work with multiple operators don't have to secure credentials from each one independently. Federated authentication makes a lot of sense when the number of registry operators is large.

Scott

---

From: Eric Brunner-Williams <[email-masked]>
Date: Wed, 24 Jul 2013 10:08:06 -0700

> control.  I look forward to the RDAP-vs-OSICANNWhois bakeoff.)

Plus One. The new (now 8 months into what may be just a three year
gig) CEO/President made a pre-informed commitment just about 8 months
ago (at the Toronto meeting) and the policy volunteers in this area
include ... wicked few of us.

On to numbers.

Eric