Subject: [weirds] Search Engines Indexing RDAP Server Content

From: "Hollenbeck, Scott" <[email-masked]>
Date: Fri, 29 Jan 2016 13:34:35 +0000

(cross-posting to multiple lists - sorry if that's inconvenient)

So I saw a tweet from Gavin Brown (@GavinBrown) that describes how one particular search engine has indexed the RDAP server of a gTLD registry operator:

https://twitter.com/GavinBrown/status/692718904058191872

This is all the more reason to work on a client authentication specification that includes support for varying responses based on client identity and authorization. I've been working on such a specification and welcome feedback on the approach:

https://datatracker.ietf.org/doc/draft-hollenbeck-weirds-rdap-openid/

It also begs the question of the need for a BCP describing operational practices for server operators. There are ways for web servers to influence or restrict crawler behavior, but what's appropriate in this context?

Scott

---

From: Andrew Newton <[email-masked]>
Date: Fri, 29 Jan 2016 10:14:37 -0500

> https://twitter.com/GavinBrown/status/692718904058191872

Interesting... especially that most of the results are entities but it
doesn't seem to be that many.

> https://datatracker.ietf.org/doc/draft-hollenbeck-weirds-rdap-openid/

+1

-andy

---

From: "John R Levine" <[email-masked]>
Date: 29 Jan 2016 12:30:15 -0500

>> It also begs the question of the need for a BCP describing operational 
>> practices for server operators. There are ways for web servers to influence 
>> or restrict crawler behavior, but what's appropriate in this context?
>
> It would be good to keep in mind that respecting mechanisms such as 
> robots.txt is entirely voluntary on the crawler’s side.

Right -- the only reasonable assumption is that if casual users can find 
it with http, evil search engines (of which there are plenty) will too.

The search engines that most people use such as Google and Bing all obey 
robots.txt so it will keep info out of casual searches.  But I'd want to 
understand what the threat model is before inventing solutions.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.