Subject: Re: [weirds] [Regops] Search Engines Indexing RDAP Server Content

From: "John R Levine" <[email-masked]>
Date: 29 Jan 2016 11:47:50 -0500

> So I saw a tweet from Gavin Brown (@GavinBrown) that describes how one 
> particular search engine has indexed the RDAP server of a gTLD registry 
> operator:
> 
> https://twitter.com/GavinBrown/status/692718904058191872
> 
> This is all the more reason to work on a client authentication specification 
> that includes support for varying responses based on client identity and 
> authorization. I've been working on such a specification and welcome feedback 
> on the approach:

I don't see what the problem is.  If you set up an http server that 
returns interlinked data, search engines will find it and index it.  All 
the information RDAP returns to an unauthenticated query is presumably 
public, so what's the harm in making it easier to find?

But anyway, if you don't want them to do that, there's plenty of ways to 
keep them out.

The easiest is to publish a /robots.txt file.  All legit search engines 
will stay out if that's what it says.

Another is to look at the agent string the client sends.  Google's is 
googlebot, Bing's is bingbot, Yahoo's is scooter.  It's easy enough to 
find a list of common spider names.  If the agent is a spider, tell it to 
go away or redirect it to a help page.

Another is to look at the Accept: header.  An RDAP client should ask for a 
JSON media type.  For a client that asks for html or anything else, return 
an html version with meta fields in the header saying NOINDEX and 
NOFOLLOW.

The big search engines spider at low speed from hosts all over the world 
to avoid overloading the sites they index.  You're not going to keep them 
out via authentication without also keeping out everyone else who doesn't 
have a password.  I don't think that's a good idea.

R's,
John

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Fri, 29 Jan 2016 16:54:35 +0000

> -----Original Message-----
> From: weirds [mailto:[email-masked]] On Behalf Of John R
> Levine
> Sent: Friday, January 29, 2016 11:48 AM
> To: [email-masked]
> Subject: Re: [weirds] [Regops] Search Engines Indexing RDAP Server
> Content
> 

[snip]

> have a password.  I don't think that's a good idea.

We have a difference of opinion, because I tend to think that it's a *good* idea to restrict access to *some* data based on a client's identify and level of authorization. I do not want *my* PII appearing in search engine search results because it's accessible via RDAP.

Scott

---

From: "John Levine" <[email-masked]>
Date: 29 Jan 2016 17:23:38 -0000

>We have a difference of opinion, because I tend to think that it's a *good* idea to restrict access
>to *some* data based on a client's identify and level of authorization. I do not want *my* PII
>appearing in search engine search results because it's accessible via RDAP.

Sorry, but I really don't understand what you want here.  There have
been vertical WHOIS search engines forever that let you search WHOIS
data in various ways.  Most of them charge money, but as often as not
some access is free.  That horse left the barn a decade ago.

The stuff that search engines can see is exactly the same as what
casual users see.  If you're saying that you want to make some stuff
unavailable to anyone who doesn't have a password, OK, but now aren't
you running into the whole ICANN mess of what's in WHOIS and what
isn't?

R's,
John

PS: On the third hand, if you want to make it slightly harder for
casual searches to find RDAP info, publishing a robots.txt that says
"Disallow: /" should take about 30 seconds. and will keep the entire
site out of all of the legit search engines.

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Fri, 29 Jan 2016 17:31:21 +0000

> From: John Levine [mailto:[email-masked]]
> Sent: Friday, January 29, 2016 12:24 PM
> To: [email-masked]
> Cc: Hollenbeck, Scott

Yes, definitely. I see little point in deploying RDAP if all we get is JSPON-encoded WHOIS data. We have a tool that let's us fix the whole "all data visible to everyone" problem. We should try to use it.

Scott