Subject: [weirds] Entity name searches

From: Brian Mountford <[email-masked]>
Date: Fri, 6 Nov 2015 15:54:53 -0500

RFC 7483 describes entity name searches. As I understand it, the searches
are to use the same rules as for domains and nameservers: a prefix,
followed by at most one wildcard, followed by some number of domain labels.
I have been assuming that there will be no domain labels for entity names,
so it would just be a prefix search.

My coworker has pointed out to me that this isn't necessarily so useful.
You could search for contacts by first name, but not by last, and so on. Is
that in fact what you have in mind with these sorts of searches?

And only names? There is no provision for searching entities by address,
etc.?

Also, what is to be done with non-ASCII characters? Should the names be
folded to ASCII values, so that a search for "Bohm" will match an O with an
umlaut? Or is that not correct behavior? Is this to be a case-insensitive
search? What about whitespace? We didn't see clear directives in the RFC,
but maybe we missed them.

Thanks.

Brian

---

From: Andrew Newton <[email-masked]>
Date: Sat, 7 Nov 2015 06:47:36 +0900

Welcome to the world of internationalization, where the concept of a
last name vs a first name is not universal, and US ASCII is not
representative all the characters used.

Since you know the data in your database best, you should taylor the
interpretation of the query input to that which works best with your
registry.

> And only names? There is no provision for searching entities by address,

I don't think that ever came up. It would require an RDAP extension. I
do question how useful such a thing would be. Why does anybody care
that a particular registry has contacts living on Mumford Lane in East
Westover? Are they searching all the registries for that information?
Do they need a telephone book instead?

-andy

---

From: Brian Mountford <[email-masked]>
Date: Mon, 9 Nov 2015 12:12:52 -0500

Well, for instance, one might want to search the organization. When I do a
WHOIS query for google.com, the contacts have an organization of Google
Inc. I might want to search for all contacts with that organization.

I can try to tailor the interpretation, but since the search string syntax
does not allow for arbitrary suffix searching, it's not clear what
tailoring is possible. Are you saying that I could take an entity name
search string of "McB*" and interpret that as a search for names any of
whose words begin with McB, so that it would find Joe McBride as well as
McBurns Simpson? That seems to be playing pretty loose with the partial
search string rules, since the RFC takes pains to define that syntax so
precisely.

Brian

---

From: Dave Piscitello <[email-masked]>
Date: Mon, 9 Nov 2015 17:44:59 +0000

More generally, if you are looking at 100,000 registration records that have been associated with spam, you’d want the ability to search or pivot on any data/string that establish relationships among the domain strings (e.g., botnet DGA) but as importantly, _any_ element of the registration record that subsets share: POC, creation date, name server…

> On Nov 9, 2015, at 12:12 PM, Brian Mountford <[email-masked]> wrote:
> 
> Well, for instance, one might want to search the organization. When I do a WHOIS query for google.com <http://google.com/>, the contacts have an organization of Google Inc. I might want to search for all contacts with that organization.
> 
> I can try to tailor the interpretation, but since the search string syntax does not allow for arbitrary suffix searching, it's not clear what tailoring is possible. Are you saying that I could take an entity name search string of "McB*" and interpret that as a search for names any of whose words begin with McB, so that it would find Joe McBride as well as McBurns Simpson? That seems to be playing pretty loose with the partial search string rules, since the RFC takes pains to define that syntax so precisely.
> 
> Brian
> 
> On Fri, Nov 6, 2015 at 4:47 PM, Andrew Newton <[email-masked] <mailto:[email-masked]>> wrote:
> On Sat, Nov 7, 2015 at 5:54 AM, Brian Mountford <[email-masked] <mailto:[email-masked]>> wrote:
> >
> _______________________________________________
> weirds mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/weirds

---

From: Andrew Newton <[email-masked]>
Date: Mon, 9 Nov 2015 15:32:32 -0500

Ok. I understand the use case. It still requires an extension to RDAP.
That type of search never came up during the standardization process.

At ARIN we do offer more targeted RESTful searches, but it has been my
experience they are used almost exclusively for data mining. And I
suspect that if left wide open, that's how it would be used in RDAP
for DNRs. So to get a "system" to work, there would need to be an
authorization system... and since we are talking multiple DNRs, that
would be federated authorization. And because there are multiple DNRs,
these queries would have to be run in parallel against each one (using
the bootstrap to find them). And then you have the problem that some
domain registries are thin. Those are a lot of problems to solve, and
I'm not sure who has the will to do it.

-andy

On Mon, Nov 9, 2015 at 12:44 PM, Dave Piscitello
<[email-masked]> wrote:
> More generally, if you are looking at 100,000 registration records that have
> been associated with spam, you’d want the ability to search or pivot on any
> data/string that establish relationships among the domain strings (e.g.,
> botnet DGA) but as importantly, _any_ element of the registration record
> that subsets share: POC, creation date, name server…
>
> WHOIS query for google.com, the contacts have an organization of Google Inc.
> I might want to search for all contacts with that organization.
>
> does not allow for arbitrary suffix searching, it's not clear what tailoring
> is possible. Are you saying that I could take an entity name search string
> of "McB*" and interpret that as a search for names any of whose words begin
> with McB, so that it would find Joe McBride as well as McBurns Simpson? That
> seems to be playing pretty loose with the partial search string rules, since
> the RFC takes pains to define that syntax so precisely.
>
> Brian
>

---

From: Andrew Newton <[email-masked]>
Date: Mon, 9 Nov 2015 15:45:33 -0500

It depends. Are you trying to do something useful or trying to meet a
conformance checkbox?

If it is the former, I'm unsure who would complain with your broadened
search results. If the issue is that the semantics in the
specification should be better, its best to have a concrete proposal
to put in an Internet Draft.

If you are trying to conform to something like an ICANN agreement, I'd
do as the RFC says because ICANN has signalled they intend to submit
Internet Drafts describing the search capabilities they specifically
need. Anything you guess at today is likely just to be extra work.

-andy

---

From: Dave Piscitello <[email-masked]>
Date: Mon, 9 Nov 2015 20:45:37 +0000

Can we clear up one question before we venture into extensions?

Q: What is the purpose of such an extension versus the ability that is naturally available if one is able to collect structured data into a local data store?

For my example, so long as I’m able to gather the 100,000 structured records, I and others will happily take structured data, incorporate these into SQL or other databases, and search to our hearts’ content. What has inhibited the kinds of _domain name_ investigations my example illustrates in the past was the combination of non-standard, unstructured data, rate limiting, and other completely non-RDAP related matters.

Perhaps I’m missing something, but the extension(s) required to do what I suggested on a query level would require federated {authentication,authorization,auditing… } that would be quite remarkable objective for the numbers world and a monumental undertaking for the domain world, no?

> On Nov 9, 2015, at 3:32 PM, Andrew Newton <[email-masked]> wrote:
>

---

From: Andrew Newton <[email-masked]>
Date: Mon, 9 Nov 2015 17:17:20 -0500

Yes, that is certainly doable but what is your initial query? "Give me
all your data", even if it is 1000 results at a time, is probably not
a good starting point.

-andy

---

From: Dave Piscitello <[email-masked]>
Date: Tue, 10 Nov 2015 02:51:42 +0000

> On Nov 9, 2015, at 5:17 PM, Andrew Newton <[email-masked]> wrote:
> 

I think we’re losing the thread. I was offering an example to illustrate how Whois “search" is used in security/operations today, as a counterpoint to what Brian was asking for (a search on data other than the domain).

Brian asked the original question so he’s best suited to answer “what is your initial query”.

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Tue, 10 Nov 2015 11:55:07 +0000

> -----Original Message-----
> From: weirds [mailto:[email-masked]] On Behalf Of Dave
> Piscitello
> Sent: Monday, November 09, 2015 3:46 PM
> To: Andrew Newton
> Cc: Brian Mountford; Justine Tunney; [email-masked]
> Subject: Re: [weirds] Entity name searches

[snip]

> Perhaps I’m missing something, but the extension(s) required to do what
> I suggested on a query level would require federated
> {authentication,authorization,auditing… } that would be quite
> remarkable objective for the numbers world and a monumental undertaking
> for the domain world, no?

Remarkable? Monumental? I don't think so. The technology exists and is already documented in an Internet-Draft I wrote. The biggest challenge is in developing the authorization policies, and there's no shortage of people doing policy work in this space. This would be time well spent.

Scott

---

From: Dave Piscitello <[email-masked]>
Date: Tue, 10 Nov 2015 12:59:50 +0000

This is only one of the big challenges and my experience is that authorization policies in a multi-stakeholder community are difficult agreements to reach.

And there are several degrees of separation between polices and enforcement through contracts.

However, I’ll defer to your view and wish you well in the endeavor.

For now, however, I’ll declare success if I can get sufficiently unfettered access to RDAP to be able to collect registration data and search/parse it once I have it.

> On Nov 10, 2015, at 6:55 AM, Hollenbeck, Scott <[email-masked]> wrote:
> 
> The biggest challenge is in developing the authorization policies,