Subject: [regext] Partial response and reverse search in RDAP

From: Mario Loffredo <[email-masked]>
Date: Tue, 19 Sep 2017 14:29:31 +0200

Hi all,

I have implemented two new extensions in .it RDAP prototype server 
(https://rdap.pubtest.nic.it): *partial response *and *reverse search*.
I send this mail to know your opinion about them. Personally, I think 
they could be very useful. If you are interested in these proposals, I 
would like to submit two Internet-Draft documents.
In the following, I briefly report the rationale for their implementation.

The idea of using *partial response* in RESTful API design is quite 
simple: instead of returning full objects in API responses with all the 
data fields, only a subset of data fields are returned. The benefit is 
obvious: less data transfered over the network means less bandwidth 
usage, faster server response, less CPU time spent on the server and 
client, as well as less memory usage on the client. Several leading APIs 
providers (e.g. LinkedIn, Facebook, Google) give developers just what 
they need in responses by providing an optional query parameter called 
/fields /after which you put the fields you want to be returned. In 
other contexts, for example in digital libraries, servers provide 
responses according to different tag sets. Basicly, they allow users to 
specify “brief” to have back a short response and “full” to have back 
the complete response.Currently, in RDAP, there is no way for a client 
to request a partial response. According to the user access level, the 
server can only provide the client with the complete response. Some 
examples of simple responses are reported in RFC7483 (Fig.19 and Fig.20) 
but they cannot be requested, where it is possible, as an alternative to 
full responses. Futhermore, servers cannot define their result set 
limits according to partial responses. This results in an efficiency 
mostly when users submit search queries.
Partial response in the .it RDAP server can be requested by adding the 
/fieldSet /parameter to the query string. Even if field set option is 
less flexible than fields option, it is easier to implement, more 
interoperable and more compliant to access levels. For this proof of 
concepts, I have defined three field sets:

/- id/: it contains only the identifier of each object (handle for 
entity, ldhName for domain and nameserver)
- /brief/: it contains some fields to provide a “simple” response/
- full/: it contains all the information the server can provide

Full is the default field set in this proof. Obviously, every RDAP 
implementation can define its available field sets but, to improve 
client-server interoperability, I think that at least servers should 
provide some common field sets.
I have implemented partial response only for search query where it makes 
more sense.  Result set limits are different according to the requested 
field set.//

All of us know *reverse search*, its pros and its cons. ICANN itself, in 
the “Final Report from the Expert Working Group on gTLD Directory 
Services: A Next-Generation Registration Directory Service (RDS)”, 
states that reverse Whois is one of the key benefits of the Next-Gen RDS 
if implemented enforcing requestor accreditation, authentication, 
authorization, and terms and conditions of RDS data use. Usually, 
registries (.it included) receiving requests for reverse searches use 
out-of-band mechanisms to provide a response. The main reasons behind 
such requests are:the loss of synchronization between the registrar and 
the registry databases and the need of such collections to perform 
massive EPP updates (for example, the change of the technical contact 
for a lot of domains). So when I talk about reverse search, I think of a 
controlled access for registrars which are allowed to search only for 
their own domains.
Currently, in RDAP, the only reverse operation allowed is to find the 
collection of domains related to a nameserver (ldhName or ip).
In this proof of concepts, reverse search has been implemented only for 
domains-entities relationship (the classic reverse Whois scenario) . 
Allowed paths are entityHandle and entityFn (not both). Partial matching 
is allowed too. Reverse search must be restricted with respect to a 
single entity role (in .it allowed values are: registrar, registrant, 
technical, administrative).  I think that it makes no sense to search 
for more than a role or for no role but, anyway, it is not a strict 
constraint.
There are no relevant performance issues if you use indexed fields in 
the search and if you don't use partial match with a prefixed wildcard 
as in the case of the standard search paths.
The response is really fast when you specify the "id" field set because 
the payload for the DBMS is minimum.

Here in the following two examples using both the extensions:

https://rdap.pubtest.nic.it/domains?entityHandle=NC01-ITNIC&entityRole=registrant&count=true&fieldSet=id 
<https://rdap.pubtest.nic.it/domains?entityHandle=NC01-ITNIC&entityRole=registrant&count=true&fieldSet=brief>

https://rdap.pubtest.nic.it/domains?entityHandle=NC02-ITNIC&entityRole=technical&count=true&fieldSet=brief 
<https://rdap.pubtest.nic.it/domains?entityHandle=NC02-ITNIC&entityRole=technical&count=true&fieldSet=full>

The examples make use of the count parameter that has been already 
presented in a draft about sorting and paging operators.
All the operators above are briefly described in the help response.
Unfortunately, our public test environment cannot be compared to the 
live one in terms of data amount and performance but, I hope, you will 
get an idea of the two extensions, anyway.
Sorry for the long mail.
I look forward for your feedback.

Regards,
Mario

-- 
Dr. Mario Loffredo
Servizi Internet e Sviluppo Tecnologico
CNR - Istituto di Informatica e Telematica
via G. Moruzzi 1, I-56124 PISA, Italy
E-Mail: [email-masked]
Phone: +39 050 3153497
Web: http://www.iit.cnr.it/mario.loffredo

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Tue, 19 Sep 2017 12:51:47 +0000

Mario, I think these are ideas worth exploring. I’d like to see those I-Ds published.

Scott

From: regext [mailto:[email-masked]] On Behalf Of Mario Loffredo
Sent: Tuesday, September 19, 2017 8:30 AM
To: [email-masked]
Subject: [EXTERNAL] [regext] Partial response and reverse search in RDAP

Hi all,

I have implemented two new extensions in .it RDAP prototype server (https://rdap.pubtest.nic.it): partial response and reverse search.
I send this mail to know your opinion about them. Personally, I think they could be very useful. If you are interested in these proposals, I would like to submit two Internet-Draft documents.
In the following, I briefly report the rationale for their implementation.

The idea of using partial response in RESTful API design is quite simple: instead of returning full objects in API responses with all the data fields, only a subset of data fields are returned. The benefit is obvious: less data transfered over the network means less bandwidth usage, faster server response, less CPU time spent on the server and client, as well as less memory usage on the client. Several leading APIs providers (e.g. LinkedIn, Facebook, Google) give developers just what they need in responses by providing an optional query parameter called fields after which you put the fields you want to be returned. In other contexts, for example in digital libraries, servers provide responses according to different tag sets. Basicly, they allow users to specify “brief” to have back a short response and “full” to have back the complete response.  Currently, in RDAP, there is no way for a client to request a partial response. According to the user access level, the server can only provide the client with the complete response. Some examples of simple responses are reported in RFC7483 (Fig.19 and Fig.20) but they cannot be requested, where it is possible, as an alternative to full responses. Futhermore, servers cannot define their result set limits according to partial responses. This results in an efficiency mostly when users submit search queries.
Partial response in the .it RDAP server can be requested by adding the fieldSet parameter to the query string. Even if field set option is less flexible than fields option, it is easier to implement, more interoperable and more compliant to access levels. For this proof of concepts, I have defined three field sets:

- id: it contains only the identifier of each object (handle for entity, ldhName for domain and nameserver)
- brief: it contains some fields to provide a “simple” response
- full: it contains all the information the server can provide

Full is the default field set in this proof. Obviously, every RDAP implementation can define its available field sets but, to improve client-server interoperability, I think that at least servers should provide some common field sets.
I have implemented partial response only for search query where it makes more sense.  Result set limits are different according to the requested field set.

All of us know reverse search, its pros and its cons. ICANN itself, in the “Final Report from the Expert Working Group on gTLD Directory Services: A Next-Generation Registration Directory Service (RDS)”, states that reverse Whois is one of the key benefits of the Next-Gen RDS if implemented enforcing requestor accreditation, authentication, authorization, and terms and conditions of RDS data use. Usually, registries (.it included) receiving requests for reverse searches use out-of-band mechanisms to provide a response. The main reasons behind such requests are:  the loss of synchronization between the registrar and the registry databases and the need of such collections to perform massive EPP updates (for example, the change of the technical contact for a lot of domains). So when I talk about reverse search, I think of a controlled access for registrars which are allowed to search only for their own domains.
Currently, in RDAP, the only reverse operation allowed is to find the collection of domains related to a nameserver (ldhName or ip).
In this proof of concepts, reverse search has been implemented only for domains-entities relationship (the classic reverse Whois scenario) . Allowed paths are entityHandle and entityFn (not both). Partial matching is allowed too. Reverse search must be restricted with respect to a single entity role (in .it allowed values are: registrar, registrant, technical, administrative).  I think that it makes no sense to search for more than a role or for no role but, anyway, it is not a strict constraint.
There are no relevant performance issues if you use indexed fields in the search and if you don't use partial match with a prefixed wildcard as in the case of the standard search paths.
The response is really fast when you specify the "id" field set because the payload for the DBMS is minimum.

https://rdap.pubtest.nic.it/domains?entityHandle=NC01-ITNIC&entityRole=registrant&count=true&fieldSet=id<https://rdap.pubtest.nic.it/domains?entityHandle=NC01-ITNIC&entityRole=registrant&count=true&fieldSet=brief>

https://rdap.pubtest.nic.it/domains?entityHandle=NC02-ITNIC&entityRole=technical&count=true&fieldSet=brief<https://rdap.pubtest.nic.it/domains?entityHandle=NC02-ITNIC&entityRole=technical&count=true&fieldSet=full>

The examples make use of the count parameter that has been already presented in a draft about sorting and paging operators.
All the operators above are briefly described in the help response.
Unfortunately, our public test environment cannot be compared to the live one in terms of data amount and performance but, I hope, you will get an idea of the two extensions, anyway.
Sorry for the long mail.
I look forward for your feedback.

Regards,
Mario

E-Mail: [email-masked]<mailto:[email-masked]>

---

From: Andrew Newton <[email-masked]>
Date: Tue, 19 Sep 2017 11:00:26 -0400

I agree with Scott. Both ideas are worth discussion.

-andy

On Tue, Sep 19, 2017 at 8:51 AM, Hollenbeck, Scott
<[email-masked]> wrote:
> (https://rdap.pubtest.nic.it): partial response and reverse search.
> I send this mail to know your opinion about them. Personally, I think they
> could be very useful. If you are interested in these proposals, I would like
> to submit two Internet-Draft documents.
> The idea of using partial response in RESTful API design is quite simple:
> instead of returning full objects in API responses with all the data fields,
> only a subset of data fields are returned. The benefit is obvious: less data
> transfered over the network means less bandwidth usage, faster server
> response, less CPU time spent on the server and client, as well as less
> memory usage on the client. Several leading APIs providers (e.g. LinkedIn,
> Facebook, Google) give developers just what they need in responses by
> providing an optional query parameter called fields after which you put the
> fields you want to be returned. In other contexts, for example in digital
> libraries, servers provide responses according to different tag sets.
> Basicly, they allow users to specify “brief” to have back a short response
> and “full” to have back the complete response.  Currently, in RDAP, there is
> no way for a client to request a partial response. According to the user
> access level, the server can only provide the client with the complete
> response. Some examples of simple responses are reported in RFC7483 (Fig.19
> and Fig.20) but they cannot be requested, where it is possible, as an
> alternative to full responses. Futhermore, servers cannot define their
> result set limits according to partial responses. This results in an
> efficiency mostly when users submit search queries.
> fieldSet parameter to the query string. Even if field set option is less
> flexible than fields option, it is easier to implement, more interoperable
> and more compliant to access levels. For this proof of concepts, I have
> defined three field sets:
>
> client-server interoperability, I think that at least servers should provide
> some common field sets.
> field set.
> All of us know reverse search, its pros and its cons. ICANN itself, in the
> “Final Report from the Expert Working Group on gTLD Directory Services: A
> Next-Generation Registration Directory Service (RDS)”, states that reverse
> Whois is one of the key benefits of the Next-Gen RDS if implemented
> enforcing requestor accreditation, authentication, authorization, and terms
> and conditions of RDS data use. Usually, registries (.it included) receiving
> requests for reverse searches use out-of-band mechanisms to provide a
> response. The main reasons behind such requests are:  the loss of
> synchronization between the registrar and the registry databases and the
> need of such collections to perform massive EPP updates (for example, the
> change of the technical contact for a lot of domains). So when I talk about
> reverse search, I think of a controlled access for registrars which are
> allowed to search only for their own domains.
> domains-entities relationship (the classic reverse Whois scenario) . Allowed
> paths are entityHandle and entityFn (not both). Partial matching is allowed
> too. Reverse search must be restricted with respect to a single entity role
> (in .it allowed values are: registrar, registrant, technical,
> administrative).  I think that it makes no sense to search for more than a
> role or for no role but, anyway, it is not a strict constraint.
> There are no relevant performance issues if you use indexed fields in the
> search and if you don't use partial match with a prefixed wildcard as in the
> case of the standard search paths.
> The examples make use of the count parameter that has been already presented
> in a draft about sorting and paging operators.
> Unfortunately, our public test environment cannot be compared to the live
> one in terms of data amount and performance but, I hope, you will get an
> idea of the two extensions, anyway.
> Sorry for the long mail.
> _______________________________________________
> regext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/regext
>