Subject: Re: [regext] Registration Data Access Protocol (RDAP) Query Parameters for Result Sorting and Paging

From: Brian Mountford <[email-masked]>
Date: Tue, 19 Sep 2017 15:20:04 -0400

Mario,

I went back and read your draft about result sorting and paging, because I
am looking to implement paging in our RDAP server. But I have a comment: It
seems like some of these operations will not scale well. For instance, the
count parameter will cause the database to do something like an SQL SELECT
COUNT(*) operation. That generally results in a scan through the table (or
at least an index), I think.

Likewise, paging by means of an offset, as you suggest, might result in
scanning through the result set from the start each time a new page is
requested. Might it be more efficient to have the server return, using your
clever link technique, some sort of cursor value indicating the start of
the next page? Of course, if an implementer wanted the cursor to be the
index of the next row in the result set, it could pass the offset as the
cursor token, but by defining the cursor to be an opaque string without
defined meaning, you would not lock in all implementers to that scheme.

Thanks.

Regards,
Brian

---

From: Mario Loffredo <[email-masked]>
Date: Wed, 20 Sep 2017 18:45:32 +0200

Hi Brian,

Il 19/09/2017 21:20, Brian Mountford ha scritto:
> Mario,
>
> I went back and read your draft about result sorting and paging, 
> because I am looking to implement paging in our RDAP server. But I 
> have a comment: It seems like some of these operations will not scale 
> well. For instance, the count parameter will cause the database to do 
> something like an SQL SELECT COUNT(*) operation. That generally 
> results in a scan through the table (or at least an index), I think.

The draft does not aim at exploring all the possible implementations of 
the count parameter. It only  states that the implementation of new 
parameters is technically feasible because counting, paging and sorting 
operators are currently supported by major RDBMSs and No-SQL DBMSs as 
well. If you are not satisfied of the standard SQL operators peformance 
you can adopt different solutions. If you are considering a RDBMS, there 
are a number of tips to follow in order to obtain the number of results 
and there are available a lot of comparisons between the different 
performances. Obviously, the RDAP search paths must be at least mapped 
on db indexed fields because, not only count, but the row selection can 
take a lot of time.
Under such conditions, SQL SELECT count(*) performs well even if the 
query generates a huge result set.

There are some other facts to consider when you decide to implement RDAP 
searches in general, for example, which types of searches are allowed 
according to the different access levels and, consequently, if it's 
worth to set up an appropriate technical strategy to obtain the best 
performance in any type of searches.
Basicly, the decisions you take in your RDAP profile affect your RDAP 
implementation.

>
> Likewise, paging by means of an offset, as you suggest, might result 
> in scanning through the result set from the start each time a new page 
> is requested. Might it be more efficient to have the server return, 
> using your clever link technique, some sort of cursor value indicating 
> the start of the next page? Of course, if an implementer wanted the 
> cursor to be the index of the next row in the result set, it could 
> pass the offset as the cursor token, but by defining the cursor to be 
> an opaque string without defined meaning, you would not lock in all 
> implementers to that scheme.

The only cursor coming in my mind should be based on the values of the 
sorted field (you need a default one if a value for sortby is not 
provided). This is the most common way to simulate paging. For example, 
if you want to search for a set of domains sorted for ldhName in 
ascending order and the page contains 100 domains, you should use the 
100-th domain ldhName to build a where condition (e.g. "ldhName > 100-th 
ldhName of the current page") of the query producing the next page.
In this case, the value of the ldhName property is the cursor.
However, I see some drawbacks:

- the property or the properties of the cursor have to be a unique key  
because otherwise you will meet some inconsistencies; ldhName is a 
unique key but if you want to sort for registrationDate, you have 
generate the cursor as the concatenation of registrationDate and ldhName 
and this makes the building of the where condition much more complicated
- the building of the where condition depends on both the sorting order 
and the scrolling direction;  the server might decide to provide also a 
link to the previous page
- the use of a cursor does not allow you to jump across the results as 
you want; the server provides a link  to the next page but you might 
want to jump N objects forward
- if you use a cursor, you don't have a perception of the page position 
within the result set, you can derive it by the limit value and the 
number of pages scrolled but it 's not intuitive; the proposed 
counting/paging parameters give you this perception

So the use of a cursor raises some issues too. It could be more 
efficient under some specific conditions but it is less flexible, less 
intuitive and more complicated than offset.
Anyway, here again, I think that all depends on the search capabilities 
the RDAP server allows and, consequently, on the size of the result sets.
IMHO RDAP searches should not deal with huge result sets (hundreds of 
thousands or milions of objects).
The count parameter should be used to evaluate the query precision and 
personally I believe that additional parameters should be introduced in 
RDAP to filter and furtherly restrict a result set.
Another solution to deal with big collections of objects in RDAP could 
be the use of  partial responses (please see my yesterday's mail to the 
WG).
Definitively, if result sets are not huge (less than a hundred thousand 
of domains), offset is not a problem.
I made some tests on some searches dealing with 50000/60000 domains: I 
specified the parameter fieldSet="id" (only objectClassName and ldhName 
in the domainSearchResultsarray) and
set the number of domains per page, initially, to 1000 and then to 5000.
I didn't find relevant perfomance issues.

Thanks for your interest in the draft.
Regards,
Mario

>
> Thanks.
>
> Regards,
> Brian
>
>
> _______________________________________________
> regext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/regext

-- 
Dr. Mario Loffredo
Servizi Internet e Sviluppo Tecnologico
CNR - Istituto di Informatica e Telematica
via G. Moruzzi 1, I-56124 PISA, Italy
E-Mail: [email-masked]
Phone: +39 050 3153497
Web: http://www.iit.cnr.it/mario.loffredo