Subject: [weirds] LACNIC implementation of rate limiting and API keys

From: "Carlos M. Martinez" <[email-masked]>
Date: Wed, 17 Apr 2013 13:37:53 -0300

Guys, first of all apologies for the delay. As we promised in Orlando,
here are some notes related to our implementation of rate limits and API
keys.

I also ask the workgroup whether you believe this topic should be
addressed in any of the current drafts, in a new draft, or if it should
be left up to implementers.

thanks,

~Carlos

=======================

LACNIC Restful WHOIS Rate Limit Implementation
Dario Gomez
Gerardo Rada
Arturo Servin
Carlos Martinez

Rate limiting is used to control the rate on which requests to the rdap
service are served. Traffic that is less or equal to the specified rate
of requests is served. Instead, if the request rate exceeds a threshold,
the request produces an error (currently HTTP code 403).

The rdap service that we are operating only allows clients to make a
limited number of requests. This policy affects the service in different
ways. Our policy for the beta rdap service is:
o	100 queries every 5 minutes
o	1000 queries every 60 minutes

By default, the LACNIC’s RDAP server checks the number of queries for
the IP (v4 or v6) address that originated the incoming request. If that
IP address exceeds the rate, the application returns a HTTP 403 error
code with the message “rate limit exceeded for IP: xxx.xxx.xxx.xxx”.

To implement our rate limiting solution, we need to save the requests
status or log the requests for each incoming IP address. Our solution
stores IP addresses and timestamps for every requests received in a
database table.

When a new request is received, we check the counts for previous request
saved on the data table and verify it according with the configured
policies.

As the requests log data table can grow a lot we implemented a clean up
mechanism where every 24 hours every record older than 24 hours is purged.

Some users of our WHOIS service require higher limits. Currently our
port 43 service implements whitelisting for some cases, but other users
are forced to curb their request stream in order to comply with the
current rate limits. For the RDAP service we decided to take advantage
of the more modern nature of the protocol and introduced the possibility
of using API keys in order to provide differentiated services to users
with different requirements.

The API key subsystem allows us to define different query limits for
each key. API keys may be also source-IP-limited, only allowing a given
range of source addresses to use a given key.

In order to make use of an API key an “apkikey” parameter is added to
request URL, as in the following example:
http://rdap.labs.lacnic.net/rdap/entity/AIL?apikey=1234

Currently API keys are to be requested manually by interested parties.

---

From: Jean-Philippe Dionne <[email-masked]>
Date: Wed, 17 Apr 2013 14:06:43 -0400

> be left up to implementers.

Hi Carlos,

Thank you for sharing this info.

I am curious to know if you have considered using the http 
authentication mechanism (or other mechanism suggested by 
draft-ietf-weirds-rdap-sec) instead of the "?apikey=1234" inside the url?

Thanks
Jean-Philippe

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Wed, 17 Apr 2013 15:54:10 -0300

Hello Jean-Philippe,

Our implementation somewhat predates draft-xx-rdap-sec. The apikey +
source-ip restriction is more similar and feels familiar when compared
to our current operational procedures for port 43 WHOIS. Also this is
why we currently return a 403 instead of a 429.

Personally (and this is just my opinion), embedding the apikey in the
URL lends itself more easily for casual users (simple bash scripts or
event just browser access as it is bookmarkable)

That said, we do plan to support draft-xxx-rdap-sec as well, probably
using the same API key as username / password with auth-basic /
auth-digest, although we haven't made a design decision just yet. We
also plan to start sending 429s instead of 403s and to send Retry-After
hints so well-behaved clients can, well, behave themselves :D

Warm regards,

~Carlos

On 4/17/13 3:06 PM, Jean-Philippe Dionne wrote:
> _______________________________________________
> weirds mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/weirds

---

From: Andy Newton <[email-masked]>
Date: Thu, 18 Apr 2013 12:00:58 +0000

>hints so well-behaved clients can, well, behave themselves :D

We have api keys for our templates and restful provisioning system, and we
are considering their use with our Whois restful system (not RDAP, the one
we currently have).

I think draft-ietf-weirds-rdap-sec should mention api keys and define a
manner in which they are used. Api keys are quite common in restful web
services.

The above suggestion of using them with auth-basic,digest looks workable.

-andy

---

From: Kaveh Ranjbar <[email-masked]>
Date: Thu, 18 Apr 2013 14:13:37 +0200

> manner in which they are used. Api keys are quite common in restful web

+1

> The above suggestion of using them with auth-basic,digest looks workable.

How are we going to (nicely) translate an API key to a [username:realm:password] combination? 

In my opinion, these two mechanisms represent two distinct scenarios: in case of an API key, you obtain it after going through an auth. process which will generate a key for you based on your credential privileges or services level VS in case of username, password those privileges or service level restrictions are tied to the account itself.

All the best,
Kaveh.

---

From: Andy Newton <[email-masked]>
Date: Thu, 18 Apr 2013 13:27:19 +0000

From: Kaveh Ranjbar <[email-masked]<mailto:[email-masked]>>
Date: Thursday, April 18, 2013 8:13 AM
To: Andrew Newton <[email-masked]<mailto:[email-masked]>>
Cc: Carlos Martinez <[email-masked]<mailto:[email-masked]>>, Jean-Philippe Dionne <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [weirds] api keys in rdap-sec ( was Re: LACNIC implementation of rate limiting and API keys )

Just theorizing, but we could have the username as "apikey" and the password as the actual api key.

Api keys are handy, but I've never liked the idea that they surface in URIs as parameters.

-andy

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Thu, 18 Apr 2013 14:29:24 +0000

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Andy Newton
> Sent: Thursday, April 18, 2013 8:01 AM
> To: [email-masked]; Jean-Philippe Dionne
> Cc: [email-masked]
> Subject: [weirds] api keys in rdap-sec ( was Re: LACNIC implementation
> of rate limiting and API keys )
> 

Please send text.

Scott

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Thu, 18 Apr 2013 12:56:39 -0300

I was thinking along that lines, too:

Username: rdapclient
Realm: apikey
Password: 0ath1s1smy4p1ke1

What I like about having apikeys in the URLs is the 'bookmarkability'
(!) they provide. However, I'm not religious about it either.

Other possibility would be a custom header, something along the lines of:

X-RDAP-APIKey: 0ath1s1smy4p1ke1

Warm regards,

~Carlos

On 4/18/13 10:27 AM, Andy Newton wrote:
> From: Kaveh Ranjbar <[email-masked] <mailto:[email-masked]>>
> To: Andrew Newton <[email-masked] <mailto:[email-masked]>>
> Cc: Carlos Martinez <[email-masked] <mailto:[email-masked]>>,
> Jean-Philippe Dionne <[email-masked]
> <mailto:[email-masked]>>, "[email-masked]
> <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
> Subject: Re: [weirds] api keys in rdap-sec ( was Re: LACNIC
> implementation of rate limiting and API keys )
> 
>     In my opinion, these two mechanisms represent two distinct
>     scenarios: in case of an API key, you obtain it after going through
>     an auth. process which will generate a key for you based on your
>     credential privileges or services level VS in case of username,
>     password those privileges or service level restrictions are tied to
>     the account itself.
> 
>

---

From: Simon Perreault <[email-masked]>
Date: Thu, 18 Apr 2013 18:02:23 +0200

> Password: 0ath1s1smy4p1ke1

Note that you can't get to the plain-text password if you use an auth 
scheme like digest.

Simon

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Thu, 18 Apr 2013 13:06:12 -0300

Of course. But you can store both api keys and their corresponding
digests in your DB. Or we may decide to only spec plain-text for realm
apikey.

I don't have strong views on the subject... as Woody Allen put it
'Whatever Works' :D

~C.

---

From: Simon Perreault <[email-masked]>
Date: Thu, 18 Apr 2013 18:10:44 +0200

> digests in your DB. Or we may decide to only spec plain-text for realm

If it's not important to get to the plain-text apikey, then I don't 
understand the technical difference between an apikey and a password.

Could anyone explain like I'm a four year old?

Thanks,
Simon

---

From: Jean-Philippe Dionne <[email-masked]>
Date: Thu, 18 Apr 2013 12:16:34 -0400

> The above suggestion of using them with auth-basic,digest looks workable.

Hi,

I am wondering what is be the relation of authentication with the 
redirection mechanism.

What happens when a client is redirected to a server that has a 
different authentication method or different authentication realm?

I guess the client should maintain a list of credentials classified per 
realm, server name and authentication method.

Jean-Philippe

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Thu, 18 Apr 2013 13:23:51 -0300

It's the same idea, the semantics is different.

An username/password set (supposedly) proves identity.

An API key is just a token for some sort of contract between client and
server. In this case, a contract for a given rate limiting policy. It is
not intended to prove identity.

Regards,

~Carlos