Subject: [weirds] Rate limiting

From: Simon Perreault <[email-masked]>
Date: Thu, 14 Mar 2013 18:15:14 -0400

I don't remember it being mentioned at the mic, so here's a pointer to 
HTTP status code 429 that can be used for rate limiting:

http://tools.ietf.org/html/rfc6585#section-4

Simon
-- 
DTN made easy, lean, and smart --> http://postellation.viagenie.ca
NAT64/DNS64 open-source        --> http://ecdysis.viagenie.ca
STUN/TURN server               --> http://numb.viagenie.ca

---

From: Arturo Servin <[email-masked]>
Date: Thu, 14 Mar 2013 18:41:49 -0400

Thanks.

	We are testing rate-limiting in our server, this is going to be very
useful.

Regards.
as

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Fri, 15 Mar 2013 02:09:43 -0400

Thanks, this will be indeed useful!

~Carlos

---

From: Michael Young <[email-masked]>
Date: Fri, 15 Mar 2013 09:17:04 -0400

Just out of curiosity, how would you implement that? 

Here's my sensibility on poor acting clients, where its a logged in known client I initially sleep the offending client connection on basis of the known user. I don't break the connection, I don't provide information in the response. If they keep going I extend the sleep periods and apply them to other client connections under the same client login. Why? I don't want to work my server hard by responding even with an error message to their likely largely distributed computing resources - sleeping their response also idles their computer resources at least a bit.

For current Whois I limit based on the source IP address and if they switch to near IPs I start to apply the rate limiting to the netblock. I escalate the netblock as high as I need to or add other blocks if they keep hopping around. Sleep periods are longer the more the abuse continues. This approach breaks down under IPv6, where vast amounts of IPv6 space is assigned I'm not so confident there isn't collateral harm in blocking such large swathes of IP space. This is where I start to believe all public Whois service should require the client to sign up for a user ID and login. Now I can rate limit by that userid versus IP.

I don't give error responses due the server resource conservation and because almost all server pounding is deliberate or it's incompetence with perhaps a dash of willful ignorance. I'd rather publish a clear policy and description of what happens if you trigger my rate limiting.

Michael Young

M:647-289-1220

On 2013-03-15, at 2:09, "Carlos M. Martinez" <[email-masked]> wrote:

> _______________________________________________
> weirds mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/weirds

---

From: Simon Perreault <[email-masked]>
Date: Fri, 15 Mar 2013 09:33:25 -0400

> Just out of curiosity, how would you implement that?

that = ?

> Here's my sensibility on poor acting clients, where its a logged in
> known client I initially sleep the offending client connection on
> basis of the known user. I don't break the connection, I don't
> provide information in the response. If they keep going I extend the
> sleep periods and apply them to other client connections under the
> same client login. Why? I don't want to work my server hard by
> responding even with an error message to their likely largely
> distributed computing resources - sleeping their response also idles
> their computer resources at least a bit.

I don't see why you couldn't return 429 after sleeping.

> For current Whois I limit based on the source IP address and if they
> switch to near IPs I start to apply the rate limiting to the
> netblock. I escalate the netblock as high as I need to or add other
> blocks if they keep hopping around. Sleep periods are longer the more
> the abuse continues. This approach breaks down under IPv6, where vast
> amounts of IPv6 space is assigned I'm not so confident there isn't
> collateral harm in blocking such large swathes of IP space. This is
> where I start to believe all public Whois service should require the
> client to sign up for a user ID and login. Now I can rate limit by
> that userid versus IP.

I don't see why it wouldn't work similarly for IPv6. Just use larger 
prefixes.

> I don't give error responses due the server resource conservation and
> because almost all server pounding is deliberate or it's incompetence
> with perhaps a dash of willful ignorance. I'd rather publish a clear
> policy and description of what happens if you trigger my rate
> limiting.

So why not return 429 after sleeping?

Simon

---

From: Michael Young <[email-masked]>
Date: Fri, 15 Mar 2013 09:39:48 -0400

I actually return the requested response after sleeping, why? It's conditioning, follow my rules and play nice and you can then have what you asked for,....

Stick carrot stick carrot and so on

But yes you could respond with a 429, I just think if they know that's coming after the sleep they will run a very low time-out and hit the server with a new request again eating server resources

"That"= your overall approach to rate limiting such as what I just covered

Michael Young

M:647-289-1220

On 2013-03-15, at 9:33, Simon Perreault <[email-masked]> wrote:

> I don't see why it wouldn't work similarly for IPv6. Just use larger prefixes.
>

---

From: Simon Perreault <[email-masked]>
Date: Fri, 15 Mar 2013 09:57:41 -0400

Le 2013-03-15 09:39, Michael Young a écrit :
> I actually return the requested response after sleeping, why? It's
> conditioning, follow my rules and play nice and you can then have
> what you asked for,....
>

The advantage of the 429 response is that the load on the server may 
actually be lower since it doesn't need to keep connections open. That 
means less state on the server.

See also the Retry-After header that can be used to let the client know 
when to retry:
http://tools.ietf.org/html/rfc2616#section-14.37

> But yes you could respond with a 429, I just think if they know
> that's coming after the sleep they will run a very low time-out and
> hit the server with a new request again eating server resources

Keep in mind that you're not trying to defend against evil clients.
Those can just open multiple connections, or open/close a single
connection very quickly to try to hurt you. What we're considering here
is good-intentioned clients that happen to query too fast by mistake. 
Returning 429+Retry-After lets them know what is going on so that they 
can adjust their behaviour. Sleeping does not. It may even trigger 
timeout conditions in the client, which does not help anyone.

Simon

---

From: Michael Young <[email-masked]>
Date: Fri, 15 Mar 2013 10:15:38 -0400

Sure I'm defending against evil clients as well,.... Why assume its all well intentioned? Data mining is huge in our industry

Michael Young

M:647-289-1220

On 2013-03-15, at 9:57, Simon Perreault <[email-masked]> wrote:

> The advantage of the 429 response is that the load on the server may actually be lower since it doesn't need to keep connections open. That means less state on the server.
> 
> See also the Retry-After header that can be used to let the client know when to retry:
> is good-intentioned clients that happen to query too fast by mistake. Returning 429+Retry-After lets them know what is going on so that they can adjust their behaviour. Sleeping does not. It may even trigger timeout conditions in the client, which does not help anyone.
> 
> Simon
> --

---

From: Simon Perreault <[email-masked]>
Date: Fri, 15 Mar 2013 10:22:55 -0400

> Sure I'm defending against evil clients as well,.... Why assume its all well intentioned? Data mining is huge in our industry

You misunderstood my point. Let me try again.

You *have* to implement a set of defences against evil clients. That 
doesn't change whether you sleep or send 429, because both of them *do 
not* defend against evil clients. Evil clients can easily hurt you even 
if you sleep or send 429. Sleeping and 429 are not defences against evil 
clients. Something else is needed. Something like progressive 
blacklisting of IP prefixes, as you described.

Now, after you have implemented that set of defences against evil 
clients, you may then want to address the problem of good-intentioned 
clients that happen to query too quickly by mistake. That problem is 
addressed by giving clients *information* about what kind of behaviour 
is acceptable, so that they can adjust theirs. Status code 429 and the 
Retry-After header are the HTTP standard ways of conveying that 
information. Sleeping only gives implicit information, which the client 
needs to heuristically discover. It is better to provide explicit 
information because it is more robust and unambiguous.

Simon

---

From: Arturo Servin <[email-masked]>
Date: Fri, 15 Mar 2013 10:42:06 -0400

Michael,

	Checking the details, I will post something soon.

Regards,
as

---

From: Michael Young <[email-masked]>
Date: Fri, 15 Mar 2013 10:46:59 -0400

Ok thanks I understand what you meant now.  I'm not sure I agree that sleeping the bad clients isn't effective - it certainly worked for me in the past. But I do agree its not a solution by itself, progressive behaviors are needed. 

I do think that having authenticated Whois as a standard helps us all identify good and bad actors more easily. You can then vary your behaviors based on user profiling - that means you can opt to be more helpful to what you determine to be a genuinely well intentioned but poorly behaving client.  It even allows for out of band direction or instruction via contact info gathered during the user signup process.

I don't see any real reason not to go to authenticated Whois - privacy issues can still be addressed - you can opt to allow users to remain anonymous and we are still way better off than today, IMHO.

Btw Good talk Simon, your thoughts have merit.

Michael Young

M:647-289-1220

On 2013-03-15, at 10:22, Simon Perreault <[email-masked]> wrote:

> You *have* to implement a set of defences against evil clients. That doesn't change whether you sleep or send 429, because both of them *do not* defend against evil clients. Evil clients can easily hurt you even if you sleep or send 429. Sleeping and 429 are not defences against evil clients. Something else is needed. Something like progressive blacklisting of IP prefixes, as you described.
> 
> Now, after you have implemented that set of defences against evil clients, you may then want to address the problem of good-intentioned clients that happen to query too quickly by mistake. That problem is addressed by giving clients *information* about what kind of behaviour is acceptable, so that they can adjust theirs. Status code 429 and the Retry-After header are the HTTP standard ways of conveying that information. Sleeping only gives implicit information, which the client needs to heuristically discover. It is better to provide explicit information because it is more robust and unambiguous.
> 
> Simon
> --

---

From: "Carlos M. Martinez" <[email-masked]>
Date: Fri, 15 Mar 2013 10:52:15 -0400

Our rate limiting implementation deals with database harvesting issues
and privacy requirements that were set by our community a long time ago
for our legacy port 43 service.

It is not meant as a defense against DoS attacks.

We have three time windows (1 min, 5 min and 1 day, easily configurable
in our RDAP server, not so much in the port 43 server), a different
limit for each per source IP and we also have black and white IP lists.

For the RDAP server we have introduced API keys, where the limits for
the time windows can be set per API key entry.

We'll be providing implementation details to the list.

regards

~Carlos

---

From: Michael Young <[email-masked]>
Date: Fri, 15 Mar 2013 14:23:43 -0400

I look forward to seeing those details, particularly what you did with the API keys, thanks Carlos.

Michael Young

M:647-289-1220

---

From: Francisco Obispo <[email-masked]>
Date: Fri, 15 Mar 2013 11:34:42 -0700

I think it's all about the design,

We only serve whois queries from the database if they are not in the local cache, and even so, we don't query a master server, we query a replica.

We're implementing the same functionality on the RDAP server, so it takes advantage of this model as well.

This is one of the reason why it's a good idea to keep the objects separate, because those can be cached individually. If you cache the full response, you later have to re-fetch an object that already was used in another response.

francisco

>>> Now, after you have implemented that set of defences against evil clients, you may then want to address the problem of good-intentioned clients that happen to query too quickly by mistake. That p…

Francisco Obispo 
Director of Applications and Services - ISC
email: [email-masked]
Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
PGP KeyID = B38DB1BE