Subject: Re: [httpapi] RateLimit Header client implementations

From: Alex Martinez <[email-masked]>
Date: Mon, 4 Jan 2021 13:48:39 +0000

Hi Darrel,

On Wed, Dec 30, 2020 at 3:17 AM Darrel Miller <[email-masked]> wrote:

> [...]

>
> From what I understand there are a few scenarios where clients can use
> these headers:
>
>
>
>    1. Optimizing polling frequency:  i.e. check quota and window and
>    determine optimum calling interval
>    2. Prioritization between client tasks:  Don’t let low priority tasks
>    starve high priority tasks of quota
>    3. Enable user agents to avoid hitting rate limit due to some kind of
>    punitive behavior.
>
>
>
> Are you aware of other scenarios?  Are there implementations of these
> scenarios?
>

The scenario about "hitting rate limiting" is quite broad in scope. You can
have rate limits apply per API endpoint, per user identity, per connection,
per time of day, per infrastructure resources, or even per service type or
cost, or a combination of them all - which is quite a bit more than just
limiting some misbehavior.

The immediate benefit of these headers to service providers is being able
to signal expectations to clients so they can adapt. Sometimes it is just a
matter of adjusting to some specific rate of resource consumption. Some
other times the only way to adapt is just to stop using a service entirely
for a while or use another endpoint, and that's more useful than just
returning a 403 or 50X without further information, ie. during a
maintenance window you can use the headers to signal that window with 0
quota units and avoid useless traffic.

I can't think of any other specific scenarios and I don't know yet of
generic client implementations targeting specifically these headers. My
hope is that such clients support just enough of these headers to reap the
main benefits, that is, basic rate limiting information enabling some form
of throttling.

Cheers,
  Alex

---

From: Darrel Miller <[email-masked]>
Date: Tue, 5 Jan 2021 16:17:04 +0000

Hey Alex,

> From: httpapi <[email-masked]> on behalf of Alex Martinez <[email-masked]>
> Hi Darrel,
[...] 
>>  ... 
>>  Enable user agents to avoid hitting rate limit due to some kind of punitive behavior.
>>  

> The scenario about "hitting rate limiting" is quite broad in scope. You can have rate limits
>  apply per API endpoint, per user identity, per connection, per time of day, per infrastructure 
> resources, or even per service type or cost, or a combination of them all - which is quite a
>  bit more than just limiting some misbehavior.

Sorry, I wasn't clear.  I completely understand the variety of ways that API providers need to limit excessive behavior by clients.  I was meaning something different.  Some API providers we go beyond simply returning a 429 and a retry-after header to slow down the consumer.  They will "punish" the client for reaching the limits. Some services will "tarpit" the client by deliberately slowing the response down.  I believe Roberto mentioned about some services dropping the connection.  

My assertion is that if we discouraged API providers from punishing consumer for reaching a rate limit, and we promote the use of the retry-after as a signal to the client to back-off before making another call, then there are fewer scenarios where a client is required to constantly monitor "the number of calls left before reaching the limit".  Returning and monitoring "calls remaining" is potentially an expensive operation, especially when multiple clients can consume the same quota, e.g. in the case of calls per user that is using multiple applications.

> The immediate benefit of these headers to service providers is being able to signal expectations 
> to clients so they can adapt.

The question I have is with the existing non-standard headers, how many people have actually bothered to write this adaptive client code?  Do we believe that lack of standardization has been sufficient to discourage people from writing that code?  Are the only people writing that code the ones who are trying to avoid punitive behavior?  If so, maybe a best practices document that recommends not doing that would be more constructive.

>  Sometimes it is just a matter of adjusting to some specific rate of 
> resource consumption. Some other times the only way to adapt is just to stop using a service 
> entirely for a while or use another endpoint, and that's more useful than just returning a 403 
> or 50X without further information, ie. during a maintenance window you can use the headers 
> to signal that window with 0 quota units and avoid useless traffic.

Why would a 503 with a retry-after header with a period that matches the maintenance window not be sufficient here?

Darrel

---

From: Alex Martinez <[email-masked]>
Date: Thu, 7 Jan 2021 10:01:22 +0000

Hi Darrel,

On Tue, Jan 5, 2021 at 4:17 PM Darrel Miller <[email-masked]>
wrote:

>
> My assertion is that if we discouraged API providers from punishing
> consumer for reaching a rate limit, and we promote the use of the
> retry-after as a signal to the client to back-off before making another
> call, then there are fewer scenarios where a client is required to
> constantly monitor "the number of calls left before reaching the limit".
> Returning and monitoring "calls remaining" is potentially an expensive
> operation, especially when multiple clients can consume the same quota,
> e.g. in the case of calls per user that is using multiple applications.
>

That's a valid observation. That's why clients receiving these headers
should consider them a hint and the next request they perform could be
responded with new data in the headers that would make little sense if
taken in isolation without further knowledge of what could be consuming the
quotas. Service specific policies specified in the headers could help
reduce that uncertainty, but those are too broad and diverse, so there's
been little discussion on the topic other than allowing people to specify
those so service specific clients can make use of those extra bits of
information.

> The question I have is with the existing non-standard headers, how many
> people have actually bothered to write this adaptive client code?  Do we
> believe that lack of standardization has been sufficient to discourage
> people from writing that code?  Are the only people writing that code the
> ones who are trying to avoid punitive behavior?  If so, maybe a best
> practices document that recommends not doing that would be more
> constructive.
>

Yes, I believe that because there is no standardization there's been little
to no effort in writing such adaptive code in generic HTTP clients, but
this is pure speculation. I think it is very likely that most people doing
so are the ones writing service specific clients, and in many cases they
could be the same people that wrote the service in the first place, unless
the service happens to see lots of usage across the board with many
independent clients. At a minimum, retry-after could be used to signal
generic clients that no request should be performed for a while, but that
falls short of the minimum information that generic clients could use with
these headers.

> Why would a 503 with a retry-after header with a period that matches the
> maintenance window not be sufficient here?
>

You are right, the immediate effect is the same. That said, you would be
lacking some extra information about the nominal quota units and duration
of the current window and the windows applying after this one is elapsed
(ie. after recovery you'd have information about the expected rates, and
those could also signal some sort of punitive policy), though that
information might be next to useless in the current window since the
intended effect is to stop traffic altogether for a while. Yet it would not
work in a closely related scenario in which you would have some partial
maintenance work but keeping the ability to serve clients in a degraded
service manner -- that is, unless you kept giving clients shorter
no-service windows interleaved with service responses as usual, hence
generating a lot more wasteful traffic.

Cheers,
  Alex

---

From: Darrel Miller <[email-masked]>
Date: Sun, 10 Jan 2021 16:27:02 +0000

Alex,

Thanks for the additional context.
You are right, the immediate effect is the same. That said, you would be lacking some extra information about the nominal quota units and duration of the current window and the windows applying after this one is elapsed (ie. after recovery you'd have information about the expected rates, and those could also signal some sort of punitive policy), though that information might be next to useless in the current window since the intended effect is to stop traffic altogether for a while. Yet it would not work in a closely related scenario in which you would have some partial maintenance work but keeping the ability to serve clients in a degraded service manner -- that is, unless you kept giving clients shorter no-service windows interleaved with service responses as usual, hence generating a lot more wasteful traffic.
Would be useful to clarify that these additional headers are intended to support scenarios where rate limiting constraints applied by a service are dynamic over time?
 Yes, I believe that because there is no standardization there's been little to no effort in writing such adaptive code in generic HTTP clients, but this is pure speculation. I think it is very likely that most people doing so are the ones writing service specific clients, and in many cases they could be the same people that wrote the service in the first place, unless the service happens to see lots of usage across the board with many independent clients.
I look forward to seeing some feedback from implementers on the value derived from generic client code that handles these headers.

Darrel

---

From: Alex Martinez <[email-masked]>
Date: Wed, 13 Jan 2021 17:46:45 +0000

Hi Darrel,

On Sun, Jan 10, 2021 at 4:27 PM Darrel Miller <[email-masked]>
wrote:

> Would be useful to clarify that these additional headers are intended to
> support scenarios where rate limiting constraints applied by a service are
> dynamic over time?
>

Sure, we can look into clarifying this - perhaps my own reading of the
document makes this obvious to me having worked on rate limiting stuff for
some time.

>  Yes, I believe that because there is no standardization there's been
> little to no effort in writing such adaptive code in generic HTTP clients,
> but this is pure speculation. I think it is very likely that most people
> doing so are the ones writing service specific clients, and in many cases
> they could be the same people that wrote the service in the first place,
> unless the service happens to see lots of usage across the board with many
> independent clients.
>
> I look forward to seeing some feedback from implementers on the value
> derived from generic client code that handles these headers.
>

Understandably these don't exist yet - the easy thing to do is implementing
support for these in servers that do some sort of rate limiting or
integrate with a component that does, and that's why most (all?) of the
implementations we know of are server code. I also look forward to the
possibilities in clients, and it would be great to have basic support in
the most commonly used HTTP client libraries, but admittedly the
implementation work is not trivial. I don't really know whether such
projects would go for an I-D even if it's already on the standards track,
but we can certainly ask!

Cheers,
  Alex

---

From: Roberto Polli <[email-masked]>
Date: Fri, 15 Jan 2021 14:11:14 +0100

Hi @all and sorry for the late reply.

Il giorno mer 13 gen 2021 alle ore 18:47 Alex Martinez
<[email-masked]> ha scritto:
> On Sun, Jan 10, 2021 at 4:27 PM Darrel Miller <[email-masked]> wrote:
> Sure, we can look into clarifying this - perhaps my own reading of the document makes this obvious to me having worked on rate limiting stuff for some time.

This is addressed under §This Proposal and §Goals:
- https://github.com/ietf-wg-httpapi/ratelimit-headers/blame/main/draft-ietf-httpapi-ratelimit-headers.md#L143
- https://github.com/ietf-wg-httpapi/ratelimit-headers/blame/main/draft-ietf-httpapi-ratelimit-headers.md#L175

but if the statements are not clear, suggestions are very welcome. As
Alex said, a "first reader" eye would probably make a better proposal.

> [.. ] the implementation work is not trivial. I don't really know whether such projects would go for an I-D even if it's already on the standards track, but we can certainly ask!

Great. On my side, the main obstacle to the implementation was the
inability to identify the headers and their content.
How clients want to adapt their request policies to the quota-policies
exposed by service providers is not an easy guess:
some may want to focus on maximizing the throughput, others to avoid
peaks and distribute requests fairly across time-frames,
...

Our goal here is to allow clients to identify request policies and be
able to act accordingly. Some behaviors might even
be implemented in general-purpose libraries, but in many cases the
logic will be managed and implemented according
to the specific business goals of the clients.

My2¢ + Peace,
R: