Subject: Re: Retry-After header on 20X response -- HTTP/1.1 spec extension?

From: Bryce Nesbitt <[email-masked]>
Date: Fri, 21 Jan 2011 12:52:31 -0800

I would like to propose a change to the HTTP specification, to permit the
Retry-After header to be used on a 20x HTTP response.

I don't expect browsers to respect this.  My target is cooperative automated
harvesting agents, such as
http://www.openarchives.org/OAI/openarchivesprotocol.htm<http://www.openarchives.org/OAI/openarchivesprotocol.html>

The scenario here are robots that harvest huge data sets, but are
deliberately rate limited.

If I can legally return Retry-After with the 20x result, it will cut the
total requests in half.  Clients can ask for data, and know immediately how
long to wait before they ask again.  Only a client that violates the timeout
would ever see a 503.

The HTTP/1.1 spec is pretty clear (in section 14.37) that Retry-After is for
503 and 3xx return codes only.

*Under the proposal*
Client request data
Server sends 200 response, with Retry-After 5 seconds
Client waits 5 seconds and requests data
Server sends 200 response, with Retry-After 3 seconds

*Steps required under the current specification:*
Request data
200 response
Request data
503 response, Retry-After 5 seconds
(Wait 5 seconds) Request data
200 response
Request data
503 response, Retry-After 5 seconds

On Wed, Apr 15, 2009 at 12:01 AM, Mark Nottingham <[email-masked]> wrote:

> Hi Bryce,
>
> What's the advantage of using Retry-After over Cache-Control here?
>
> Cheers,
>
>
>
> On 05/01/2009, at 2:46 PM, Bryce Nesbitt wrote:
>
>  Dear http-wg members.
>>
>> Where would I go to propose a specification change to HTTP such as the one
>> below (allowing optional Retry-After headers in a 20X response)?  This is a
>> backwards compatible change, and need not have any browser support to be
>> valuable to cooperating automated harvesting robots (e.g.
>> http://www.openarchives.org/OAI/openarchivesprotocol.html ).
>>
>> On Mon, Dec 22, 2008 at 8:51 PM, Bryce Nesbitt <[email-masked]>
>> wrote:
>> Dear Working Group Folks,
>>
>> I am not a member of the working group.  But I have recently been tempted
>> to "stretch" the HTTP spec, and I'm writing to inquire if what I'm doing is
>> reasonable enough to eventually fold into the spec.
>>
>> Basically I'm sending a Retry-After header on a 20x HTTP response.
>>
>> I'm working with a "throttled" data service which rate limits connections.
>>   Clients are harvesting a huge volumes of data over time.  Presently
>> clients get some data with a 200 result, ask again right away and get a 503
>> response, then wait out the proper Retry-After time.
>>
>> If I can return Retry-After with the 20x result, it will cut the total
>> requests in half.  Clients can ask for data, and know immediately how long
>> to wait before they ask again.  Only a client that violates the timeout
>> would ever see a 503.
>>
>> The HTTP/1.1 spec is pretty clear (in section 14.37) that Retry-After is
>> for 503 and 3xx return codes only. Your thoughts?  Where would I go to
>> suggest an expansion of the Retry-After header, to be inclusive of 20x
>> results?  Is this a reasonable extension in your view?
>>
>>
>
>
>
>
>
> Dear Working Group Folks,
>
>
>> I am not a member of the working group.  But I have recently been tempted
>> to "stretch" the HTTP spec, and I'm writing to inquire if what I'm doing is
>> reasonable enough to eventually fold into the spec.
>
>
>> Basically I'm sending a Retry-After header on a 20x HTTP response.
>
>
> I'm working with a "throttled" data service which rate limits
> connections.   Clients are harvesting a huge volumes of data over time.
> Presently clients get some data with a 200 result, ask again right away and
> get a 503 response, then wait out the proper Retry-After time.
>
>
> If I can return Retry-After with the 20x result, it will cut the total
> requests in half.  Clients can ask for data, and know immediately how long
> to wait before they ask again.  Only a client that violates the timeout
> would ever see a 503.
>
>
> The HTTP/1.1 spec is pretty clear (in section 14.37) that Retry-After is
> for 503 and 3xx return codes only. Your thoughts?  Where would I go to
> suggest an expansion of the Retry-After header, to be inclusive of 20x
> results?  Is this a reasonable extension in your view?
>
>  --
> Mark Nottingham     http://www.mnot.net/
>
>
>

-- 
Bryce Nesbitt
The Berkeley Electronic Press
bepress: sustainable scholarly publishing

---

From: Bryce Nesbitt <[email-masked]>
Date: Fri, 21 Jan 2011 13:14:58 -0800

My specific proposal is in italics below, compared to the existing HTTP 1.1
specification.  For those of you reading this in plain text, it is the
paragraph discussing a MAY rule for code 2xx:

14.37 Retry-After

The Retry-After response-header field can be used with a 503 (Service
Unavailable) response to indicate how long the service is expected to be
unavailable to the requesting client.
This field MAY also be used with any 3xx (Redirection) response to indicate
the minimum time the user-agent is asked wait before issuing the redirected
request.
*This field MAY be included with any 2xx (OK) response to indicate rate
limiting: the minimum time the user-agent is asked wait before issuing a
subsequent request of the same resource.  Should the user-agent violate that
timeout, a 503 response MAY be issued.*

The value of this field can be either an HTTP-date or an integer number of
seconds (in decimal) after the time of the response.

       Retry-After  = "Retry-After" ":" ( HTTP-date | delta-seconds )

Two examples of its use are

       Retry-After: Fri, 31 Dec 1999 23:59:59 GMT
       Retry-After: 120

In the latter example, the delay is 2 minutes.