Subject: #311 Add limitations to Range to reduce its use as a denial-of-service tool

From: Yves Lafon <[email-masked]>
Date: Wed, 4 Jan 2012 09:00:38 -0500 (EST)

Here is a proposal to clarify a few things to help mitigating the 
overlapping range DoS issue:

1/ split section 4 in Combining Range in Caches (the actual content)
    and add a specific section about combining ranges in responses.
    (with a link to security considerations, and link back from security
    section to this newly created section.

2/ Responses to single and multiple range requests / Combining Ranges in a 
response.

Add:
<<
    Servers have liability to combine requested ranges when those ranges
    are ovelapping.
[Move this part of 5.2 here]
<<
    When an HTTP message includes the content of a single range (for
    example, a response to a request for a single range, or to a request
    for a set of ranges that overlap without any holes), this content is
    transmitted with a Content-Range header field, and a Content-Length
    header field showing the number of bytes actually transferred.  For
    example,

      HTTP/1.1 206 Partial Content
      Date: Wed, 15 Nov 1995 06:25:24 GMT
      Last-Modified: Wed, 15 Nov 1995 04:58:08 GMT
      Content-Range: bytes 21010-47021/47022
      Content-Length: 26012
      Content-Type: image/gif

    When an HTTP message includes the content of multiple ranges (for
    example, a response to a request for multiple non-overlapping
    ranges), these are transmitted as a multipart message.  The multipart
    media type used for this purpose is "multipart/byteranges" as defined
    in Appendix A.

    A response to a request for a single range MUST NOT be sent using the
    multipart/byteranges media type.  A response to a request for
    multiple ranges, whose result is a single range, MAY be sent as a
    multipart/byteranges media type with one part.  A client that cannot
    decode a multipart/byteranges message MUST NOT ask for multiple
    ranges in a single request.

    When a client requests multiple ranges in one request, the server
    SHOULD return them in the order that they appeared in the request.

Slash the following paragraph

<<
    If the server receives a request (other than one including an If-
    Range header field) with an unsatisfiable Range header field (that
    is, all of whose byte-range-spec values have a first-byte-pos value
    greater than the current length of the selected resource), it SHOULD
    return a response code of 416 (Requested range not satisfiable)
    (Section 3.2).
As it is redundant, and add an example to 3.2 and the following note:
<<
       Note: Clients cannot depend on servers to send a 416 (Requested
       range not satisfiable) response instead of a 200 (OK) response for
       an unsatisfiable Range header field, since not all servers
       implement this header field.

<<
HTTP/1.1 416 Requested Range Not Satisfiable
Content-Range: */42
...

Always allow the server to ignore range requests

<<
    If the server ignores a byte-range-spec because it is syntactically
    invalid, the server SHOULD treat the request as if the invalid Range
    header field did not exist.  (Normally, this means return a 200
    response containing the full representation).
becomes
<<
    If the server ignores a byte-range-spec (for example if it is
    syntactically, or if it may be seen as a denial-of-service attack),
    the server SHOULD treat the request as if the invalid Range header
    field did not exist.  (Normally, this means return a 200
    response containing the full representation).

As in all cases where the server detect some potential abuses/DoS attempts 
etc... It can close the connection, response with a 403 or do whatever is 
needed.

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 05 Jan 2012 10:47:57 +1300

On Wed, 4 Jan 2012 09:00:38 -0500 (EST), Yves Lafon wrote:
> 2/ Responses to single and multiple range requests / Combining Ranges
> in a response.
>
> Add:
> <<

 Is the phrasing "have liability" defined somewhere? I may have missed 
 it, but it still took me a few minutes to realise what this sentence 
 meant despite being part of the discussions earlier around Range.

>    SHOULD return them in the order that they appeared in the request.

 This last part seems to contradict my understanding of the last 
 discussion consensus. That ranges could be combined to reduce DoS 
 footprint.

 Did I get that completely out of place? or is this use of SHOULD a 
 downgrade intended to permit it?

>       implement this header field.

 +1 on that hunk.

>    response containing the full representation).

 The above omit to say what should be done with the Content-Range header 
 when altering the status. We are already facing responses like:
   HTTP/1.1 200 OK
   Content-Length: 1230
   Content-Range: 2-50/1234

 If that 200 status were to be naively treated as ignoring Content-Range 
 this could potentially lead to problems with the excess Content-Length. 
 It would be helpful to say drop the Content-Range header or something 
 similar during the status alteration.

>
> As in all cases where the server detect some potential abuses/DoS
> attempts etc... It can close the connection, response with a 403 or 
> do
> whatever is needed.

 AYJ

---

From: Yves Lafon <[email-masked]>
Date: Wed, 4 Jan 2012 19:18:59 -0500 (EST)

On Thu, 5 Jan 2012, Amos Jeffries wrote:

> Is the phrasing "have liability" defined somewhere? I may have missed it, but 
> it still took me a few minutes to realise what this sentence meant despite 
> being part of the discussions earlier around Range.

Hum, I will rewrite that, but you got the spirit of it.

> This last part seems to contradict my understanding of the last discussion 
> consensus. That ranges could be combined to reduce DoS footprint.
>
> Did I get that completely out of place? or is this use of SHOULD a downgrade 
> intended to permit it?

I don't see a contradiction there, it just says that the ranges should be
sent in order, however when ranges are combined, ordering can become
invalid (if the combined ranges were not directly following each other), 
so yes, it's the use of a SHOULD isntead of a MUST that allows that, while 
giving general guidelines for the general "normal" case.

> The above omit to say what should be done with the Content-Range header when 
> altering the status. We are already facing responses like:
>  HTTP/1.1 200 OK
>  Content-Length: 1230
>  Content-Range: 2-50/1234
>
> If that 200 status were to be naively treated as ignoring Content-Range this 
> could potentially lead to problems with the excess Content-Length. It would 
> be helpful to say drop the Content-Range header or something similar during 
> the status alteration.

This has been fixed by #301
http://trac.tools.ietf.org/wg/httpbis/trac/ticket/301

<<
    The "Content-Range" header field has no meaning for status codes that
    do not explicitly describe its semantic.  Currently, only status
    codes 206 (Partial Content) and 416 (Requested range not satisfiable)
    describe the meaning of this header field.

So in your example, the Content-Range: header must be ignored.

>> attempts etc... It can close the connection, response with a 403 or do
>> whatever is needed.
>
> AYJ
>
>

         ~~Yves

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 05 Jan 2012 14:24:35 +1300

On Wed, 4 Jan 2012 19:18:59 -0500 (EST), Yves Lafon wrote:
>> Is the phrasing "have liability" defined somewhere? I may have 
>> missed it, but it still took me a few minutes to realise what this 
>> sentence meant despite being part of the discussions earlier around 
>> Range.
>

 Yes. Cheers. :)

> invalid (if the combined ranges were not directly following each

 invalid? I don't see that in the new texts. single-part "multipart" is 
 MAY, which should at most solve any validity issue on merging.

> other), so yes, it's the use of a SHOULD isntead of a MUST that 
> allows
> that, while giving general guidelines for the general "normal" case.
>

 Thanks. That answers the initial Q.

 AYJ

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 05 Jan 2012 15:46:10 +1300

while we're on the topic of range requests, there are an increasing 
number of agents that fail when range requests are responded to with a 
200 and full content.

Things like

* MS Windows update
* iTunes
* AppleTV client
* various other updaters
* anything relying on BITS in Windows.

etc.

This is a conflict with any gateway AV scanning which requires the 
entire entity in order to scan it and which therefore removes Range 
headers from requests.

Could/should we add language to send a stronger message to agent authors 
to deter them from such behaviour?  It makes little or no sense to post 
an error about a misconfigured gateway simply because it downgrades all 
range requests to full requests.  The entire entity is still available 
to be sent back to the client, it just refuses to play if it can't get 
its way.

it should be clear that it's entirely valid for a server to respond to a 
Range request with 200 and full content, and entirely valid for a server 
to supply a single range response to even a multiple range request, and 
that range may or may not even cover all the ranges requested.  It 
should be entirely up to any client which wishes to use ranges to sort 
out what bits it gets out of what it asked for and make subsequent 
requests accordingly.

Adrien

On 5/01/2012 2:24 p.m., Amos Jeffries wrote:
>> other), so yes, it's the use of a SHOULD isntead of a MUST that allows

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com
WinGate 7 is released! - http://www.wingate.com/getlatest/