Subject: Re: #311 Add limitations to Range to reduce its use as a denial-of-service tool

From: Mark Nottingham <[email-masked]>
Date: Tue, 7 Feb 2012 13:53:05 +1100

The Introduction already says:

> The protocol for range requests
>    is an OPTIONAL feature of HTTP, designed so resources or recipients
>    that do not implement this feature can respond as if it is a normal
>    GET request without impacting interoperability.

Cheers,

On 05/01/2012, at 1:46 PM, Adrien de Croy wrote:

> 
> while we're on the topic of range requests, there are an increasing number of agents that fail when range requests are responded to with a 200 and full content.
> 
> Things like
> 
> * MS Windows update
> * iTunes
> * AppleTV client
> * various other updaters
> * anything relying on BITS in Windows.
> 
> etc.
> 
> This is a conflict with any gateway AV scanning which requires the entire entity in order to scan it and which therefore removes Range headers from requests.
> 
> Could/should we add language to send a stronger message to agent authors to deter them from such behaviour?  It makes little or no sense to post an error about a misconfigured gateway simply because it downgrades all range requests to full requests.  The entire entity is still available to be sent back to the client, it just refuses to play if it can't get its way.
> 
> it should be clear that it's entirely valid for a server to respond to a Range request with 200 and full content, and entirely valid for a server to supply a single range response to even a multiple range request, and that range may or may not even cover all the ranges requested.  It should be entirely up to any client which wishes to use ranges to sort out what bits it gets out of what it asked for and make subsequent requests accordingly.
> 
> Adrien
> 
> 
> On 5/01/2012 2:24 p.m., Amos Jeffries wrote:
>> On Wed, 4 Jan 2012 19:18:59 -0500 (EST), Yves Lafon wrote:
>>> On Thu, 5 Jan 2012, Amos Jeffries wrote:
>>> 
>>>> On Wed, 4 Jan 2012 09:00:38 -0500 (EST), Yves Lafon wrote:
>>>>> Here is a proposal to clarify a few things to help mitigating the
>>>>> overlapping range DoS issue:
>>>>> 1/ split section 4 in Combining Range in Caches (the actual content)
>>>>>   and add a specific section about combining ranges in responses.
>>>>>   (with a link to security considerations, and link back from security
>>>>>   section to this newly created section.
>>>>> 2/ Responses to single and multiple range requests / Combining Ranges
>>>>> in a response.
>>>>> Add:
>>>>> <<
>>>>>   Servers have liability to combine requested ranges when those ranges
>>>>>   are ovelapping.
>>>>>>> 
>>>> 
>>>> Is the phrasing "have liability" defined somewhere? I may have missed it, but it still took me a few minutes to realise what this sentence meant despite being part of the discussions earlier around Range.
>>> 
>>> Hum, I will rewrite that, but you got the spirit of it.
>>> 
>> 
>> Yes. Cheers. :)
>> 
>>>>> [Move this part of 5.2 here]
>>>>> <<
>>>>>   When an HTTP message includes the content of a single range (for
>>>>>   example, a response to a request for a single range, or to a request
>>>>>   for a set of ranges that overlap without any holes), this content is
>>>>>   transmitted with a Content-Range header field, and a Content-Length
>>>>>   header field showing the number of bytes actually transferred.  For
>>>>>   example,
>>>>> 
>>>>>     HTTP/1.1 206 Partial Content
>>>>>     Date: Wed, 15 Nov 1995 06:25:24 GMT
>>>>>     Last-Modified: Wed, 15 Nov 1995 04:58:08 GMT
>>>>>     Content-Range: bytes 21010-47021/47022
>>>>>     Content-Length: 26012
>>>>>     Content-Type: image/gif
>>>>> 
>>>>>   When an HTTP message includes the content of multiple ranges (for
>>>>>   example, a response to a request for multiple non-overlapping
>>>>>   ranges), these are transmitted as a multipart message.  The multipart
>>>>>   media type used for this purpose is "multipart/byteranges" as defined
>>>>>   in Appendix A.
>>>>> 
>>>>>   A response to a request for a single range MUST NOT be sent using the
>>>>>   multipart/byteranges media type.  A response to a request for
>>>>>   multiple ranges, whose result is a single range, MAY be sent as a
>>>>>   multipart/byteranges media type with one part.  A client that cannot
>>>>>   decode a multipart/byteranges message MUST NOT ask for multiple
>>>>>   ranges in a single request.
>>>>> 
>>>>>   When a client requests multiple ranges in one request, the server
>>>>>   SHOULD return them in the order that they appeared in the request.
>>>>>>>> 
>>>> 
>>>> This last part seems to contradict my understanding of the last discussion consensus. That ranges could be combined to reduce DoS footprint.
>>>> 
>>>> Did I get that completely out of place? or is this use of SHOULD a downgrade intended to permit it?
>>> 
>>> I don't see a contradiction there, it just says that the ranges should be
>>> sent in order, however when ranges are combined, ordering can become
>>> invalid (if the combined ranges were not directly following each
>> 
>> invalid? I don't see that in the new texts. single-part "multipart" is MAY, which should at most solve any validity issue on merging.
>> 
>>> other), so yes, it's the use of a SHOULD isntead of a MUST that allows
>>> that, while giving general guidelines for the general "normal" case.
>>> 
>> 
>> Thanks. That answers the initial Q.
>> 
>> AYJ
>> 
> 
> -- 
> Adrien de Croy - WinGate Proxy Server - http://www.wingate.com
> WinGate 7 is released! - http://www.wingate.com/getlatest/
> 
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Yves Lafon <[email-masked]>
Date: Wed, 15 Feb 2012 04:37:49 -0500 (EST)

On Thu, 5 Jan 2012, Adrien de Croy wrote:

>
> while we're on the topic of range requests, there are an increasing number of 
> agents that fail when range requests are responded to with a 200 and full 
> content.
>
> Things like
>
> * MS Windows update
> * iTunes
> * AppleTV client
> * various other updaters
> This is a conflict with any gateway AV scanning which requires the entire 
> entity in order to scan it and which therefore removes Range headers from 
> requests.
>
> Could/should we add language to send a stronger message to agent authors to 
> deter them from such behaviour?  It makes little or no sense to post an error 
> about a misconfigured gateway simply because it downgrades all range requests 
> to full requests.  The entire entity is still available to be sent back to 
> the client, it just refuses to play if it can't get its way.

It remindes me of a WebDAV client that was waiting for the connection to 
be closed after receiving a successful response to a PUT. Keeping the 
connection open was making the client stall, expecting a specific 
behaviour of a particular class of servers.

So what you ask is actually more general, it's "Do not make assumptions on 
the behaviour of the server or a proxy, as observable behaviour can change 
over time" and this is specially true for optional parts of the spec, like 
ranges.

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Yves Lafon <[email-masked]>
Date: Mon, 20 Feb 2012 12:33:58 -0500 (EST)

Based on the feedback received, here is the changeset:

<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1542>

         ~~Yves

---

From: Julian Reschke <[email-masked]>
Date: Mon, 20 Feb 2012 19:08:10 +0100

> <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1542>

I think we need an addition to the "Changes from RFC 2616" section...

---

From: Yves Lafon <[email-masked]>
Date: Mon, 20 Feb 2012 17:07:14 -0500 (EST)

> I think we need an addition to the "Changes from RFC 2616" section...

I think not, for example, saying that a server may coalesce ranges is 
implied in 2616, see
<<
A response to a request for
    multiple ranges, whose result is a single range, MAY be sent as a
    multipart/byteranges media type with one part.
so it is mostly clarifications, no real changes.

         ~~Yves

---

From: Julian Reschke <[email-masked]>
Date: Mon, 20 Feb 2012 23:26:17 +0100

> so it is mostly clarifications, no real changes.

Ack.

So is the outcome of #311 that we actually did not make any normative 
changes?

In which case it would be good to close the bug with a short summary of 
what we changed, and what we did not. (In the issue tracker, not in the 
spec...)

Best regards, Julian