Subject: NEW ISSUE: status of multipart/byteranges

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 18 Nov 2007 14:51:30 +0100

Hi,

  There appears to be some confusion as to when multipart/byteranges
bodies have to be inspected to determine the message length. It seems
that is widely considered optional and sometimes limited to ...

  In general, HTTP treats a multipart message-body no differently than
  any other media type: strictly as payload. The one exception is the
  "multipart/byteranges" type (appendix 19.2) when it appears in a 206
  (Partial Content) response ...

... this particular case, even though the specification suggest the
opposite, I read it to say, all implementations have to support that
and support it in all messages, like requests and non-206 responses.
Apache 2.2.6 for example treats

  POST / HTTP/1.1
  Host: example
  Content-type: multipart/byteranges;
    boundary=THIS_STRING_SEPARATES
  
  --THIS_STRING_SEPARATES
  ...

as two requests, a zero-length POST and a --THIS_STRING_SEPARATES to
the root which it does not support (which seems to be a bug in itself).
Would it be possible, for example, to discourage implementations to
ever generate messages where the message length is determined by this
type, and limit having to read it to 206 responses, as the text above
suggests?

regards,
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sun, 18 Nov 2007 17:40:51 +0100

>the root which it does not support (which seems to be a bug in itself).

Actually no, with neither Transfer-Encoding and Content-Length the
message would be seen as having no body and the type would not affect
the message length, so "If the message uses the media type ..." should
really be "If a response uses the media type". This does not affect
the rest of the issue though.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 18 Nov 2007 22:43:30 +0100

> Apache 2.2.6 for example treats

Not quite. See 4.4 Message Length. There is a MUST level requirement
governing when this self-delimiting media type may be used.

Note: The formatting errors/typos is already corrected. Best to read the
2616bis draft, not the original.. but you can probably guess with no
problem what the missing characters should be..

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Nov 2007 03:02:53 +0000

> governing when this self-delimiting media type may be used.

     This media type MUST NOT be used unless the sender knows that the
     recipient can parse it; the presence in a request of a Range
     header with multiple byte-range specifiers from a 1.1 client
     implies that the client can parse multipart/byteranges responses.

This suggests that there might be other situations where the sender
knows that the recipient can parse it, which aren't defined by HTTP.
Such as specific out of band knowledge of the recipient, or prior
negotiation.

It also suggests that even as defined by HTTP, the use of a Range
header with multiple byte-range specifiers from a 1.1 client permits
multipart/byteranges responses in any response to that client, not
just 206, and not just the response corresponding to that specific
request.

But however you look at it, and whether the message is allowed or not,
it is _always_ incorrect to parse a HTTP/1.1 message with media type
"multipart/byteranges" and no Content-Length and no chunked encoding
as though it has an empty body, followed by a new message.

Either parse it as multipart/byteranges, or reject it as malformed.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 19 Nov 2007 05:00:26 +0100

> Such as specific out of band knowledge of the recipient, or prior

Yes.

> just 206, and not just the response corresponding to that specific

You can't safely deduce things this way. It's a MUST NOT which only says
that you are forbidden unless, not that it's otherwise ok. You also have
to combine this requirement with all other requirements around the use
of this media type. But yes, in theory..

> as though it has an empty body, followed by a new message.

Yes, obviously.

> Either parse it as multipart/byteranges, or reject it as malformed.

Yes..

Which makes me wonder how Squid behaves if seeing a multipart/byteranges
in request. Something I need to test. Quite likely it will fail just as
badly as Apache, not expecting this message format in requests.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Nov 2007 02:51:18 +0000

> the rest of the issue though.

In RFC 2616, section 4.4, no mention is made of "206 responses", only:

    This media type MUST NOT be used unless the sender knows that the
    recipient can parse it; the presence in a request of a Range
    header with ultiple byte- range specifiers from a 1.1 client
    implies that the lient can parse multipart/byteranges responses.

This suggests that there might be other circumstances in which a
sender might know that a recipient can parse multipart/byteranges, not
specified in HTTP, otherwise this paragraph would be more specific
that it is only allowed in 206 responses to requests with Range.

A proxy cannot be expected to know what recipients can parse in
general.  They probably should always forward these messages, parsing
the boundaries.

Because "This media type MUST NOT be used unless the sender knows that
recipient can parse it", in the above POST example, any recipient of
this request should reject it as invalid, and because it does not
satisfy the basic rules for message length in section 4.4, responding
with an error and closing the connection without parsing any further
seems like the only safe response.

A server shouldn't parse the next request as if there's an empty body,
even if that's technically allowed, because it's a security hole, if
we believe there is a likelihood of proxies calculating the message
boundary differently when they forward it.

-- Jamie

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 19 Nov 2007 05:47:11 +0100

>governing when this self-delimiting media type may be used.

I can write some client code that can parse such messages, but I would
pass the request through some HTTP library which may then well consult
a proxy and so on, and it's quite likely that somewhere on the line we
have an implementation that does not determine the message length from
the self-delimiters or does so incorrectly and perhaps even forwards a
possibly problematic response.

CPAN for example has modules that help to parse such messages, and the
HTTP client modules allow you to set a Range header with multiple
values, but they are usually not prepared to cope with the self-
delimiting nature of this type. The XMLHttpRequest draft also suggests
to allow scripts to set the Range header, but says nothing about the
implementation having to support the type, and the very text you refer
to suggests that it's okay for clients to have no support for it.

Detecting the end of a message correctly is rather critical, support
for doing this based on the properties of this type is poor, and it's
not easy to implement this correctly, so it seems reasonable to high-
light the problems with this in the specification, and otherwise try
to get it out of sight as much as possible. Ideally I'd like this the
item removed from the Message Length section, and perhaps recommend
to close the connection upon sight of a response where it would have
applied, or require to read until the connection is closed, possibly
due to a timeout; but I don't know whether this would break any impor-
tant real-world setup.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 19 Nov 2007 06:03:26 +0100

>In RFC 2616, section 4.4, no mention is made of "206 responses", only:

Well the item talks about messages in general, so I took that to suggest
a Content-Type: multipart/byteranges indicates the presence of a message
body, but that interpretation is likely incorrect. Without message body,
section 4.4 does not apply at all.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Nov 2007 08:51:46 +0000

> section 4.4 does not apply at all.

I think RFC 2616 section 4.4 is intended to apply to all messages, due
to 4.4 item 1.

If you're reading it really closely, there's a contradiction, so the
language could be improved:

   _When_ a message-body is included with a message, the
   transfer-length of that body is determined by one of the following
   (in order of precedence):

    1. Any response message which "MUST NOT" include a message-body (such
       as the 1xx, 204, and 304 responses and any response to a HEAD
       request) is always terminated by the first empty line after the
       header fields, regardless of the entity-header fields present in
       the message.

The first paragraph ("When...") says the remainder of the section only
applies when there's a message body.

But the second paragraph ("1. Any...") applies when there isn't a
message body.

I take this to mean that the text is a little sloppy, is entirely a
description of how to determine the length of _all_ messages after the
headers and blank line, and can't reasonably be interpreted any other
way.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 19 Nov 2007 11:56:44 +0100

> possibly problematic response.

Forwarding of 206 responses isn't such big problem if the client asked
for it. If there is an intermediary/client component not understanding
the response format it will result in the response hanging on completion
until the server times out the connection. In worst case the client
won't understand how to deal with what it requested, which will be
notices quite quickly. There is some exceptions if pipelining is
involved, but those too is not very critical.

The main problem is if this media type is used in other contexts than
206 responses, most notably for requests where it quite likely gets
wrongly parsed by most implementations.

I would suggest the specifications is narrowed down to only allow this
method of message delimiting for responses, not requests.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 19 Nov 2007 12:53:52 +0100

> boundary differently when they forward it.

There is actually a slight conflict here. 4.3 Message Body says

   The presence of a message-body in a request is signaled by the
   inclusion of a Content-Length or Transfer-Encoding header field in
   the request's message-headers.

The correct resolution is to fix 4.4 Message Length to restrict rule 4
to 206 responses only.

I would like to also deprecate this message delimiting method as
obsolete. chunked encoding fills the gap nicely.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 27 Nov 2007 21:21:19 -0800

Now i90; <http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/issues/#i90>.

--
Mark Nottingham     http://www.mnot.net/