Subject: Re: #90: multipart/byteranges

From: Mark Nottingham <[email-masked]>
Date: Wed, 10 Dec 2008 12:08:53 +1100

4.4 says:

> 4.If the message uses the media type "multipart/byteranges", and the
>   ransfer-length is not otherwise specified, then this self-
>   elimiting media type defines the transfer-length. This media type
>   UST NOT be used unless the sender knows that the recipient can arse
>   it; the presence in a request of a Range header with ultiple byte-
>   range specifiers from a 1.1 client implies that the lient can parse
>   multipart/byteranges responses.
>
>     A range header might be forwarded by a 1.0 proxy that does not
>     understand multipart/byteranges; in this case the server MUST
>     delimit the message using methods defined in items 1,3 or 5 of
>     this section.

Ignoring the missing characters (which should be fixed by now), a few  
things strike me;

1) the most direct way to address the immediate concern would be to  
restrict the use of multipart/byteranges *as a delimiter*; e.g.,

"If a 206 Partial Content response message uses the media type  
"multipart/byteranges", ..."

2) The "This media type MUST NOT be used..." requirement is a non- 
sequitur here, and probably belongs in the appendix that defines the  
media type, or possibly in 206 Partial Content. Doing so will  
necessitate reformulating the note into something like "Servers MUST  
NOT use multipart/byteranges to delimit the message when sending to  
HTTP 1.0 clients."

Beyond that, there's still an open question of whether we want to  
discourage its use as a delimiter (without disallowing it); Henrik,  
you say as much in  <http://lists.w3.org/Archives/Public/ietf-http-wg/2007OctDec/0202.html 
 > -- is this still the case?

Cheers,

On 16/11/2008, at 3:19 AM, Henrik Nordstrom wrote:

> On fre, 2008-11-14 at 18:49 -0800, Mark Nottingham wrote:
>> I think the issue is that allowing a special case where it is allowed
>> makes parsers significantly more complex, and isn't in wide use  
>> anyway.
>
> Now I am confused what this is about.
>
> multipart/byteranges as delimiter is defenitely in wide use for 206
> responses.
>
> What is not supported in wide use is multipart/byteranges as delimiter
> for any other purpose such as sending requests..
>
> I don't see how we can disallow multipart/byteranges as delimiter. All
> clients sending multi-range requests MUST support it in 206  
> responses to
> such requests. A client not supporting multipart/byteranges as  
> delimiter
> MUST NOT sent a multi-ranged request without also using Connection:
> close.
>
> Regards
> Henrik

--
Mark Nottingham     http://www.mnot.net/

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 10 Dec 2008 12:43:37 +0100

> "multipart/byteranges", ..."

In 2616 it's already restricted to 206 responses to multirange requests,
as that's the only case where support for multipart/byteranges is
negotiated. 

> NOT use multipart/byteranges to delimit the message when sending to  

There is already in 4.4 .4 "This media type (M)UST NOT be used unless
the sender knows that the recipient can (p)arse it;"

>  > -- is this still the case?

Yes. There is absolutely no reason to broaden the scope of
multipart/byteranges as delimiter wihin HTTP/1.1. If the length isn't
known use chunked, period.

Anyone thinking of using this as delimiter outside multirange 206
responses must first make sure to negotiate this support on a hop-by-hop
basis (it's a transport feature, not a message property)

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 11 Dec 2008 00:10:39 +1100

> as that's the only case where support for multipart/byteranges is

Where?

2616 says

> When an HTTP 206 (Partial Content) response message includes the  
> content of multiple ranges (a response to a request for multiple non- 
> overlapping ranges), these are transmitted as a multipart message- 
> body. The media type for this purpose is called "multipart/ 
> byteranges".

but that doesn't preclude other uses.

> the sender knows that the recipient can (p)arse it;"

I'm proposing that this requirement be rewritten and relocated because  
it's confusing as it is; taken out of context, the requirement states  
that the media type can only be used when the sender knows that the  
recipient can parse it, which makes sense, given its use to convey  
multiple byteranges.

However, in context, it has a completely different meaning -- that the  
media type must not be used *to delimit messages* unless (...).

Even if your interpretation is the latter one, a much simpler  
formulation is that it can't be used for message delimitation with  
HTTP/1.0 receivers (next hop).

"the presence in a request of a Range header with ultiple byte-range  
specifiers from a 1.1 client implies that the lient can parse  
multipart/byteranges responses" isn't true or relevant in the context  
of message delimitation; a HTTP/1.0 proxy with a HTTP/1.1 client  
behind it will send multiple byte-range specifiers and not know  
anything about using multipart/byteranges for message delimitation.

> basis (it's a transport feature, not a message property)

I think we're in agreement here.

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 10 Dec 2008 14:52:23 +0100

> media type must not be used *to delimit messages* unless (...).

Well, one is a subset of the other. You can't use it as delimiter if you
can't use the media type.

> HTTP/1.0 receivers (next hop).

That's already there in the next sentence of the same 4.4 §4.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Wed, 10 Dec 2008 22:37:13 +0100

> HTTP/1.0 receivers (next hop).

No, it MST NOT be used for message delimitation to ANY recipient
including HTTP/1.1 clients unless they indicate they support the media
type (where the only method doing so in 2616 is multirange requests).
And MUST NOT be used as delimiter if there is reason to suspect you are
talking to a HTTP/1.0 proxy.

Additionally the media type as such SHOULD NOT be used in contexts where
it does not make sense, but that's a pretty minor and obvious to anyone
I'd say.

The text in 4.4 §4 is mainly about delimiting. The fact that it also by
sideeffect restricts when multipart/byteranges may at all be sent is
coincidental and not the main purpose of that text. It can not be moved
out from 4.4 without loosing context, but I am quite fine with having
4.4 narrowed down to only limit when multipart/byteranges is valid as
delimiter. It's already covered sufficiently at other places how it
should be used as media type in it's normal intended use (see
Content-Range), and there is not really any reason why 4.4 should limit
other applications of the media type as such, only it's property as
delimiter.

Proposed updated text:

   4.If the message uses the media type "multipart/byteranges", and the
     transfer-length is not otherwise specified, then this self-
     delimiting media type defines the transfer-length. This media type
     MUST NOT be used as delimiter unless the sender knows that the
     next hop recipient can parse it; the presence in a request of a
     Range header with multiple byte-range specifiers from a 1.1
     client implies that the client can parse multipart/byteranges
     responses.

Note: I am not entirely happy with the language in "the presence in a
request of a Range ..." but that may just be my limited english.. kept
the original text as much as possible and that text do say the right
thing.

Regards
Henrik

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 11 Dec 2008 11:42:28 +1100

I agree that all necessary information is already in the spec. For  
whatever reason, however, people are being confused by it.

I think that the core of the issue is that the current text --  
intentionally or not -- leaves the door open for use of multipart/ 
byteranges as a delimiter in other situations, provided that the  
sender "knows" that the receiver can parse it (one way of knowing this  
is specified, but others may exist).

As such, the question is whether we want to leave that door open.  
Either way, the intent should be clearly stated.

Separately, the requirements are structured in an odd way that's  
difficult to read; I'm concerned that this will only be more  
pronounced, since message parsing and byterange operation are being  
split into separate sections. I  don't see why preserving this  
confusing language is desirable.

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 11 Dec 2008 01:59:58 +0100

> is specified, but others may exist).

I don't have a problem with that, but needs to be clarified that this is
a hop-by-hop transport property, not a message property. (done in the
proposed text).

Regards
Henrik

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 10 Dec 2008 17:25:27 -0800

On Dec 10, 2008, at 4:42 PM, Mark Nottingham wrote:

> sender "knows" that the receiver can parse it (one way of knowing  
> this is specified, but others may exist).
>

I say "no" -- close the loophole and remove any notion that multipart
delimits message framing.

....Roy

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 11 Dec 2008 02:38:16 +0100

> delimits message framing.

Unfortunately that's not entirely an option. We need to have parsers
still understand this delimiting of 206 responses as they are in
widespread use and specified so in 2616 (and 2068).

But I am entirely fine with denying servers from using this method of
delimiting and instead wrap multipart/byteranges in chunked encoding,
and keeping the parser requirements only for compatibility with earlier
HTTP/1.1 specifiations.

Regards
Henrik

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 10 Dec 2008 18:12:39 -0800

> widespread use and specified so in 2616 (and 2068).

They aren't used that way.  I don't know of any implementation
that uses the multipart delimiter to find the end of body (they
use CL, chunked, or a closed connection).

> and keeping the parser requirements only for compatibility with  

I am entirely fine with removing multiple byteranges as a feature,
since I regard it as a DoS problem, but that's another matter.

....Roy

---

From: Henrik Nordstrom <[email-masked]>
Date: Thu, 11 Dec 2008 03:38:00 +0100

> use CL, chunked, or a closed connection).

Hmm.. you may be right. On a closer inspection I see that I misread some
of the data. Sorry for the confusion.

So it looks like dropping 4.4 §4 entirely may be an option.

Any other opinions?

Regards
Henrik

---

From: Dan Winship <[email-masked]>
Date: Thu, 11 Dec 2008 09:47:24 -0500

> use CL, chunked, or a closed connection).

And even if they wanted to, they couldn't, because the handful of
existing server implementations of multipart/byteranges-delimited
responses are inconsistent about whether or not the response ends with a
CRLF. (About 3/4 use a CRLF and 1/4 don't. If you carefully compare part
3, 3.3.2 against the multipart-body grammar in RFC 2046, it probably
says that the minority is correct here, and the CRLF is forbidden.)

So another vote for banning the special byteranges-delimited response
type. And Part 3, 3.3.2 also needs to be revisited.

-- Dan

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 11 Dec 2008 17:17:18 +0000

> So it looks like dropping 4.4 §4 entirely may be an option.

If it's possible to drop support for multipart/byterange delimited
messages, because it's not used anyway, that would be a really good
thing imho.  It's an ugly, pointless, unused and probably poorly
implemented complication in message parsers.

-- Jamie

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 12 Dec 2008 11:13:29 +1100

It seems like there's pretty strong support for removing multipart/ 
related as a message delimiter (I.e., removing RFC2616 section 4.4,  
bullet #4).

However, I want to move carefully here, because this is a change at a  
very low level, with potentially far-reaching effects.

To that end, could folks who have implementations, or have tested  
them, please fill out the implementation spreadsheet;
   http://spreadsheets.google.com/ccc?key=pAZLaupBqGqXrtsaexTjPYg&hl=en
I've added an appropriate column on the clients, intermediaries and  
servers sheets.

(if you can't or don't have time to post to the spreadsheet, please  
send to the list).

Cheers,

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 14 Dec 2008 20:16:03 +0000

> I've added an appropriate column on the clients, intermediaries and  

I'm a bit surprised that the spreadsheet isn't viewable read-only
without a Google account.  It redirects to a login page.  Is there a
"public" readable link?

-- Jamie

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 15 Dec 2008 08:20:13 +1100

Hmm, I ticked 'view without signing in', but that doesn't seem to work  
(although there may be some propagation lag...).

Try:
   http://spreadsheets.google.com/pub?key=pAZLaupBqGqXrtsaexTjPYg
in the meantime.

---

From: Dan Winship <[email-masked]>
Date: Mon, 15 Dec 2008 08:50:14 -0500

Jamie Lokier wrote:
> without a Google account.

I wasn't able to edit it either, even with a google account.

-- Dan