Subject: Re: issue 85 - range unit extensions

From: Julian Reschke <[email-masked]>
Date: Tue, 05 Aug 2008 18:21:36 +0200

Mark Nottingham wrote:
> ...
> Straw-man proposal (not exact text):
> 
> Range-unit extensions SHOULD NOT be specific to any format. This is 
> because caches are allowed to combine ranges from multiple responses, 
> and to serve range requests out of cache; format-specific range units 
> make it less likely that implementations will be able to do this 
> (although there may be exceptions, hence the SHOULD NOT). An 
> informational reference to p6 to highlight this to potential extenders 
> would be a good idea.
> ...

Playing the devil's advocate: is there *any* range unit other than 
"bytes" we can think of that is indeed independent of formats? Right 
now, I can't think of any.

So if we strongly believe in this requirement we should probably 
consider to close this extension point. That being said, maybe that 
requirement is too strong.

 > ...
> We'll also need to determine what the requirements for registration are 
> (standards-track?), and set up the registry.
> ...

That's quite some effort for something where we're not sure whether it's 
going to be used. Should we consider delaying the registry until later?

> Furthermore, Section 6.2 - the BNF says:
> 
>> Content-Range = "Content-Range" ":" content-range-spec
>>
>> content-range-spec      = byte-content-range-spec
>> byte-content-range-spec = bytes-unit SP
>>                           byte-range-resp-spec "/"
>>                           ( instance-length | "*" )
>>
>> byte-range-resp-spec = (first-byte-pos "-" last-byte-pos)
>>                                | "*"
>> instance-length           = 1*DIGIT
> 
> This does not allow other range units to be used. If we keep them, this 
> BNF needs to be changed to something like:
> 
> Content-Range = "Content-Range" ":" content-range-spec
> 
> content-range-spec      = byte-content-range-spec / ext-content-range-spec
> byte-content-range-spec = bytes-unit SP
>                           byte-range-resp-spec "/"
>                           ( instance-length | "*" )
> 
> byte-range-resp-spec = (first-byte-pos "-" last-byte-pos)
>                                | "*"
> instance-length           = 1*DIGIT
> ext-content-range-spec = other-range-unit SP CHAR*

+1

I'm not totally sure we need to fix this for Microsoft's use case (which 
uses "=" instead of SP). It seems to use SEARCH (can somebody contact 
the author of that spec ... ;-), while "Content-Range" seems to apply 
only to GET. Or doesn't it?

> Note that section 6.2 places several requirements on the use of the 
> Content-Range header which assume that byte-ranges are in use; we'd need 
> to adjust the language appropriately.
> 
> Beyond that, I can't see what else we'd need to specify; everything else 
> is unit-specific.
> 
> 
> Thoughts?

My proposal would be to just fix the grammar in 6.2, and restructure 6.2 
so that the "bytes" specific stuff ends up in a subsection 6.2.1, and we 
add a 6.2.2 talking about extensions.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 5 Aug 2008 11:51:06 -0500

Julian Reschke wrote:
> Mark Nottingham wrote:
> Playing the devil's advocate: is there *any* range unit other 
> than "bytes" we can think of that is indeed independent of 
> formats? Right now, I can't think of any.
> 
> So if we strongly believe in this requirement we should 
> probably consider to close this extension point. That being 
> said, maybe that requirement is too strong.
...
> That's quite some effort for something where we're not sure 
> whether it's going to be used. Should we consider delaying 
> the registry until later?

...
> > This does not allow other range units to be used. If we keep them, 
> > this BNF needs to be changed to something like:
...
> > Content-Range header which assume that byte-ranges are in use; we'd 
> > need to adjust the language appropriately.

I agree with Julian. It seems like a lot of work for something that is
probably never going to interoperate. Since the RFC 2616 grammar and section
6.2 don't even allow for other units it would be best to remove the
extension point. 

If the extension is going to stay because of Microsoft's rows extension,
then it should accommodate their syntax. There's no point in specifying a
grammar that nobody will use and that will be violated by the most common
use of the extension point.

- Brian

---

From: Yves Lafon <[email-masked]>
Date: Tue, 5 Aug 2008 13:27:49 -0400 (EDT)

On Tue, 5 Aug 2008, Julian Reschke wrote:

>> Range-unit extensions SHOULD NOT be specific to any format. This is because 
>> caches are allowed to combine ranges from multiple responses, and to serve 
>> range requests out of cache; format-specific range units make it less 
>> likely that implementations will be able to do this (although there may be 
>> exceptions, hence the SHOULD NOT). An informational reference to p6 to 
>> highlight this to potential extenders would be a good idea.
>> ...
>
> Playing the devil's advocate: is there *any* range unit other than "bytes" we 
> can think of that is indeed independent of formats? Right now, I can't think 
> of any.

The text proposed by Mark uses SHOULD NOT, not MUST NOT. In some case, it 
might be ok to use a format specific range unit, like 'character' (for 
text content in multi/variable-bytes encodings), or 'frames' for video 
content. I am not saying that those are good examples, just that it may 
happen.

> going to be used. Should we consider delaying the registry until later?

Yes, it can probably wait, but if it's a low hanging fruit to add another 
small registry...

>> This does not allow other range units to be used. If we keep them, this BNF 
>> needs to be changed to something like:

Well, is it forbidden to ask servers to give back byte-ranges that a cache
can... cache ? What is transported is bytes, so requesting the 10 first 
characters of an utf-8 document and receiving 12 bytes using a 
Content-Range in bytes might help caches to DTRT, so I am not sure that 
the proposed BNF change is desirable (as "exotic" Content-Ranges won't be 
cached for sure).

> I'm not totally sure we need to fix this for Microsoft's use case (which uses 
> "=" instead of SP). It seems to use SEARCH (can somebody contact the author 
> of that spec ... ;-), while "Content-Range" seems to apply only to GET. Or 
> doesn't it?
>
>
>> Content-Range header which assume that byte-ranges are in use; we'd need to 
>> adjust the language appropriately.
>> 
>> Beyond that, I can't see what else we'd need to specify; everything else is 
>> unit-specific.
>> 
>> 
>> Thoughts?
>
> My proposal would be to just fix the grammar in 6.2, and restructure 6.2 so 
> that the "bytes" specific stuff ends up in a subsection 6.2.1, and we add a 
> 6.2.2 talking about extensions.
>
> BR, Julian
>
>

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Julian Reschke <[email-masked]>
Date: Tue, 05 Aug 2008 20:06:41 +0200

Yves Lafon wrote:
> The text proposed by Mark uses SHOULD NOT, not MUST NOT. In some case, 
> it might be ok to use a format specific range unit, like 'character' 
> (for text content in multi/variable-bytes encodings), or 'frames' for 
> video content. I am not saying that those are good examples, just that 
> it may happen.

I totally agree this could happen. My point was that in practice most 
extensions may be format specific, thus even a SHOULD NOT would be to 
strong.

> the proposed BNF change is desirable (as "exotic" Content-Ranges won't 
> be cached for sure).
 > ...

It would be useful for caches, but not for the client, right? It would 
still need to know what range with respect to the original range unit 
was returned.

I also note that Range (Section 6.4) would need additional work as well.

BR, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 5 Aug 2008 19:30:08 +0100

> now, I can't think of any.

Since we're playing devil's advocate: to be independent of a subset of
formats, it's easy to imagine e.g. things which are specific to
everything under text/* or application/xml+*.

To be fully format independent, bytes have to be involved.  But there
are some interesting "ranges" such as "all portions of the entity
whose MD5 matches A,B,C,D,E,F,G or H", and other logical ways of
slicing and dicing bytes.  Some of those things are even logically
cachable and mergable.  Since we're playing devil's advocate :-)

-- Jamie

---

From: Yves Lafon <[email-masked]>
Date: Tue, 5 Aug 2008 15:25:51 -0400 (EDT)

On Tue, 5 Aug 2008, Julian Reschke wrote:

>> Content-Range in bytes might help caches to DTRT, so I am not sure that the 
>> proposed BNF change is desirable (as "exotic" Content-Ranges won't be 
>> cached for sure).
>> ...
>
> It would be useful for caches, but not for the client, right? It would still 
> need to know what range with respect to the original range unit was returned.

Well, in that case, the range is supposed to match the request, otherwise 
a 416 (or anything else than a 206) would be returned.

> I also note that Range (Section 6.4) would need additional work as well.

Indeed.

         ~~Yves

---

From: Lisa Dusseault <[email-masked]>
Date: Wed, 6 Aug 2008 10:40:59 -0700

> now, I can't think of any.

CRLF or LF could be used as the defined separators for a "lines" range  
count.  This wouldn't be useful for all formats, but could be useful  
for many text and text-based formats or even binary formats that  
happened to use LF as a line separator.

I know MS software uses "rows" counters for XML-formatted data but as  
you say that's dependent on a particular format.  I'm not aware of any  
other implemented range units beyond bytes.

Lisa

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 8 Aug 2008 17:27:07 +1000

I don't hear anybody arguing strongly that we should be closing this  
extensibility point (although I'm of half a mind to do so myself).

If we keep it, it sounds like we want to just tidy things up, not  
establish a registry.

I'm also thinking noting the relationship to caching and the issues  
associated with format-specific range units may be worth discussing  
when we document how to do this type of extension (see charter).

Cheers,

On 07/08/2008, at 3:40 AM, Lisa Dusseault wrote:

>> "bytes" we can think of that is indeed independent of formats?  
>> Right now, I can't think of any.
>
> CRLF or LF could be used as the defined separators for a "lines"  
> range count.  This wouldn't be useful for all formats, but could be  
> useful for many text and text-based formats or even binary formats  
> that happened to use LF as a line separator.
>
> I know MS software uses "rows" counters for XML-formatted data but  
> as you say that's dependent on a particular format.  I'm not aware  
> of any other implemented range units beyond bytes.
>
> Lisa
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Fri, 08 Aug 2008 10:10:42 +0200

> extensibility point (although I'm of half a mind to do so myself).

Correct. We should consider this when a proper fix turns out to be too 
expensive.

+1

BR, Julian

---

From: "Larry Masinter" <[email-masked]>
Date: Fri, 8 Aug 2008 02:48:29 -0700

There is a new W3C working group proposed for working on addressing
and accessing temporal fragments of timed media (video & audio).
One could imagine time ranges that might be independent of the
format.

On the other hand, I don't understand the reasons that the
JPEG 2000 committee rejected using HTTP range retrieval for
the J2 protocol, instead adopting query parameters in the URLs.
In this case, the range retrieval might look like it was
media independent even though J2 defined it specifically for
JPEG 2000 encoded images. 

It might be worth looking into the reasons for the design
choice in J2.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> On Behalf Of Lisa Dusseault
> Sent: Wednesday, August 06, 2008 10:41 AM
> To: Julian Reschke
> Cc: Mark Nottingham; HTTP Working Group
> Subject: Re: issue 85 - range unit extensions
> 
> 
>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 08 Aug 2008 12:15:31 +0200

> JPEG 2000 encoded images. 

The ability to bookmark a fragment?

BR, Julian

---

From: Martin Duerst <[email-masked]>
Date: Fri, 22 Aug 2008 18:02:49 +0900

>It might be worth looking into the reasons for the design

I have no clue, but it could be that one argument for not using
ranges is that you cannot expose them in an URI. This means that
instead of sending your colleague a mail saying "Look at this
interesting snippet: [URI here]" you have to say "Look at this
video [URI here], the interesting stuff is at xx:yy:zz into the
video".

In other words, range, as currently used, is a device for network
optimization (e.g. restart of long downloads that failed in the
middle), without user-related meaning (users don't count bytes!).

Regards,   Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]

---

From: Yves Lafon <[email-masked]>
Date: Wed, 27 Aug 2008 05:44:15 -0400 (EDT)

> video [URI here], the interesting stuff is at xx:yy:zz into the

Well, when fragments are used to identify part of an HTML document, the 
fragment is not sent at the HTTP level, although in the case of HTML the 
whole document is always fetched, but in the case of pictures or video, 
being able to access a part of the image using ranges derived form the URI 
instead of dowloading the whole document before finding the fragment would 
be far more efficient (like accessing XMP fragment inside a picture, 10s 
out of a 1h movie in fullHD).

         ~~Yves

---

From: Martin Duerst <[email-masked]>
Date: Wed, 27 Aug 2008 20:00:08 +0900

>Well, when fragments are used to identify part of an HTML document, the fragment is not sent at the HTTP level, although in the case of HTML the whole document is always fetched, but in the case of pictures or video, being able to access a part of the image using ranges derived form the URI instead of dowloading the whole document before finding the fragment would be far more efficient (like accessing XMP fragment inside a picture, 10s out of a 1h movie in fullHD).

Yves, you are talking about fragment ids, which have the disadvantages
you mention, but Larry says they are using query parts, which are
sent to the server.

Regards,    Martin.

---

From: Yves Lafon <[email-masked]>
Date: Wed, 27 Aug 2008 08:15:32 -0400 (EDT)

On Wed, 27 Aug 2008, Martin Duerst wrote:

>> fragment is not sent at the HTTP level, although in the case of HTML 
>> the whole document is always fetched, but in the case of pictures or 
>> video, being able to access a part of the image using ranges derived 
>> form the URI instead of dowloading the whole document before finding 
>> the fragment would be far more efficient (like accessing XMP fragment 
>> inside a picture, 10s out of a 1h movie in fullHD).
>

Well, to me it is not a disadvantage, as it clearly identifies the 
partial content of the resource you want to access without creating a new 
resource (like what happens using query parameters).
It just puts the logic of querying the fragment in the client (via the use 
of HTTP range) instead of asking the server to do it via query parameters.

Both solutions (query parameters or fragment) allows identification of the 
same binary content. I was just replying to the argument for not using 
ranges that it is not (easily) possible to surface ranges in the URI 
space.

         ~~Yves