Subject: Re: issue 85 - range unit extensions

From: Jamie Lokier <[email-masked]>
Date: Mon, 1 Sep 2008 10:25:35 +0100

Julian Reschke wrote:
> >interoperability with servers by following the HTTP specification as 
> >closely as possible, so servers have a real standard to go off of 
> >instead something we made up. It seems like leveraging the range/partial 
> >content mechanism with alternate range unit is the approach that HTTP 
> >would suggest, and I have no reason to believe it is wrong. Retrieving a 
> >paged subset of data is merely a different representation of the same 
> >resource.
> 
> Yes. But, making up new range units shares has similar problems as 
> making up query parameters, doesn't it?
> 
> To make this robust, we'd really need a registry.

Yes, a registry or a convention for range-unit namespaces, like
"com.dojo.items", or even "items{uuid=UUID}".

Or simply declare that "items" is application-specific.  Only use it
with known resources, and caches should not do anything clever with
it.

Or use a different header: Dojo-Range, with the response containing
"Vary: Dojo-Range".  That would make it cachable by generic HTTP
caches, which sounds rather desirable.

-- Jamie

---

From: Yves Lafon <[email-masked]>
Date: Mon, 1 Sep 2008 08:23:05 -0400 (EDT)

> "com.dojo.items", or even "items{uuid=UUID}".

It should contain "universal" units (bytes, seconds, characters), as well 
as custom one, how about an URI to identify the unit, as well?

> caches, which sounds rather desirable.

The "different header" will not explain what the meaning of the range is, 
unless it is a standardized header meant to describe in general 
range units.

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 1 Sep 2008 15:12:03 +0100

> unless it is a standardized header meant to describe in general 

Range "items" doesn't explain what the meaning is either.  It just
gives a really vague idea.  What kind of items?

"Dojo-Range: 6 items" seems clearer.

Are caches allowed (in principle) to split/merge these ranges, if they
knew how?  Would such caches operate on the item syntax -
i.e. structured JSON in the message body?

-- Jamie

---

From: "Kris Zyp" <[email-masked]>
Date: Mon, 1 Sep 2008 08:22:45 -0600

----- Original Message ----- 
From: "Julian Reschke" <[email-masked]>
To: "Kris Zyp" <[email-masked]>
Cc: <[email-masked]>
Sent: Monday, September 01, 2008 2:22 AM
Subject: Re: issue 85 - range unit extensions

> Kris Zyp wrote:
>>> Out of curiosity: did you consider simply using a query parameter 
>>> instead? If you did, and decided not to, that would be interesting for 
>>> the question whether custom ranges are a good idea after all.
>>
>> Yes, I considered that, but here is no standard for query parameters, 
>> AFAIK. Since Dojo is a client side library, we are aiming for maximum
>
> Nope.
>
>> closely as possible, so servers have a real standard to go off of instead 
>> something we made up. It seems like leveraging the range/partial content 
>> mechanism with alternate range unit is the approach that HTTP would 
>> suggest, and I have no reason to believe it is wrong. Retrieving a paged 
>> subset of data is merely a different representation of the same resource.
>
> Yes. But, making up new range units shares has similar problems as making 
> up query parameters, doesn't it?

True, but it seems much more in line with specifications, and requires much 
less "invention" than query parameters (specifying start, count, as well 
response mechanisms for returned count, index, and total count).

> To make this robust, we'd really need a registry.

That's fine, sounds good to me.
Kris

---

From: "Kris Zyp" <[email-masked]>
Date: Mon, 1 Sep 2008 08:25:00 -0600

> with known resources, and caches should not do anything clever with

It certainly is not intended to be application specific, it is intended to 
be very general purpose, I wouldn't want it tied to Dojo, if possible.

> gives a really vague idea.  What kind of items?

At least in the context of JSON, items can be reasonably clear. I would 
think other structured data formats may also have entity that could be 
well-defined.
> "Dojo-Range: 6 items" seems clearer.

This is kind of the antithesis of what we were aiming for. We wanted 
something interoperable with other technologies, not Dojo-specific.

> i.e. structured JSON in the message body?

Yes and yes, I thought that was the idea of Ranges.

Kris

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 1 Sep 2008 15:38:10 +0100

> think other structured data formats may also have entity that could be 

Slight unclarity, sorry.  By application-specific, I meant specific to
"a particular structured form of JSON".

> something interoperable with other technologies, not Dojo-specific.

Fair enough.  JSON-as-Dojo-uses-Range? :-)

> Yes and yes, I thought that was the idea of Ranges.

Sounds good.

Right now, cache range operations are independent of content type.

For the proposal, the name "items" seems too generic for something
which only works with JSON (perhaps even only JSON structured in a
particular way?).

Unless it's understood that cache range operations on "items" must
take into account other headers such as Content-Type.  I think that
would be reasonable.

(Btw, one can easily imagine ranges "characters", "xpath", "xquery",
"grep"... even "session".  I'm not sure if that's a desirable road to
go down.  Maybe it would be very useful.)

-- Jamie

---

From: Yves Lafon <[email-masked]>
Date: Mon, 1 Sep 2008 11:01:47 -0400 (EDT)

> Right now, cache range operations are independent of content type.
Well, just because 'bytes' a pretty much independant of CT :)

> go down.  Maybe it would be very useful.)

I read ranges as "a part, possibly the whole part (0-) of a document"
in the case of characters for text-based documents, it's ok, but grep, 
xpath won't be able to access the whole document using the "0-" syntax.

But maybe even the way ranges are defined may change (after all it's 
defined as "byte-range-resp-spec" now), still I find this perverting the 
meaning of ranges and go in the transformation domain.

         ~~Yves

---

From: "Kris Zyp" <[email-masked]>
Date: Mon, 1 Sep 2008 09:51:49 -0600

> take into account other headers such as Content-Type.  I think that

I certainly don't mind using a different name, if the HTTP working feels 
that's more appropriate. I just figured "items" might be applicable in 
multiple content types (applicable to content types that support a top-level 
array-like construct), but I understand that if it is felt to be too generic 
to have a real clear purpose in every situation. Maybe "array-items"?

> go down.  Maybe it would be very useful.)

"date" ranges is the one other unit that I have felt would be useful, but 
"items" is certainly the most valuable alternate IMO.

Kris

---

From: "Robert Brewer" <[email-masked]>
Date: Mon, 1 Sep 2008 11:13:54 -0700

Kris Zyp wrote:
> I certainly don't mind using a different name, if the HTTP working
> feels that's more appropriate. I just figured "items" might be
> applicable in multiple content types (applicable to content types
> that support a top-level array-like construct), but I understand
> that if it is felt to be too generic to have a real clear purpose
> in every situation. Maybe "array-items"?

These are all great points and I'd personally like to see items-ranges
become commonplace. However, such a beast *is* quite different from
bytes-ranges since the individual ranges must be joined and enveloped.
When a cache, for example, has the bytes-ranges:

    aaa bbb ccc

...and wishes to combine them into a single response, it joins them with
nothing ('') and envelopes them with nothing ('') to return 'aaabbbccc'.
But a cache holding valid JSON items-ranges:

   "aaa" "bbb" "ccc"

...that wished to return them combined in a single response would have
to join them with a comma and envelop them in brackets to be valid JSON:
'["aaa", "bbb", "ccc"]'.

There may be other media types that follow those same rules, but off the
cuff it seems pretty media-type-specific (unless we abandon the hope
that caches could do that recombination). It's certainly *not*
vendor-specific--there's no need to call them 'Dojo-items' or
'com.dojo.items'. One possibility would be to leave the range
declaration media-type-neutral ('items') and to add range-recombination
rules to the media-type itself, so that RFC 4627 (application/json)
would eventually grow a paragraph on its rules for items-ranges.

> "date" ranges is the one other unit that I have felt would be useful,

Which dates did you have in mind?

> but "items" is certainly the most valuable alternate IMO.

Agreed. Collections are everywhere.

Robert Brewer
[email-masked]

---

From: "Kris Zyp" <[email-masked]>
Date: Mon, 1 Sep 2008 13:41:29 -0600

> would eventually grow a paragraph on its rules for items-ranges.

Yes, that is what I would hope for.

> Which dates did you have in mind?

Well, it had occurred to me that it might be useful for a client to be able 
to retrieve a set of all new (or modified) resources from a collection by:

GET /MyCollection/ HTTP/1.1
Range: date=Mon, 01 Sep 2008 19:33:44 GMT-

This would retrieve all the entities in the collection since the provided 
date. For a frequently updated collection, it would seem more efficient to 
be able to get a list of only the items/resources that are new or updated 
since the last time it was retrieved, rather than using an If-Modified-Since 
and downloading the whole collection if something has changed.

Anyway, I didn't intend to distract, I haven't put all that much thought 
into this approach, it may be too high level for HTTP or simply wrong. 
Indexed based ranges is certainly my main desire.

Thanks,
Kris

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 1 Sep 2008 18:25:51 -0700

> in every situation. Maybe "array-items"?

Range "items" is fine.  It is not specific to the media type.  Each  
resource
can have its own notion of what it means to be an item boundary --  
all that
matters is that they be sequentially numbered from a standard start  
value
(0 or 1) and that the items combined in order are consistent with  
what would
have been received in a 200 response to GET.

> "date" ranges is the one other unit that I have felt would be  
> useful, but "items" is certainly the most valuable alternate IMO.

Date ranges are much harder to get right because the boundary is not  
clear.
The typical use for such a range is to retrieve whatever has been  
appended
to the resource state since a given time, but time units are not unique
and thus require some overlap in requests and removal of duplicates.

A Range unit of Since="strong-etag" would be interesting as well.

....Roy

---

From: "Kris Zyp" <[email-masked]>
Date: Mon, 1 Sep 2008 21:34:56 -0600

>> multiple content types (applicable to content types  that support a 
>> top-level array-like construct), but I understand  that if it is felt to 
>> be too generic to have a real clear purpose  in every situation. Maybe 
>> "array-items"?
>
> can have its own notion of what it means to be an item boundary --  all 
> that
> matters is that they be sequentially numbered from a standard start  value
> (0 or 1) and that the items combined in order are consistent with  what 
> would

Yes, agreed. Start value of 0 seems most logical and consistent with byte 
range units.

Kris

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 2 Sep 2008 07:56:17 +0100

> to join them with a comma and envelop them in brackets to be valid JSON:

Not just merging rules, but also splitting rules.  A cache which
already has items 0-100 and a strong Etag, and is asked for 50-200,
could convert the request to 50-200 with a conditional request.

For this, the cache needs to know how to merge and how to split the
entity syntax.

> would eventually grow a paragraph on its rules for items-ranges.

My concern was the combination of media-type
"application/x-javascript" (or even "text/plain") and the "items"
range is not specific enough to apply the JSON combination rules.

That's what I was getting at by dojo-items.

If it's only used with the "application/json" media-type, and it can
define that "items" always refers to _array_ items (i.e. numbered) and
the JSON _top-level_ object is an array, then I have no such concern.

I'm a bit surprised that the top-level object in a JSON request would
be an array, though.  For round-trip minimisation in AJAX applications
isn't it usual to send a bit of auxiliary metadata, or a few objects
together, and therefore the top-level JSON object tends to be an
object (i.e. several named data items) with one of its members being
an array, rather than the top-level object being an array itself?

This was also what I was getting at by dojo-items, since I imagined
the merging/splitting rules would tend to apply to a sub-object rather
than the top-level object.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 02 Sep 2008 14:02:15 +0200

> Yes and yes, I thought that was the idea of Ranges.

Can it do so without parsing the body? I.e. by simple concatenation of
the items?

If it can then range units is appropriate.

If the bodies of the parts involved needs to be parsed and reformatted
in some way to merge them in a larger response then range units is not
appropriate as the response entities then really is separate entities on
their own and not just parts of a larger entity.

Regards
Henrik

---

From: "Kris Zyp" <[email-masked]>
Date: Tue, 2 Sep 2008 08:35:03 -0600

> the JSON _top-level_ object is an array, then I have no such concern.

I agree, it should only be applicable when the top-level entity is an array.

> an array, rather than the top-level object being an array itself?

I am trying to move away from that approach, moving metadata to headers 
(ironically the offset and total count are the most common metadata items, 
and these are handled by the Content-Range header), allowing the actual 
content to be a "pure" representation of the resource, and therefore an 
array is the most natural top-level construct when requesting a collection 
of objects.

Kris

---

From: "Kris Zyp" <[email-masked]>
Date: Tue, 2 Sep 2008 08:37:56 -0600

> Can it do so without parsing the body? I.e. by simple concatenation of

No, it requires some level of JSON parsing.

> their own and not just parts of a larger entity.

I didn't know there was such rules for alternate range units.

Kris

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 02 Sep 2008 21:44:01 +0200

> I didn't know there was such rules for alternate range units.

There isn't any struct rules for range units, but this property is
highly desireable for different implementations to interoperate properly
when it comes to merging and splitting.

Regards
Henrik

---

From: "Kris Zyp" <[email-masked]>
Date: Tue, 2 Sep 2008 13:49:54 -0600

> when it comes to merging and splitting.

After thinking about this, I actually have no problem with this rule being 
enforced, I think that is a good suggestion. I had been thinking it would be 
more desirable that the individual "ranged" response representations all be 
valid complete JSON messages themselves, but I don't think that is 
necessary. I am perfectly fine with have ranges all being concatenable 
without any JSON parsing.

GET /jsonResource
Range: items=0-1

response:
["a","b",

GET /jsonResource
Range: items=2-

response:
"c","d"]

GET /jsonResource

response:
["a","b","c","d"]

A puts little extra overhead on the client doing JSON parsing to properly 
pre/postpend the string to parse it, but this is a negligible cost...

Thanks,
Kris

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Sep 2008 00:22:56 +0100

Kris Zyp wrote:
> enforced, I think that is a good suggestion. I had been thinking it would 
> be more desirable that the individual "ranged" response representations all 
> be valid complete JSON messages themselves, but I don't think that is 

What about

Range: items=2-4
Range: items=2-5

Is the JSON parser expected to look at the text it receives to decide
whether to append "]"?

If so, why bother sending the "[" and "]" at all?

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Sep 2008 00:24:03 +0100

> array is the most natural top-level construct when requesting a collection 

Unfortunately, putting metadata into headers isn't a very tidy way of
representing metadata which has structure itself, which is not
uncommon with JSON.

-- Jamie

---

From: "Kris Zyp" <[email-masked]>
Date: Tue, 2 Sep 2008 17:43:52 -0600

----- Original Message ----- 
From: "Jamie Lokier" <[email-masked]>
To: "Kris Zyp" <[email-masked]>
Cc: "Henrik Nordstrom" <[email-masked]>; "ietf-http-wg" 
<[email-masked]>
Sent: Tuesday, September 02, 2008 5:22 PM
Subject: Re: issue 85 - range unit extensions

>> pre/postpend the string to parse it, but this is a negligible cost...

In my example there were four items in the array, so your Range requests 
(2-4 and 2-5) should result in 416 Requested Range Not Satisfiable.

> Is the JSON parser expected to look at the text it receives to decide

The JSON parser could look at the text or the Content-Range header (to see 
if it received the last item). I am not sure which would be more reliable.

> If so, why bother sending the "[" and "]" at all?

The cumulative concatenated representation must be valid JSON.

Kris

---

From: "Kris Zyp" <[email-masked]>
Date: Tue, 2 Sep 2008 17:55:43 -0600

> representing metadata which has structure itself, which is not

Not wishing to infer assent, but could we presume that if an alternate JSON 
structure was defined that had a top level object with prescribed property 
containing the array/items, that it could have application content type 
which provided the definition of how it should be segmented in response to 
an Range: items request? I think it is assumed that creating subset 
representations via a items range is content type specific (it would 
certainly be different for XML, for example), so it seems reasonable that 
application sub-types could also provide a definition of item range unit 
handling.
(Concatenation can still remain content-type agnostic)

Kris

---

From: "Robert Brewer" <[email-masked]>
Date: Wed, 3 Sep 2008 13:47:31 -0700

Kris Zyp wrote:
> > define that "items" always refers to _array_ items (i.e. numbered)
> > and the JSON _top-level_ object is an array, then I have no such
> > concern.
> 

Except...there are a number of people who close a set of XSS attacks by
mandating their JSON implementations never return a top-level array,
only an object.

Cf
http://www.kid666.com/blog/2006/12/23/security-ajax-json-satisfaction/

Robert Brewer
[email-masked]

---

From: "Kris Zyp" <[email-masked]>
Date: Wed, 3 Sep 2008 15:07:50 -0600

Avoiding top-level JSON arrays is pretty hackish way of protecting against 
exploits, there are much better forms of security. It doesn't seem like this 
practice should influence range units. Once again, I would think that those 
that really want a top-level object, for security or for metadata reasons 
could create a sub-format/content type that defined the top level object, 
the collection property, and the proper behavior for items range units with 
that format.
Kris