Subject: Headers vs Response Code for 2NN Contents Of Related

From: Sandro Hawke <[email-masked]>
Date: Sat, 27 Sep 2014 07:46:25 -0400

We seem to be stalled in discussion of the 2NN Contents Of Related proposal [1].  I'd like to briefly summarize the concerns I've heard and provide summary responses.   Then I'd like to sketch an alternative proposal which might be more widely acceptable.  It has the particular advantage of not needing IETF consensus to go forward, since it just requires new headers, not a new response code.

First, I think these are the main points in the discussion so far:  

Q.  Why do you want a new response code?

We have a community of Linked Data users, passing around RDF over HTTP connections, and this is something they want.  I'm specifically representing the W3C Linked Data Platform (LDP) WG which wants it for paging, to avoid a roundtrip.   Others have said it would be useful for them, including the W3C TAG which says it helps with their httpRange-14 issue.

Q.  Why is one additional roundtrip so important?

If you're getting 1000 pages, then obviously adding one roundtrip is a trivial additional cost, but we expect the common behavior to be just looking at the first page.  When you look at search results, do you go through every page?   No, the response was, "There are a lot of results, here are some", and that's often enough, especially if they're ordered in some useful way.   In this case of just looking at a small first page, saving the roundtrip could nearly double the speed.

Q.  These interactions wont always make full use of caching.

True, not with the current cache architecture, they wont.  We have some ideas for how to make caching work much better for Linked Data applications, but that can be handled separately.

Q.  Why not use Range, with a new type of units?

That's not the way WebApps are usually written.  There are a lot of details that would have to be figured out which are already figured out for next/prev paging, and that's what the developers seem to want. Plus, since the bar for new range units is IETF consensus, we're not optimistic that we'd be able to move forward even if we figured it all out.

Q.  Why not use 200 OK + Content Location?

200+CL is defined as returning a representation of the requested resource, but in next/prev paging, each of those pages is conceptualized as its own resource.  Logically, something can't be the same resource as its own first page.  In practical terms, if a server used 200+CL to return just the first page, how would the client know it had only been sent the first few triples (instead of all of them, as usual)?   The Content-Location header is already used is con-neg so it can't also carry this information.

Q.   Why not use 200+Link?

It's true that in LDP paging there are Link headers that would tell the client it got the first page instead of the whole resource, but then we've still misused 200 OK and are sending a representation of a different resource.   We'd have to carefully avoid caching, and it's unclear what other things might break by having links change the semantics.

This does however lead to my current proposal, which is that we use a new header (really a pair of headers) for this purpose.  The details could be tweaked, but it would be something like this:

In general, the purpose and data flow and caching rules are the same as in the 2NN proposal [1].  The syntax differs as follows:

(1) the client signals that it implements and wants to use this mechanism by sending something like

    Please-Follow-Redirects: 303

instead of

    Prefer: contents-of-related

The values for this new header would be 3xx codes.   Repeated values would mean that multiple kinds of redirects should be followed.

(2) the server which chooses to use this mechanism (and MUST only do this if the client requested it) responds including these headers:

    Redirect-Followed: 1;303 See Other;<http://example.org/other-content>
    Vary: Please-Follow-Redirects

instead of

    2NN Contents of Related
    Location: http://example.org/other-content

The values for this new header would be triples of (redirection sequence number, redirection response code, new location).  Repeated values must have distinct sequence numbers, and mean that multiple redirection steps were taken.  The actual response code, and all the other headers, will be from the final location, the one whose representation is actually returned and whose URI is in the highest-numbered Redirect-Followed result.

Thoughts?   I believe this is on the agenda for the W3C TAG meeting this coming week, so an early response would be helpful.

A simplification could be that this is only for 303, and it only reports the last hop if multiple redirects were followed.   But if we're going to add this, it seems to me it could also be useful for 301 and 307.  Or is it better not to touch that stuff?   Let me know.

      -- Sandro

[1] https://tools.ietf.org/html/draft-prudhommeaux-http-status-2nn-00

---

From: Julian Reschke <[email-masked]>
Date: Sat, 27 Sep 2014 23:38:16 +0200

On 2014-09-27 13:46, Sandro Hawke wrote:
> We seem to be stalled in discussion of the 2NN Contents Of Related
> proposal [1]. I'd like to briefly summarize the concerns I've heard and
> provide summary responses. Then I'd like to sketch an alternative
> proposal which might be more widely acceptable. It has the particular
> advantage of not needing IETF consensus to go forward, since it just
> requires new headers, not a new response code.
>
> We have a community of Linked Data users, passing around RDF over HTTP
> connections, and this is something they want. I'm specifically
> representing the W3C Linked Data Platform (LDP) WG which wants it for
> paging, to avoid a roundtrip. Others have said it would be useful for
> them, including the W3C TAG which says it helps with their httpRange-14
> issue.
>
> If you're getting 1000 pages, then obviously adding one roundtrip is a
> trivial additional cost, but we expect the common behavior to be just
> looking at the first page. When you look at search results, do you go
> through every page? No, the response was, "There are a lot of results,
> here are some", and that's often enough, especially if they're ordered
> in some useful way. In this case of just looking at a small first page,
> saving the roundtrip could nearly double the speed.
>
> True, not with the current cache architecture, they wont. We have some
> ideas for how to make caching work much better for Linked Data
> applications, but that can be handled separately.
>
> That's not the way WebApps are usually written. There are a lot of
> details that would have to be figured out which are already figured out
> for next/prev paging, and that's what the developers seem to want. Plus,
> since the bar for new range units is IETF consensus, we're not
> optimistic that we'd be able to move forward even if we figured it all out.
>
> 200+CL is defined as returning a representation of the requested
> resource, but in next/prev paging, each of those pages is conceptualized
> as its own resource. Logically, something can't be the same resource as
> its own first page. In practical terms, if a server used 200+CL to
> return just the first page, how would the client know it had only been
> sent the first few triples (instead of all of them, as usual)? The
> Content-Location header is already used is con-neg so it can't also
> carry this information.
>
> Q. Why not use 200+Link?
>
> It's true that in LDP paging there are Link headers that would tell the
> client it got the first page instead of the whole resource, but then
> we've still misused 200 OK and are sending a representation of a
> different resource. We'd have to carefully avoid caching, and it's
> unclear what other things might break by having links change the semantics.
>
>
> This does however lead to my current proposal, which is that we use a
> new header (really a pair of headers) for this purpose. The details
> could be tweaked, but it would be something like this:
>
> In general, the purpose and data flow and caching rules are the same as
> in the 2NN proposal [1]. The syntax differs as follows:
>
> The values for this new header would be 3xx codes. Repeated values would
> mean that multiple kinds of redirects should be followed.
 > ...

Why don't you just make it:

   Prefer: retrieve-contents-of-303-target

(with a more suitable name...)

?

Best regards, Julian

---

From: Sandro Hawke <[email-masked]>
Date: Sat, 27 Sep 2014 19:57:14 -0400

> (with a more suitable name...)

Hmmm.   What do you think about the rest of the proposal?   Do you 
prefer this header-based design over the response-code based one? Are 
you okay with both or neither or just this one?

On use of Prefer, is there guidance somewhere?   I can't really tell 
where one would use Prefer over a new header.   If Prefer existed, would 
it have been better to make many of the existing request headers be done 
with prefer?  It's easy to see how the Accept* headers and the If-* 
headers could be thought of as client preferences.     Also Range.   And 
Cache-Control.   Even Authorization and Cookie could kind of be seen as 
client preferences on server behavior.

I guess the difference is that Prefer can be dropped by intermediaries 
and, essentially, can never cause any error condition.   So Accepts that 
have a wildcard entry could be prefers.  And the If-*s could be.   
Authorization could not be, since it can result in an error.

Am I getting that right?   I really don't care -- I don't see any 
advantage to Prefer, but I don't see any disadvantage, either.

       -- Sandro

---

From: Julian Reschke <[email-masked]>
Date: Sun, 28 Sep 2014 09:20:17 +0200

> you okay with both or neither or just this one?

My point being is that you don't need a new header field, just the 
prefer parameter.

> client preferences on server behavior.

That's a good question; my gut feeling is that it depends on how 
frequently it is used.

> advantage to Prefer, but I don't see any disadvantage, either.

The disadvantage of defining a new header field is that you need to 
define a syntax and write parsers, whereas with Prefer the definition 
has been done already, and the parser needs to be written only once.

Best regards, Julian

---

From: Sandro Hawke <[email-masked]>
Date: Sun, 28 Sep 2014 15:01:31 -0400

> My point being is that you don't need a new header field, just the 

Okay....   So the presence of "Preference-Applied: 
retrieve-content-of-303-target" (or whatever it ends up called, maybe 
just "follow-303") in the response headers would have the same semantics 
as the 2NN Contents Of Related response code would have! (And a Location 
header would be added saying where we ended up.) Yes, as far as I can 
see that would work technically.   Nice and simple.   I like it.

I imagine some people will still be uncomfortable with the idea of a 200 
OK response providing a representation of a different resource.   In a 
sense this violates various specs, or at least various people's ideas of 
how things should work.  But since this will only happen when the client 
and the server both agree to it, I'm not seeing what harm it would do.

So I'm happy with this if the TAG and HTTP WG don't think it breaks too 
much.    It's certainly easier than a new response code!

     -- Sandro

---

From: Julian Reschke <[email-masked]>
Date: Sun, 28 Sep 2014 21:14:16 +0200

> see that would work technically.   Nice and simple.   I like it.

It would be "retrieve", not "follow". So you'd still get a 303, but the 
response body would be for the resource Location points to.

Best regards, Julian

---

From: Sandro Hawke <[email-masked]>
Date: Sun, 28 Sep 2014 17:04:12 -0400

On 09/28/2014 03:14 PM, Julian Reschke wrote:
> It would be "retrieve", not "follow". So you'd still get a 303, but 
> the response body would be for the resource Location points to.
>

I think the response code would need to be 200, not 303.   I've been 
told things wont work well with 3xx responses having bodies.

Really I'd suggest the response code should be whatever the target of 
the 303 provides.

Also, all the headers should be what the target of the 303 provides, 
except the Preference-Applied and Vary: Prefer.

To my ear that sounds like the server "following" the 303, but I'm not 
attached to that terminology.  "retrieve" doesn't seem quite right since 
in many cases it'll actually be on the same server. Maybe 
"respond-as-if-from-target-of-303"?   How about "if-303-provide-content"?

        -- Sandro

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 29 Sep 2014 09:13:55 +1000

On 29 September 2014 07:04, Sandro Hawke <[email-masked]> wrote:

> I think the response code would need to be 200, not 303.   I've been told
> things wont work well with 3xx responses having bodies.
> ​
> ​
>
>
​If it's 200 you have to be careful to set the cache control headers etc.
so that intermediate caches don't screw things up. At least with a 303 the
default/naive behaviour is closer to what you want anyway. (I.e. in the
worst case, the right thing happens, albeit slower)​

> ​
> ​

​
> Really I'd suggest the response code should be whatever the target of the
> 303 provides.
> ​
> ​
>
>
​If it would have been anything other than 200, the server shouldn't send
it. Just send the original 303. Maybe add a warning or something ("If you
get that it might fail").

> ​
> ​
> ​
​​
> ​
> ​
​That's why it's risky to send a 200. If you send a 303 it's clear (to both
endpoints, and to any intermediaries) that the metadata applies to the
*requested* resource. If you ask me, "follow redirect" is a dangerous
metaphor, and it might be better to instead use an encapsulation metaphor.
In the current world, the response to "GET /foo" is "You should see
/foo/bar instead" -- everyone understands that, things happen. I suggest
the response to "GET /foo, but also send me the eventual response if
there's a redirect" should be "You should see /foo/bar instead, which is:
{...}". Uninformed brainstorm thought: send a 303 with
"content-type:message/http"; might need a bit more machinery in the client,
not sure how much though.

A dumb cache that doesn't understand the "which is..." part either: 1)
passes everything through as-is, and everyone is happy; or 2) strips the
unknown bits and you're left with a 303, and everyone gets to the right
place eventually.

> ​
> ​
> ​
​​
> attached to that terminology.  "retrieve" doesn't seem quite right since in
> many cases it'll actually be on the same server. Maybe

If you want advice on colouring, I quite like the shade of "indirect".
However if you switch to an encapsulation metaphor, something along the
lines of "Prefer: include-other" isn't too bad. (I keep coming back to the
word "other" because 303 is "See Other", and that "other" is what we're
including.)

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Sandro Hawke <[email-masked]>
Date: Sun, 28 Sep 2014 23:54:44 -0400

On 09/28/2014 07:13 PM, Matthew Kerwin wrote:
> On 29 September 2014 07:04, Sandro Hawke <[email-masked] 
> <mailto:[email-masked]>>wrote:
>
>         It would be "retrieve", not "follow". So you'd still get a
>         303, but the response body would be for the resource Location
>         points to.
>
>
>     I think the response code would need to be 200, not 303.  I've
>     been told things wont work well with 3xx responses having bodies.
>     ​
>     ​
>
>
> ​If it's 200 you have to be careful to set the cache control headers 
> etc. so that intermediate caches don't screw things up.

It sounds like you don't trust the Vary: Prefer to do its job. Are you 
just being cautious, or is there reason to think Vary doesn't actually 
work (or perhaps that I'm misunderstanding what it does).

> At least with a 303 the default/naive behaviour is closer to what you 
> want anyway. (I.e. in the worst case, the right thing happens, albeit 
> slower)​
>

Except if intermediate system wont even pass any body through with a 3xx 
response code.   That's what I was told, but I haven't done any kind of 
survey.

>     ​
>     ​
>
>     ​
>     Really I'd suggest the response code should be whatever the target
>     of the 303 provides.
>     ​
>     ​
>
>
> ​If it would have been anything other than 200, the server shouldn't 
> send it. Just send the original 303. Maybe add a warning or something 
> ("If you get that it might fail").
>
>     ​
>     ​
>     ​
>
>     ​ ​
>     Also, all the headers should be what the target of the 303
>     provides, except the Preference-Applied and Vary: Prefer.
>     ​
>     ​
>
>
> ​ That's why it's risky to send a 200. If you send a 303 it's clear 
> (to both endpoints, and to any intermediaries) that the metadata 
> applies to the *requested* resource. If you ask me, "follow redirect" 
> is a dangerous metaphor, and it might be better to instead use an 
> encapsulation metaphor. In the current world, the response to "GET 
> /foo" is "You should see /foo/bar instead" -- everyone understands 
> that, things happen. I suggest the response to "GET /foo, but also 
> send me the eventual response if there's a redirect" should be "You 
> should see /foo/bar instead, which is: {...}". Uninformed brainstorm 
> thought: send a 303 with "content-type:message/http"; might need a bit 
> more machinery in the client, not sure how much though.
>
> passes everything through as-is, and everyone is happy; or 2) strips 
> the unknown bits and you're left with a 303, and everyone gets to the 
> right place eventually.
>

Yes, there's a lot to be said for this design (sending 303 and a body), 
if it would work.    I only have it second hand that it doesn't work, so 
I don't even know the original source of my claim that it doesn't.

>     ​
>     ​
>     ​
>
>     ​ ​
>     To my ear that sounds like the server "following" the 303, but I'm
>     not attached to that terminology.  "retrieve" doesn't seem quite
>     right since in many cases it'll actually be on the same server.
>     Maybe "respond-as-if-from-target-of-303"?   How about
>     "if-303-provide-content"?
>
>
> However if you switch to an encapsulation metaphor, something along 
> the lines of "Prefer: include-other" isn't too bad. (I keep coming 
> back to the word "other" because 303 is "See Other", and that "other" 
> is what we're including.)

Understood, thanks.

        -- Sandro

---

From: Matthew Kerwin <[email-masked]>
Date: Mon, 29 Sep 2014 14:16:31 +1000

On 29 September 2014 13:54, Sandro Hawke <[email-masked]> wrote:

> just being cautious, or is there reason to think Vary doesn't actually work
> (or perhaps that I'm misunderstanding what it does).
> ​
> ​
>

I'm not entirely trusting, no. It might be paranoia, but it might also come
from random interactions with HTTP/1.0 proxies in the wild. I still send
Pragma headers, too. :\

<snip>

> Yes, there's a lot to be said for this design (sending 303 and a body), if
> it would work.    I only have it second hand that it doesn't work, so I
> don't even know the original source of my claim that it doesn't.
>
>
That's definitely the crux, then. Whichever failure mode is more likely
(poisoning caches in spite of Vary, vs. stripping body of 303) is the one
that should be handled by default. I'd be doing a survey here, and some
field tests, to inform the decision.

How about adding a(n optional?) response status code parameter to the
preference: Prefer:get-other=200 vs Prefer:get-other=303, with the default
(if any) going to whichever case you think is more likely. That way you can
get around/through known-bad proxies if you want. Also: sorry for just
thinking of yet another name for the preference, I can't remember what the
current front-runner is, to use that.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 29 Sep 2014 07:31:59 +0200

> told things wont work well with 3xx responses having bodies.

Can we please base this on more than hearsay?

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 29 Sep 2014 19:23:07 +1300

On 29/09/2014 5:16 p.m., Matthew Kerwin wrote:
> On 29 September 2014 13:54, Sandro Hawke wrote:
> 
>> ​If it's 200 you have to be careful to set the cache control
>> headers etc. so that intermediate caches don't screw things up.
>> 
>> 
>> It sounds like you don't trust the Vary: Prefer to do its job.
>> Are you just being cautious, or is there reason to think Vary
>> doesn't actually work (or perhaps that I'm misunderstanding what
>> it does). ​ ​
>> 
> 
> I'm not entirely trusting, no. It might be paranoia, but it might
> also come from random interactions with HTTP/1.0 proxies in the
> wild. I still send Pragma headers, too. :\
> 
> <snip>
> 
> 
>> Yes, there's a lot to be said for this design (sending 303 and a
>> body), if it would work.    I only have it second hand that it
>> doesn't work, so I don't even know the original source of my
>> claim that it doesn't.
>> 
>> 
> That's definitely the crux, then. Whichever failure mode is more
> likely (poisoning caches in spite of Vary, vs. stripping body of
> 303) is the one that should be handled by default. I'd be doing a
> survey here, and some field tests, to inform the decision.
> 

303 status is and always has been clearly defined as having a payload.
It is also a common redirect used in captive portals for delivery with
session initiation pages. Any middleware stripping those is breaking
HTTP and well deserving of bug reports.

Amos

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 29 Sep 2014 12:03:07 +0200

> We have a community of Linked Data users, passing around RDF over HTTP connections, and this is something they want. I'm specifically representing the W3C Linked Data Platform (LDP) WG which wants…

Generally speaking, when an application-specific group of developers say
that they need a change made to HTTP in order to support their needs,
it is almost always because they are making incorrect assumptions about
HTTP.  Avoiding such changes is what keeps HTTP (relatively) simple.

> If you're getting 1000 pages, then obviously adding one roundtrip is a trivial additional cost, but we expect the common behavior to be just looking at the first page. When you look at search resul…

Yes, which is why you get paged results on search queries.  This does not
require any changes to HTTP.

> True, not with the current cache architecture, they wont. We have some ideas for how to make caching work much better for Linked Data applications, but that can be handled separately.

But not as well as the existing solutions.

> That's not the way WebApps are usually written. There are a lot of details that would have to be figured out which are already figured out for next/prev paging, and that's what the developers seem…

As opposed to the suggested changes, which also require IETF consensus?

I would not use Range because that works in the opposite direction:
it is the client expressing a desired limitation on the server's results.
What you have described is a server deciding to limit the client's results.

> 200+CL is defined as returning a representation of the requested resource, but in next/prev paging, each of those pages is conceptualized as its own resource.

That isn't a relevant concern.  The requested resource is paged.  So what?
You are confusing what you are requesting with the data stored on the back-end.
If you get a paged representation, then that is what you requested.
The origin server is responsible for determining what the URI means.
It is necessarily a different resource in the sense that it presents a
different view of the same data.  Why is that a problem for LD when it
isn't a problem for any other part of the Web?

The abstract interface is the essential difference between HTTP's
architecture of providing a representation of the data (instead of a
client-imposed assumption about the nature of the resource) and most
other systems reliance on shared data assumptions (like FTP's focus
on files and RPC's focus on procedures).

> Logically, something can't be the same resource as its own first page.

Logically, the client doesn't know what the resource is just because it
has a URI.  It doesn't know whether it will get a paged result or not.
The server is deciding that for the client, usually based on the amount
of data or the time required to produce results.  Regardless, the nature
of that representation is explained IN THE REPRESENTATION, either within
the data format or within the associated metadata.

> In practical terms, if a server used 200+CL to return just the first page, how would the client know it had only been sent the first few triples (instead of all of them, as usual)?

Because that's what the data says.

What tells the client that it has received only a page of the potential
result set is the metadata sent within (or alongside) that data.
That's what prev/next links are in HTML.  Formal relationships can be
defined for any data set.  These can be placed in the header fields,
in the body content (if the data format defines a place for them), or
even within a parameter on the media type.

The only real distinction between a paged 200 response and any other
resource is the presumption that there exists, somewhere, some other
resource that can represent the complete data set in one go.  However,
the only reason to provide a link to that other resource is to support
direct authoring (e.g., PUT).  In all other cases, access to the
"complete" resource tends to be limited for scalability reasons.

> The Content-Location header is already used is con-neg so it can't also carry this information.

No, it wouldn't, though it does provide a link to the source of *this*
representation.  Paging is just another form of content negotiation.

> It's true that in LDP paging there are Link headers that would tell the client it got the first page instead of the whole resource, but then we've still misused 200 OK and are sending a representat…

Seriously confused.  You are making an assumption about the resource.
That assumption is false.  So, don't make that assumption.  Instead,
look at the data returned and it will tell you if it is a paged result.

Perhaps you might want to define some formal link relations, or add
metadata to some data formats, but your application does not require
any changes to HTTP.

....Roy