Subject: Proposed HTTP SEARCH method update

From: James M Snell <[email-masked]>
Date: Fri, 10 Apr 2015 09:00:52 -0700

Please see: http://datatracker.ietf.org/doc/draft-snell-search-method/

Comments welcome.

- James

---

From: Michael Sweet <[email-masked]>
Date: Fri, 10 Apr 2015 13:49:15 -0400

James,

> On Apr 10, 2015, at 12:00 PM, James M Snell <[email-masked]> wrote:
> 

Seems like all this is currently defining is a generic extension to the WebDAV SEARCH method without actually defining an extension that can be used.  Seems like it would be more useful to define a basic plain-text search format that could be widely used for typical "site search" functionality?

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: James M Snell <[email-masked]>
Date: Fri, 10 Apr 2015 10:52:50 -0700

It's not so much an extension to WebDAV SEARCH as a complete
redefinition of the SEARCH method. It can still be used in WebDAV but
could be used in places WebDAV isn't.

---

From: Yorkie Liu <[email-masked]>
Date: Sat, 11 Apr 2015 02:06:42 +0800

> Seems like it would be more useful to define a basic plain-text search
format that could be widely used for typical "site search" functionality?

Agreed on Michael's this point

Plus, the `limit`, `sort` and etc. are the common props that we are used at
most of query languages(SQL/noSQL), maybe we can define them in request
headers? and it would could optimize and simplify the query statement for
common use.

And from this spec, perhaps needs more examples or specs for detailing the
"conditional SEARCH" and related stuffs. But James, thanks for your work,
most of things are clear enough and learned much from your work :)

On Sat, Apr 11, 2015 at 1:52 AM, James M Snell <[email-masked]> wrote:

> > Seems like all this is currently defining is a generic extension to the
> WebDAV SEARCH method without actually defining an extension that can be
> used.  Seems like it would be more useful to define a basic plain-text
> search format that could be widely used for typical "site search"
> functionality?
> >

-- 
Yorkie Liu
https://github.com/yorkie
+18600219033

---

From: James M Snell <[email-masked]>
Date: Fri, 10 Apr 2015 12:55:32 -0700

To be clear, HTTP SEARCH is currently defined as being restricted to
WebDAV use cases, this spec updates the definition so that it's no
longer specific to WebDAV allowing SEARCH to be used in applications
that have nothing to do with WebDAV at all. One could, for instance,
send a SEARCH request carrying a Sparql Query
(http://www.w3.org/TR/rdf-sparql-query/).

---

From: "[email-masked]" <[email-masked]>
Date: Fri, 17 Apr 2015 10:37:13 +0200

> On 10 Apr 2015, at 18:00, James M Snell <[email-masked]> wrote:
> 

[[
  The payload returned in response to a SEARCH cannot be
  assumed to be a representation of the resource identified by the
  effective request URI.
]]

I think that should rather be 

[[
  The payload returned in response to a SEARCH is a partial
  representation of the resource identified by the effective 
  request URI.
]]

The way I think one has to understand SEARCH is in the context of the
other the two other HTTP verbs:

- GET: returns a representation of the effective request URI
- PUT: changes the representation of the effective request URI to the new representation

then SEARCH is like PATCH just a way of efficiently doing GET and PUT

- SEARCH: allows on to get a part of the full representation of the request URI ( using whatever is the 
 most appropriate query language )
- PATCH: allows one to replace part of the representation of the request URI 

It may be that by making this clear in the document it becomes easier for reviewers
to understand that of course this does not require a representation to be specified, the
same way that the GET verb does not require one.

Also I am not sure what in 4.2 you mean by a ""SPAQRL query.

Henry

Social Web Architect
http://bblfish.net/

---

From: Julian Reschke <[email-masked]>
Date: Fri, 17 Apr 2015 11:09:50 +0200

>    representation of the resource identified by the effective

I don't think that's the intention, nor would it be consistent with the 
current use of SEARCH.

>   most appropriate query language )

Depending on the query language, the result might just be a list of 
URIs. I don't see how to interpret this as partial representation...

> Also I am not sure what in 4.2 you mean by a ""SPAQRL query.

Best regards, Julian

---

From: "[email-masked]" <[email-masked]>
Date: Fri, 17 Apr 2015 12:13:42 +0200

> On 17 Apr 2015, at 11:09, Julian Reschke <[email-masked]> wrote:
> 
> I don't think that's the intention, nor would it be consistent with the current use of SEARCH.
> 
> Depending on the query language, the result might just be a list of URIs. I don't see how to interpret this as partial representation...

Ah yes, I had forgoten that case. 

In that case can one not think of the query combined with the answer as forming a partial 
representation of the resource?

In your hypotehtical SPAQRL example you request parts of the remote table. If the client 
made a number of such requests, say asking for each column individually, it could in the 
end re-constitute the whole table. ( assuming the etag had not changed )

It would be interesting if the client could make such deductions from a number of 
queries.

Social Web Architect
http://bblfish.net/

---

From: ashok malhotra <[email-masked]>
Date: Sat, 18 Apr 2015 17:01:31 -0400

Also. Henry, the response may be the location of the
result set as in Example 4.2
-- 
All the best, Ashok

---

From: Mark Baker <[email-masked]>
Date: Sun, 19 Apr 2015 23:09:22 -0400

> Please see: http://datatracker.ietf.org/doc/draft-snell-search-method/

I don't have any substantive comments on the heart of the spec, but
take issue with the characterization of some of the motivations for
the use of SEARCH;

     "Without specific knowledge of the resource and server to which the
      GET request is being sent, there is no way for the client to know
      that a search operation is being requested.  Identical requests
      sent to two different servers can implement entirely different
      semantics."

It is architecturally beneficial that a client has no knowledge of
whether a search takes place or not when a server responds to a GET,
as that is an implementation detail. If that information needs to be
exposed, then a new method needs to be used, as this draft specifies,
but that is in no way a disadvantage of using GET. It is actually a
reason why someone might want to avoid using SEARCH, or other
get-me-this-info-not-using-GET methods, e.g. PROPFIND).

     "Encoding query parameters directly into the request URI
      effectively casts every possible combination of query inputs as
      distinct resources.  For instance, because mechanisms such as HTTP
      caching handle request URIs as opaque character sequences, queries
      such as 'http://example.org/?q=foo' and
      'http://example.org/?q=Foo' will be treated as entirely separate
      resources even if they yield identical results."

The first sentence describes, AFAICT, problems with parameter
ordering, but that is not an interpretation of the meaning of a
parametrized URL that is licensed by either specification or deployed
implementation. It concludes with an example of case-sensitivity, but
I fail to see either the connection to parameter ordering, or any
problem with this situation as we know that the http URI scheme is
case sensitive past the authority component. Those URIs identify
different resources and could represent two different "searches" for
differently-capitalized words.

I'd also point out that SEARCH response are specified as
non-cacheable, so it hardly seems appropriate to describe a perceived
inefficiency in GET response caching as inferior to a complete
inability to cache.

On the flip side, I was surprised to not see the value of avoiding
overloading POST's meaning as an advantage, as I consider that far and
away SEARCH's biggest selling point.

Mark.

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 21 Apr 2015 22:33:23 +0200

So the context of this answer is in short the sentence 
"The response to a SEARCH request is not cacheable."

> On 18 Apr 2015, at 23:01, ashok malhotra <[email-masked]> wrote:
> 

True. But that does not really affect the cachability of the response does it? ( I am 
trying to wrap my head around this, so please consider these questions just a dialectical
method - in the Socratic sense - to see if we can understand this )

The 303 points to another document that itself can be cacheable. I suppose that would
in fact be the point of the 303 even, to give a URL for the specific query that can then
be re-used and sent around - and cached even.

So the 303 in SEARCH does not constitute evidence that SEARCH is not cacheable, just like a 303 in HTTP GET does not show that GET is not cacheable.

So now I wonder why it is that you think SEARCH results cannot be cached?
They may not be cacheable currently in existing caches, but I fail to see why with proper use
of etags, they should not be.

Henry

Social Web Architect
http://bblfish.net/

---

From: James M Snell <[email-masked]>
Date: Tue, 21 Apr 2015 14:08:00 -0700

If the SEARCH response has a Content-Location header and that was used for
the cache, then yes, the response would be cashable. Otherwise, you cannot
assume cacheability because the SEARCH response (a) does not represent the
state of the resource identified by the effective request URI and (b)
caches are currently unable to vary based on request payload.
On Apr 21, 2015 1:38 PM, "[email-masked]" <[email-masked]>
wrote:

> So the 303 in SEARCH does not constitute evidence that SEARCH is not
> cacheable, just like a 303 in HTTP GET does not show that GET is not
> cacheable.
>

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 21 Apr 2015 16:23:19 -0700

> On 21 Apr 2015, at 2:08 pm, James M Snell <[email-masked]> wrote:
> 
> If the SEARCH response has a Content-Location header and that was used for the cache, then yes, the response would be cashable.

ONLY if the C-L is the same as the effective request URI.

Cheers,

> Otherwise, you cannot assume cacheability because the SEARCH response (a) does not represent the state of the resource identified by the effective request URI and (b) caches are currently unable to vary based on request payload.
> 
> On Apr 21, 2015 1:38 PM, "[email-masked]" <[email-masked]> wrote:

--
Mark Nottingham   https://www.mnot.net/

---

From: James M Snell <[email-masked]>
Date: Tue, 21 Apr 2015 16:27:11 -0700

Ah Yeah, good point... Pretty much underscores that the SEARCH response
itself is not cacheable.
On Apr 21, 2015 4:23 PM, "Mark Nottingham" <[email-masked]> wrote:

> > If the SEARCH response has a Content-Location header and that was used
> for the cache, then yes, the response would be cashable.
>
> > Otherwise, you cannot assume cacheability because the SEARCH response
> (a) does not represent the state of the resource identified by the
> effective request URI and (b) caches are currently unable to vary based on
> request payload.
> >

---

From: "[email-masked]" <[email-masked]>
Date: Sat, 25 Apr 2015 20:09:34 +0200

> On 22 Apr 2015, at 01:23, Mark Nottingham <[email-masked]> wrote:
> 
> 

Is this just a limitiation of current caching protocols?

Let's put it differently. I'd expect that if I sent the same query to the same resource with the
same conditional request (on the same etag), I'd get the same result. Why should I not?

So that means that a cache could check something like SEARCH + URL + mime-type and body of response and store the result, which it could return directly in another request if the  appropriate headers + body of request matched.

It seems that your claim that SEARCH responses are not cachable is related to a very specific idea of how caches operate. Am I right? Perhaps you can point me to the document.

Perhaps that document would allow  one for the sake of argument to put the query in the header. Would it then be cacheable? 
If that is correct then it might be worth noting this, because one can imagine caching services evolving too.

Also along these lines it seems to me that there is something that could be said about how multiple queries could end up allowing a client to get a complete overview of the remote data - which can only work if the above cacheability holds I think.

For example JavaScript clients may want to make SEARCH requests to get fast and partial answers for very large data sets, and be able to explore the data in small chunks at a time as needed, all in the knowledge that they are getting a better and better view of the remote resource, be it in small chunks.

Henry

Social Web Architect
http://bblfish.net/

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 25 Apr 2015 11:33:54 -0700

> On 25 Apr 2015, at 11:09 am, [email-masked] wrote:
> 
> 

No, it's a limitation of the semantics of Content-Location:

http://httpwg.github.io/specs/rfc7231.html#header.content-location

Resource A on a server can't cause something to be cached under the URI of Resource B, because they may not have the same owner.

---

From: "[email-masked]" <[email-masked]>
Date: Sat, 25 Apr 2015 20:49:41 +0200

> On 25 Apr 2015, at 20:33, Mark Nottingham <[email-masked]> wrote:
> 
>> 

Yes, that makes sense, and I do agree with that. 

I suppose what I don't understand is James M. Snell's remark in response to your message where he wrote

"Ah Yeah, good point... Pretty much underscores that the SEARCH response itself is not cacheable."

If search is cacheable when the Conent-Location of the response matches the effective request uri, how does that show that the SEARCH response is not cacheable?

Social Web Architect
http://bblfish.net/

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 26 Apr 2015 06:56:51 +1200

> On 26 Apr 2015, at 6:49 am, [email-masked] wrote:
> 

It is cacheable — in the sense that it can be stored. However, that stored response can only be used to satisfy future GET requests to the same URI — which is probably not what you want.

HTTP caching operates upon representations of resource state, which means accessing the contents of the cache is always GET (or HEAD).

This makes methods that also return things that look like representations difficult to work with — e.g., OPTIONS, PROPFIND. Generally, the better way to do things like this is to make them proper resources, and use links to relate them to the resource they're "talking" about. 

Cheers,

---

From: James M Snell <[email-masked]>
Date: Sat, 25 Apr 2015 12:10:20 -0700

> It is cacheable — in the sense that it can be stored. However, that stored response can only be used to satisfy future GET requests to the same URI — which is probably not what you want.

It definitely is not what we want. The SEARCH response is a not
representation of the resource. It's NOT what you'd expect to get back
when sending a GET to the same URI.

- James

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 26 Apr 2015 07:40:13 +0200

> On 25 Apr 2015, at 20:56, Mark Nottingham <[email-masked]> wrote:
> 
> 

Great!  We are getting closer to the core of the problem, which I had tried to address earlier :-)

I agree that HTTP caching should operate upon representations of resource state. My point it that SEARCH always returns a partial representation of the resource state, so that it should be cachable too. This means
improving the caching stack so that it knows how to update partial representations.

To illustrate the different cases, let us take the example from http://datatracker.ietf.org/doc/draft-snell-search-method/ . The resource <http://example.org/contacts> is a small table of contacts.
Let us imagine that

A. GET followed by SEARCH
-------------------------

1. James makes a GET request on <http://example.org/contacts> via a cache C and C caches the returned  representation with etag1 ( and the same Location header )
2. Ashok then makes a conditional SEARCH request on <http://example.org/contacts> via the same cache C with etag1 too

I think it is important that SEARCH guarantee the following:

it should not be necessary for the cache C to send the search request all the way back to the server example.org if it knows the query lanuage used in the SEARCH request, because it should be able to answer that
itself using the information from the GET in 1. above.

B. SEARCH followed by SEARCH
----------------------------

1. a JS Agent in a browser does not know how large </contacts> is, and only needs to render a couple of fields from that table, so it sends a SEARCH for those fields to the server example.org via it's local in browser cache BC. 
2. The same JS Agent later needs the same to fields again for some different purpose, and sends the same query again using a conditional GET with the same etag. 

Here it should be clear that the in browser cache should be able to immediately return the previous response without needing to connect to example.org. One way to think about it is that this is just a short cut for the previous example A, where the script would have first done a GET for the full resource and then a SEARCH on the representation returned by the GET.

Interestingly enough in this case the cache could be much dumber and not even know abut the query language. As long as the query is the same byte for byte, has the same mime type, is on the same resource with the same etag, it should be able to return the correct result.

> This makes methods that also return things that look like representations difficult to work with — e.g., OPTIONS, PROPFIND. Generally, the better way to do things like this is to make them proper r…

Here we are dealing with partial representations of a remote resource. This is why I say that PATCH is to PUT what SEARCH is to GET. They are just ways to save bandwidth. But in both cases they work on the representation of the resource.

This may require altering the capacities of caches over time. Current caches that don't know about SEARCH will not do anything wrong by not caching, but future optimised caches will be able to make a big difference to the responsiveness of the web.

I hope this helps.

Henry

Social Web Architect
http://bblfish.net/

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 26 Apr 2015 19:48:21 +1200

On 26/04/2015 5:40 p.m., [email-masked] wrote:
>>> If search is cacheable when the Conent-Location of the response
>>> matches the effective request uri, how does that show that the
>>> SEARCH response is not cacheable?
>> 
>> It is cacheable — in the sense that it can be stored. However, that
>> stored response can only be used to satisfy future GET requests to
>> the same URI — which is probably not what you want.
>> 
>> HTTP caching operates upon representations of resource state, which
>> means accessing the contents of the cache is always GET (or HEAD).
> 
> I agree that HTTP caching should operate upon representations of
> resource state. My point it that SEARCH always returns a partial
> representation of the resource state, so that it should be cachable
> too. This means improving the caching stack so that it knows how to
> update partial representations.
> 
> To illustrate the different cases, let us take the example from
> http://datatracker.ietf.org/doc/draft-snell-search-method/ . The
> resource <http://example.org/contacts> is a small table of contacts. 
> Let us imagine that
> 
> A. GET followed by SEARCH -------------------------
> 
> 1. James makes a GET request on <http://example.org/contacts> via a
> cache C and C caches the returned  representation with etag1 ( and
> the same Location header ) 2. Ashok then makes a conditional SEARCH
> request on <http://example.org/contacts> via the same cache C with
> etag1 too

What you describe here is a GET being cached and then used to answer a
SEARCH.

This has no bearing on whether SEARCH is cacheable. The response to the
SEARCH remains itself regardless of whether it was generated from the
origin resource or a *complete* copy of the origin resource.

> 
> B. SEARCH followed by SEARCH ----------------------------
> 
> 1. a JS Agent in a browser does not know how large </contacts> is,

Therefore the cache cannot answer SEARCH C requesting 5 records, if the
SEARCH A only caused 3 records to be cached.

Meaning the SEARCH has dynamic variance in responses. The method alone
cannot state that the response is cacheable since it depends on other
case-specific criteria.

> and only needs to render a couple of fields from that table, so it
> sends a SEARCH for those fields to the server example.org via it's
> local in browser cache BC. 2. The same JS Agent later needs the same
> to fields again for some different purpose, and sends the same query
> again using a conditional GET with the same etag.

Wrong. It would need to send SEARCH again with the same variant
negotiation details (ie query and language). *IF* the cache has for any
reason discarded its earlier copy the server will be getting the
request. You really dont want it to be GET on a large database.

SEARCH as a method supplies similar cache needs and semantics to
GET+Range request. Except that there is no byte offsets defined in the
SEARCH request to make things easy for caches to identify object byte
overlaps in stored responses.

As such, Range can be used to best describe the caching problem with
SEARCH. ... Caching Range responses is possible but a 206 response with
10 bytes of length cannot be used to supply a single Range of 20 bytes.

With SEARCH this problem is slightly worse because we dont know if or
where there the missing records might be in the resource. We are forced
to send the whole SEARCH to the server to find out - which means the new
reply effectively replaces any cached one and there is no gain.
 At least with Range the cache could have optimize the backend query to
ask for bytes 11-20.

IFF we make the assumption that SEARCH is cacheable *somehow* - the
cache will need to compare the method, URI and all other negotiation
criteria defined by SEARCH as being relevant before it will serve up the
cached response. Any non-match is a different SEARCH query requiring the
backend server to supply new response.

I like your proposal that the response to SEARCH be treated as a partial
response. To the point that I also believe it should be indicated that
the preferred response status code is 206 with Content-Range header
indicating the positions or numbers (in bytes, or maybe a new range
type) of the fetched records within the base resource.

The 206 status is already defined in RFC 7233 with suitable criteria to
cover the SEARCH cases. The wording for it also allows a SEARCH "not
cacheable by default" definition to be ignored - that could be improved
by SEARCH adding a mention that 206 might be cacheable.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 26 Apr 2015 19:54:10 +1200

> Please see: http://datatracker.ietf.org/doc/draft-snell-search-method/

In section "1. Introduction" the second example is titled

 "A typical use of HTTP GET for requesting a search"

... then begins with a POST.

Amos

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 26 Apr 2015 11:09:33 +0200

> On 26 Apr 2015, at 09:48, Amos Jeffries <[email-masked]> wrote:
> 

This example was just to show that the SEARCH should be completely
dependent on the representation returned by the GET.

> cannot state that the response is cacheable since it depends on other

yes.

> request. You really dont want it to be GET on a large database.

Duh! I meant "using a conditional SEARCH"! Thanks for spotting that.

> overlaps in stored responses.

yes

> backend server to supply new response.

yes. If all the relevant fields match exactly then it should be possible 
to just send back the same byte for byte response. ( it remains to be determined
what the relevant fields are )

Note that if the SEARCH query is place in the body of the request as is suggested
by draft-snell-search-method ( which makes a lot of sense ), then the 
content of the request body needs to be taken into account too. 

> by SEARCH adding a mention that 206 might be cacheable.

yes. That is what I was getting at. Thanks for helping me clarify that in
terms of established RFCs. 206 (Partial Content) is a good find :-)

I think in this case the range is defined by the SEARCH query sent in the body
and the mimte type of the body of course. ( an initial investigation path may
be to consider a hash of those as a potential candidate Content-Range ) 

So for the hypothetical SPAQRL query one could imagine the first query requesting
the number of columns and rows, and then subsequent queries filling in different
chunks of columns and rows. A SPAQRL aware cache could then look at the data it already
had for a particular resource to see if it could answer a particular query without needing
to make a new request to the server. Say the first 10 queries ( QUERY was the original 
name of the method proposed btw http://www.w3.org/Protocols/HTTP/Methods.html ) gave it 
all the information  for all the columns of the first 100 rows, then a new query that 
only made requests within that chunk of the table, would allow the cache to respond from 
the data available  to it. At the limit the cache should even be able to work out if 
it had received the whole view, and from that be able to respond to a GET, even though it
had never made a GET request at all.

This is probably the way partial responses using byte offsets work currently with
caches. If a number of requests give the partial offset for byte 0 to 100 thousand, then
a new request for bytes 500 to 10 thousand should be able to be served without the cache 
needing to make a new request to the origin server. 

SEARCH does require more semantics though than byte offsets, but it is nice to see
that it can fit in that mould. This extra complexity of SEARCH also explains why it
has taken so much longer to make its way to the IETF .

Henry

Social Web Architect
http://bblfish.net/

---

From: Julian Reschke <[email-masked]>
Date: Sun, 26 Apr 2015 11:17:55 +0200

On 2015-04-26 07:40, [email-masked] wrote:
> ...

And you seriously believe that people are going to upgrade their caching 
stack specifically for SEARCH?

> A. GET followed by SEARCH

That not only implies that caches special-case the SEARCH method, but 
also know the semantics of SEARCH payloads.

> ...

I'd prefer to concentrate on the concrete method definition first.

As Mark said, once you leave the world of GET and HEAD, standard caches 
won't help you anymore.

Of course a server can indicate that a specific SEARCH response is 
indeed cacheable (cache-control!), but it's unlikely that generic caches 
will start to implement that.

If what's needed is a mechanism to discover GETtable resources that 
implement a specific query, something like 
<https://greenbytes.de/tech/webdav/draft-reschke-http-get-location-latest.html> 
might be needed.

Best regards, Julian

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 26 Apr 2015 12:52:49 +0200

> On 26 Apr 2015, at 11:17, Julian Reschke <[email-masked]> wrote:
> 
> And you seriously believe that people are going to upgrade their caching stack specifically for SEARCH?

I suppose you are talking about more difficult to change caching stacks like those deployed by ISPs or even countries. ( In 1994 I remember the UK had a country wide cache to help us get pages from the  USA faster ). Those caches will presumably only change when the need for it is strong enough. Luckily they don't have to change those services until they feel like it. 

What will happen immediatley though is that people like me and others on this list, will write  local JS caches for data using any of the numerous storage mechanisms available in browsers.  We will also write servers that can implement SEARCH for local and remotely proxied resources, as I have partly done here in Scala here

  https://github.com/read-write-web/rww-play/wiki/Curl-Interactions#sparql-queries

Some people may wish to implement SEARCH for XML query languages, other for CSV formats, others for JSON, etc. I'll be doing something for JSON-LD.

But we might as well write the spec for SEARCH in such a way that when the market grows enough that the large infrastructure companies will want to buy new enhanced proxies tuned for the languages that are most used ( probably nothing to do with what I am doing ) to improve the service for their customers, that these can then together with the stacks the initial group has deployed to satisfy their more limited needs.

> That not only implies that caches special-case the SEARCH method, but also know the semantics of SEARCH payloads.

yes, and I am just building a client Scala-JS store and a server Scala store to do that for a couple of query formats, under the Apache licence. 

> As Mark said, once you leave the world of GET and HEAD, standard caches won't help you anymore.

Yes, but I am not relying on standard caches. I am writing in-browser and server caches myself. The server caches are needed anyway because of the limitations of CORS.

> 
> Of course a server can indicate that a specific SEARCH response is indeed cacheable (cache-control!), but it's unlikely that generic caches will start to implement that.
> 
> If what's needed is a mechanism to discover GETtable resources that implement a specific query, something like <https://greenbytes.de/tech/webdav/draft-reschke-http-get-location-latest.html> might be needed.

It is important for me that the query be made directly on the resource, and that it be clear to the intermediate caches that the results are partial views on that resource, so that the intelligent cache can built these partial views together into ever more complete ones.

Social Web Architect
http://bblfish.net/

---

From: Ashok Malhotra <[email-masked]>
Date: Sun, 26 Apr 2015 11:51:22 -0400

Hello Henry:
I understand that you want intelligent caches that use knowledge about the
resource and the results.  This is a fine goal but our priority now is to get
the SEARCH proposal accepted with the existing infrastructure. Let's work on
getting that done.  Then we can work on intelligent caching.
All the best, Ashok

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 26 Apr 2015 18:20:50 +0200

> On 26 Apr 2015, at 17:51, Ashok Malhotra <[email-masked]> wrote:
> 
> Hello Henry:

Well it looks like there is support for SEARCH which is good news :-)

Btw. one should count to existing infrastructure web servers that need to
write web proxies that cache remote results in order for JS apps to be 
able to work on distributed data.

I tend to think that it can't harm to think a bit about how SEARCH fits 
into the wider set of IETF specifications and if it does then that is even 
better news, and could be very useful in helping define what it is. 

For example, my intuition that SEARCH returns a partial representation 
has led us  to discover ( thanks to Amos Jeffries ) that we should look 
to adopt  a 206 status code as defined in RFC 7233.

Does this mean that example 4.1 from draft-snell [1] where the SEARCH returns 
a 200 is wrong? Or is it correct that SEARCH is not cacheable other than
via a 206 route? Is this something one can add later, or may not specifying
this now lead to problems further down the road?

The best time to answer this is now, rather than later, to make sure these 
extensions that don't make it into the spec can be specified later.

Henry

[1] http://datatracker.ietf.org/doc/draft-snell-search-method/?include_text=1

Social Web Architect
http://bblfish.net/

---

From: Ashok Malhotra <[email-masked]>
Date: Sun, 26 Apr 2015 13:42:30 -0400

> this now lead to problems further down the road?
Good question, since it can be argued that Range Requests bear some relation to SEARCH!
Julian?

Best, Ashok

---

From: Julian Reschke <[email-masked]>
Date: Sun, 26 Apr 2015 21:02:28 +0200

> Good question, since it can be argued that Range Requests bear some

Right now I neither see how a SEARCH response is necessarily a partial 
representation of the request resource, nor any relation to HTTP Range 
requests.

Best regards, Julian

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 27 Apr 2015 07:36:10 +0200

> On 26 Apr 2015, at 21:02, Julian Reschke <[email-masked]> wrote:
> 
> Right now I neither see how a SEARCH response is necessarily a partial representation of the request resource, nor any relation to HTTP Range requests.

It would help if you explained how you disagree with the arguments put forward.
Let's try the Socratic method then:

(1) Do you agree that SEARCH is (should be) a method that is applied to the resource on which the request is made?
  (a) if yes, then is the result usually the full representation returned as with GET? Or is it a partial representation?
  (b) if no, how does the result of a SEARCH relate to the resource on which the method was applied?
      
Henry

Social Web Architect
http://bblfish.net/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 27 Apr 2015 08:59:10 +0200

> (1) Do you agree that SEARCH is (should be) a method that is applied to the resource on which the request is made?

Depends on the definition of "applied". You might want to read 
<http://greenbytes.de/tech/webdav/rfc5323.html#rfc.section.2.2.1>.

>    (b) if no, how does the result of a SEARCH relate to the resource on which the method was applied?

How and whether it relates depends on the request payload format and the 
actual request.

Best regards, Julian

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 27 Apr 2015 10:52:01 +0200

> On 27 Apr 2015, at 08:59, Julian Reschke <[email-masked]> wrote:
> 
> Depends on the definition of "applied". You might want to read <http://greenbytes.de/tech/webdav/rfc5323.html#rfc.section.2.2.1>.

I suppose I don't find that notion of "search arbiter" very satisfying in WebDAV. It is defined just as "A resource that supports the SEARCH method". My feeling is that by specifying SEARCH more generally in its own RFC we could do a lot better than this, and have something that ties in more coherently with the other methods. This is why I'd like to strengthen the following parallel: SEARCH is to GET what PATCH is to PUT. If we can get this to work then we build on very well understood intutions of GET and PUT, which are at the core of the Web, in a clearly RESTful manner.

Let us assume we do find a way to place SEARCH on more solid foundations like this. We'd still need to see if we can satisfy the intutions that WebDAV was trying to get at, but now in a RESTful manner. So I'll try to do both of these below.

WebDAV seems to be saying smething like this: an arbiter is an agent that can make requests on a number of other resources and respond to requests about them. This seem to define a resource that is dangerously close to a remote procedure call - it has a SOAPy feeling to it. Can we do this in a more declarative fashion?

I think so. We just need to notice that a resource - be it as simple as a 1993 web page - could make statements (true or false) about other resources. Most web pages do this. A trivial example but somehow exceedingly widely used one, are the set of web pages that point to other web pages with statements that they contain a funny cat picture. In more formal logical terms we can have belief statements that state what others believe eg: statements of the form S believes that P, S knows that P, S wishes that P, etc... This is knows as belief contexts and the more general notion has been described for exammple by Ramanathan V. Guha [1] - now VP of engineering at Google -  in his thesis "Contexts, a Formalisation and some Applications". Why do I mention this? Because once we see that a resources state may contain statements about other resources as shown by the representation returned on doing a GET on that resource,  then we can think declaratively about the SEARCH method, as giving a partial representation of the state of that resource, which may contain partial representation about other resources. And so we should not need this notion of a "search arbiter" anymore.

It may help if I  build up from simple to more complex examples.

1) Note that the CSV examples used in draft-snell-search-method-00 [3] are pure factual statements that contain no contexts. I would suggest the example first show an example of a GET representation of that resource, perhaps a partial representation, to emphasize that the resource may be very large. The SEARCH query on that resource should then return information that is a partial representation of that resource, and will not contain information about another resource, since the GET on that resource does not.

2) A slightly more complex example may be an Atom Feed for example from rfc4287 which contains information about other resources. The simplest example here has just one entry. A GET on such a feed returns us

<?xml version="1.0" encoding="utf-8"?>
   <feed xmlns="http://www.w3.org/2005/Atom">

     <title>Example Feed</title>
     <link href="http://example.org/"/>
     <updated>2003-12-13T18:30:02Z</updated>
     <author>
       <name>John Doe</name>
     </author>
     <id>urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af6</id>

     <entry>
       <title>Atom-Powered Robots Run Amok</title>
       <link href="http://example.org/2003/12/13/atom03"/>
       <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
       <updated>2003-12-13T18:30:02Z</updated>
       <summary>Some text.</summary>
     </entry>
   </feed>

If I SEARCH this with a query language such as XQUERY I could end up with information about another resource, as seen by the feed resource - note that the feed resource may be out of date - and in that case the search on that state will also be out of date. For example I could send a SEARCH that would ask for titles of all the entries, which would return something like

<result>
  <title>Atom-Powered Robots Run Amok</title>
</result>

( a more complete example would be useful )
It seems that the client should be able to get the same information by 
1) doing a GET
2) applying the Query directly on the returned result

3) a Search engine like Google has a resource that has a state that is a practically infinitely long statement about what its crawler has seen on the web. Note that for some resources this may be different from what others may see when the do a GET on those resources, because the resources may have changed in the mean time, because the resource may be individualised for each user, or because the web site may lie to Google to get better search ranking. That is when Google makes a statement about what other pages on the web say, it is quoting what those sites told it (contexts are a formal quoting mechanism). When one does a SEARCH on that resource one gets back a view of that resource. Clearly doing a full GET on Google is practically impossible and not desireable. But logically the same thing is happening as above. The resource we would be SEARCHing on Google has as a state information about what it believes the state of other resources are on the web. By SEARCHing that resource we get the equivalent of what we would get were we to have first downloaded the state of their database and done ourselves a query on that.

I hope this helps give weight to my proposed tag line "SEARCH is to GET what PATCH is to PUT", and that this shows how we can get a more RESTful description of SEARCH and thereby putt it on a more solid foundation.

[1] http://www.guha.com/cv.html - VP of engineering at Google
[2] "Contexts, a Formalisation and some Applications" http://www-formal.stanford.edu/buvac/guha-thesis.ps
[3] http://datatracker.ietf.org/doc/draft-snell-search-method/
> How and whether it relates depends on the request payload format and the actual request.

And on the state of the resource. My point is: SEARCH results should depend on only two things:

1) the state of the resource
2) the query

such that if the client ( or the cache ) had the full state of the resource, it could apply the query to that state and get the same result back.

Henry

> Best regards, Julian

PS. Thinking about 206 I think that it would make sense for SEARCH when the result returned by only a partial answer of the SEARCH result, so that RFC 7233 would allow one to page through huge SEARCH results.

Social Web Architect
http://bblfish.net/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 27 Apr 2015 11:52:47 -0700

[resending from the right address]

> On Apr 27, 2015, at 1:52 AM, [email-masked] wrote:
> 

The original SEARCH method was removed from HTTP in 1994 because it discouraged
the use of identifiers for important resources. In short, it weakened the Web.

What MS implemented as SEARCH [1] and WebDAV defined (in RFC5323) was a really bad idea
for a site-wide query interface, along the same bad taste lines as service endpoints
for web services.

 [1] https://msdn.microsoft.com/en-us/library/aa143053%28v=exchg.65%29.aspx

In both of those definitions, SEARCH was not a method that scoped its results to the requested
URI.  That URI, in fact, had almost nothing to do with the results, making implementation
of the method a significant security risk.

Neither of those were RESTful in any sense of that term.  Regardless, the method was defined
and registered as such for HTTP (which is not limited to RESTful interaction), and there is
no reuse of method names allowed in HTTP.  You need to use a new method name if you want to
define a method with different semantics.

In any case, it remains true that generalized retrieval in the form of a method other
than GET would be actively harmful to the Web.  Partial query within the scope of the
current representation of a single resource is a very distinct concept that, if defined,
ought to have a distinct method name.  However, I would still consider that a bad idea,
since it is a loss in every way when compared to a link template to sub-resources.

BTW, HTTP defines caches based on the generic method/response relationship, not based on the
GET/representation relationship.  It is possible to cache responses to arbitrary methods if
the response indicates it is cacheable.  What is difficult is how to determine the cache key
for a given request, which HTTP only defines only for GET and for responses containing the
Content-Location header field (new methods would have to define their own cache keys).
Generic HTTP caches only implement for GET/HEAD because the likelihood of receiving a
later hit on a cached entry for anything other than GET is so ridiculously tiny that it isn't
worth caching outside of application-specific client caches (for which generic semantics
are usually irrelevant because the application can be tightly coupled to the cache).

Cheers,

Roy T. Fielding                     <http://roy.gbiv.com/>
Senior Principal Scientist, Adobe   <http://www.adobe.com/>

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 27 Apr 2015 22:34:11 +0200

> On 27 Apr 2015, at 20:52, Roy T. Fielding <[email-masked]> wrote:
> 

It is my feeling that the authors of this draf want to do the right thing here, but were worried
to come up with a new method name, and favored going with something existing like SEARCH rather 
than try something new. I was hoping in the previous mail that SEARCH could be redefined to do 
the right thing. But if it cannot then another method name is welcome. 

For this I propose to use the verb QUERY, which I think would be much clearer and less heavy sounding than SEARCH, which is too active and open.

> since it is a loss in every way when compared to a link template to sub-resources.

I do have some further arguments for QUERY that I feel would argue strongly in favor of it. 
Those I can think of at present are:

• Query templates (urls with attribute values following a '?' have a number of drawbacks:
   - size limit: the URL are limited practically to 2k or so.
   - URL explosion: one ends up with a lot more URLs - and hence resource - than needed, 
 with most resources being just partial representation of the original resource
   - no query language: If one wanted to use advanced query languages in the template, this would make for even worse URLs, that would if one wanted them to be somewhat transparent require even mime type information to be added to the URL! Some services do put XQuery or SPARQL in the URL. I just don't feel that is right.

• QUERY is better for caching.
Query templates do not allow the caching behavior that we are looking for.
  - An intelligent cache would need to be able to learn the semantics of the attribute values following the ? to be able to work out that a change to the original resource would change the query template URL too. This cannot be done in a generic manner across web sites. Somehow caches would need to understand the relation between a number of query urls and the original resource.
  This is not the case for a QUERY verb following the restrictions you proposed. Here a cache seeing that a successfil PUT or a PATCH on the same URL has taken place can immediately invalidate all the QUERY results it has for that resource. Of course a DELETE would invalidate all the QUERY cache too.
  - The same is true for time to live information for query urls, which are disconnected from the 
 original resource
  - The cache cannot itself respond to queries, which it might be able to do if it had the full representation returned by a GET on that resource. ( something we would be able to explore if we 
define it correctly )

• Access Control
   With query urls you have a huge number of URLs referring to resources with exactly the same access control rules as the non query resource. This means much more complex access control rules, more work tracking of rights managements, privacy settings, etc...

> are usually irrelevant because the application can be tightly coupled to the cache).

Speaking from my own experience, which I think can generalise to other frameworks, I know there are applications using the semantic web where we want to write client applications that can follow linked data around the web using just RESTful interaction patterns where QUERY would be very important method: it would allow the client to be able to build up quickly user interfaces by fetching just the data in a resource that it needs for the current display: eg. the minimal bounded graph around a node in a graph. Here time is critical: the User Interface has to display quickly and be reactive. If we are limited only to GET on a resource that is being downloaded somewhere from the  web by following links, that may also be of unknown size, we won't be able to gain adoption. This is why for me it is very important that the "query be within the scope of the current representation of a single resource", even if this resource can of course make statements about other resources, as atom feeds do all the time. This is because the client needs to know that the query on a particular resource is just a part of what it would have gotten had it asked for a GET. It would be of no use for the client if a QUERY on one resource just gave it information about some arbitrary other resource.

 To give a bit of context: we are trying to build generic applications following RESTful protocols. If we succeed then "application specific client caches" will become standardised, and so this space will be huge. Potentially every server on the web - and here we will want to count the hundreds of million of Freedom Boxes where citizens can host their own social network - will have a cache that will work in the background fetching feeds, receiving notifications, etc... 

Anyway, it is in writing these  user interfaces that the importance of QUERY became clear to me. I think
others will find many of the arguments apply to their use cases too, and to other query languages, even if they don't buy into the passion which drives me to work on these open source projects.

Thanks a lot for your insight,

	Henry Story

Social Web Architect
http://bblfish.net/

---

From: ashok malhotra <[email-masked]>
Date: Mon, 27 Apr 2015 16:34:46 -0400

Roy:
Consider a collection of books.  The collection has a URI
and each book has a URI and there may be link relations between
the collection and each book.  But sometimes, I want to find books
by title or author or ISBN instead of following links.
This is the usecase I am getting from several sources.

Would changing the name of the verb help?

All the best, Ashok

---

From: ashok malhotra <[email-masked]>
Date: Mon, 27 Apr 2015 16:36:36 -0400

I prefer QUERY but was outvoted.

All the best, Ashok

On 4/27/2015 4:34 PM, [email-masked] wrote:
> Speaking from my own experience, which I think can generalise to other frameworks, I know there are applications using the semantic web where we want to write client applications that can follow linked data around the web using just RESTful interaction patterns where QUERY would be very important method: it would allow the client to be able to build up quickly user interfaces by fetching just the data in a resource that it needs for the current display: eg. the minimal bounded graph around a node in a graph. Here time is critical: the User Interface has to display quickly and be reactive. If we are limited only to GET on a resource that is being downloaded somewhere from the  web by following links, that may also be of unknown size, we won't be able to gain adoption. This is why for me it is very important that the "query be within the scope of the current representation of a single resource", even if this resource can of course make statements about other resources, as at!
 om feeds
 do all the time. This is because the client needs to know that the query on a particular resource is just a part of what it would have gotten had it asked for a GET. It would be of no use for the client if a QUERY on one resource just gave it information about some arbitrary other resource.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 27 Apr 2015 22:49:39 +0200

> the right thing. But if it cannot then another method name is welcome.

There are already three safe HTTP methods that take a request payload 
(PROPFIND, SEARCH and REPORT). Some software stacks already know about 
these (for instance, wrt whether a request can be safely repeated on 
network failure). There's simply no reason to create yet another one, 
when, from an HTTP point of view, it does exactly the same thing.

> For this I propose to use the verb QUERY, which I think would be much clearer and less heavy sounding than SEARCH, which is too active and open.

I don't see any difference between these, but maybe it's because English 
is not my mother language.

Best regards, Julian

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 28 Apr 2015 00:41:32 +0200

> On 27 Apr 2015, at 22:49, Julian Reschke <[email-masked]> wrote:
> 
> There are already three safe HTTP methods that take a request payload (PROPFIND, SEARCH and REPORT). Some software stacks already know about these (for instance, wrt whether a request can be safely repeated on network failure). There's simply no reason to create yet another one, when, from an HTTP point of view, it does exactly the same thing.

I think the disagreement is that SEARCH does what we need. To quote Roy Fielding's 
answer:

 " In both of those definitions, SEARCH was not a method that scoped its results 
   to the requested URI.  That URI, in fact, had almost nothing to do with the 
   results, making implementation of the method a significant security risk. "

I tried to argue that SEARCH could and should be seen as doing exactly that. If you 
have an argument that it does, by quoting the spec, then of course that would be great
as it would avoid another verb being created. I tried to defend SEARCH here

  https://lists.w3.org/Archives/Public/ietf-http-wg/2015AprJun/0244.html

by arguing that a resource can have information about another resource. I gave as
examples the Atom Syntax. In RDF I could do the same using the N3 notation to express 
something like this formally by using { } quotations, or without that by using rdfxml 
literals. For a human everyday example consider that above I quoted Roy Fielding's 
statement. You could now query this e-mail about what Roy Fielding wrote, but that 
would not be the same as having received and read the e-mail he sent. Hopefully the
two overlap closely but it is quite possible that he has changed his mind in the mean
time or that I quoted him out of context, and lost an important part of his 
argument. Still if you QUERY this e-mail with a standard query language, then I should
be able to do the same with my version of this e-mail and reach the same results.

IF you search for what Roy said, then you'd probably try to go to the archive to find
what he said there. 

So if SEARCH ( the HTTP verb used by WebDAV) could work by scoping its results to a 
resource R it could nevertheless still give results about other resources iff R 
made statements about other resources too. ( or perhaps at the limit if it contained them )
 
I gave an example for Atom xml. 

The reason it is important is that I SHOULD not get a certain result when querying R
directly and a completely different result when GETing R and then querying the 
representation.

Otherwise QUERY or SEARCH won't give me any efficiency advantage: all I'd have now are
two different results, and I'd need to make two requests for each resource: a GET and a 
SEARCH, and if I relied on one I might be coming to a conclusion different from what
someone using the other method would arrive at, which would lead to security issues.

That is why the results must be scoped to the result URI. Another way of putting it,
is that the only thing that matters about a SEARCH is the state of the resource and
the QUERY/SEARCH sent to it. 

If you accept that a resource can talk about another one, then the state of a resource
may be queried about what it believes the state of another resource is. It nevertheless
remains the state of that resource. If it is a reliable resource then of course you'd
hope that it makes correct statements about other resources. But that is just another
additional fact that need not come into the definition of QUERY/SEARCH.

I hope I have understood Roy correctly.  You know enough about WebDAV to tell me if
this is in the spirit of SEARCH, or if Roy is correct in stating that SEARCH was 
weakly defined. Note that it may have been weakly defined but that pragmatically all
implementations just do the right thing. It is just more work to proove that.

> I don't see any difference between these, but maybe it's because English is not my mother language.

SEARCH is active. When I search for something it is usually because I have lost it, and so
I go round and round in circles, I call people, I try to remember where I went, ... I try to
bring as much external information to find what I am looking for. I may also look for something
that I am not quite sure I know about ( that's Meno's paradox ).

QUERY is much ligher. It is asking a question. You can search by asking questions, but questions are
primary. (If you don't have a question, do you know what you are searching for?) Querying is also one of those primary speech acts in philosophy of language. In a query a part of the sentence is missing, and you ask your interlocutor to find ways of filling it with information he has ( in his state of mind ) to make the sentence filled in true.

Q: Where is the cat?
A: On the mat.

=> The cat is on the mat.

Hope that helps,

	Henry

Social Web Architect
http://bblfish.net/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 28 Apr 2015 07:29:11 +0200

>     results, making implementation of the method a significant security risk. "

Well, Roy is overstating things. That particular usage pattern depends 
on the query grammar, and is totally optional in the only grammaer 
defined in RFC 5323; see 
<http://greenbytes.de/tech/webdav/rfc5323.html#rfc.section.5.4>.

Best regards, Julian

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 28 Apr 2015 14:44:22 +0200

[ Where I try to argue more formally to see if there is space for agreement
between Julian and Roy. I also give 2 arguments as to why SOAP SEARCH will not
be viable in a hyperdata web. But this reasoning ends me wondering wether QUERY 
might not fold neatly into GET. ]

> On 28 Apr 2015, at 07:29, Julian Reschke <[email-masked]> wrote:
> 
> Well, Roy is overstating things. That particular usage pattern depends on the query grammar, and is totally optional in the only grammaer defined in RFC 5323; see <http://greenbytes.de/tech/webdav/rfc5323.html#rfc.section.5.4>.

Interesting. Thanks for the pointer. 

My feeling is that there is the sliperiness of the notion of scope may be leading to confusion.
( But Roy may be basing his statement on other documents, so I can't comment for him on that. )

As I argued previously if I query a document that makes a statement about another document,
as we are constantly doing on this mailing list, by quoting each other, then one can at least
make queries as to what a document says another document says. That is I suppose uncontroversial.
So I could query my previous mail asking it whether it stated that Roy stated that SEARCH
was not scoped to the request URI. If the mail archive were QUERY/SEARCH enabled with an advanced
human language query language, then it could answer to "yes" to that question.
So I suppose Roy has no issue with that type of scoping. If I asked the same resource if Roy 
made a statement about caching in that same e-mail, it would have to say either no, since 
I did not quote him as saying so, or perhaps "not as far as I can tell from this resource".

Perhaps if we now consider the more difficult case that may seem to be behind the clash between 
you and Roy on WebDAV. First I note that here WebDAV, Atom, and LDP ( Linked Data Platform ) [1] 
use very  similar concepts. They all have a notion Containers and their contents. LDP even defines 
the  ldp:contains relation relating containers to the contents they created. Because I know LDP best
I can demonstrate this with a simple example. To take the LDP Primer [2] figure 1, I could have a </container/> resource that returned on a GET requesting text/turtle, the following [3]

<> a ldp:Container;
   ldp:contains <jane>, 
                <image> .

Here we have a container containing a data resource <jane> and an image. A DELETE on </container/image>, MUST remove the ldp:contains relation from the container. One can therefore consider - as with WebDAV depth
1 scoping - that the relation between the container and the contents are close enough here that the container could potentially *quote* the contents too ( as Atom does with its <content> element ). 
It ABSOLUTELY MUST quote the contents or else someone could lie about the the container in the content, and mess everything up by claiming the container contained things it did not contain. So potentially the container could return on a GET with the text/n3 the direct content and the quoted
content.

<> a ldp:Container;
   ldp:contains <jane>, 
                <image> .

<jane> log:semantics {
   <jane> foaf:primaryTopic <jane#me>.
   <jane#me> a foaf:Person;
      foaf:name "Jane";
      foaf:depiction <image>;
      foaf:knows <http://www.w3.org/People/Berners-Lee/card#i> .
}

Here <jane> log:semantics {...} functions like the <content> element in 
atom xml.
 
(Of course quoting an image inside the container is just ugly, and if you 
hair don't stand on your head at the thought then you are probably not reading 
this list :-)

Still it would make sense to be able to QUERY/SEARCH the container with a language
that understood quotations. SPARQL [5] for example allows this and one could ask
something like

QUERY /container/ HTTP/1.0
Content-Type: application/sparql-query; charset=UTF-8
...

CONSTRUCT { ?person foaf:name ?name }
WHERE {
 <> ldp:contains ?g .
 GRAPH ?g {
  ?g foaf:primaryTopic ?person.
  ?person foaf:name ?name .
} 

This would make more sense if this container contained a lot of content of course,
and this is what Ashok is referring to in his question about books.

Now here again I think we can still argue, if we are very careful, that the query
is only covering the state of the resource, even if the resource has a very strong
binding with its contents, which can and should be queried independently. (My guess
is that if the contents are that tightly linked to the content, then any change to
the contained resources should automatically also alter the containers etag.)

So we have to be very careful here because if this is abused then we would end up
with a "generalized retrieval in the form of a method other than GET [which] would 
be actively harmful to the Web." (Roy Fielding) The fear here is that the only method
would then be SEARCH on the root container, and then we would have a new form of
SOAP going through SEARCH, which would mess up a lot of nice properties of the web.

So what is the counter-weight to this happening? I think it is simply the growing
deployment of Linked Data ( as shown by http://lod-cloud.net ). A Linked Data
application finds the definition of a term by dereferencing the URL of the object.
So for example if my JS client wants to know how to display 

  <http://www.w3.org/People/Berners-Lee/card#i>

Then it will have to do a GET on <http://www.w3.org/People/Berners-Lee/card>
to get its meaning as shown in the WebID diagram [6]. It would not know initially
to do a SEARCH on <http://www.w3.org/People/> , or on <http://www.w3.org/>. The
authoritative resource is Tim Berners Lee's WebID without the fragment identifier 
( as explained by §3.5 of RFC 3986 on URIs [7] ).

Furthermore a search on <http://www.w3.org/People/> would be much more complex, as
it would need to be stated in terms of what <http://www.w3.org/People/> believes to
be the state of graphs it contains, or for <http://www.w3.org/>, were it to be a container,
what containers it contains believe their content to contain. The further we get away
from the resource the more levels of nesting of quotation we have, and the less likely 
we are to really know what is going on. We might as well ask the original resource. 

So these are a couple of reasons why in a hyperdata space GET will remain the primary
retrieval mechanism, and all engulfing SEARCH SOAP Bubbles will have very short half
lives.

Well if GET is the method used to find the authoritative meaning of terms, how does 
SEARCH/QUERY come to be used then? In rww-play the server implements SEARCH 
( I can switch to QUERY without problem ) and the JS knows this, so it can query the 
resources using that method - and this makes sense as long as it is guaranteed that 
it is getting just a part of the information about the resource which it would also 
have with GET.  For remote resources the JS goes through a protected caching proxy 
( urls on the server that can asynchronously fetch remote resources, for the user ) 
and the JS also knows that SEARCH/QUERY is available for those. In the long 
term if QUERY were to be widely enough available the JS could directly make a request on the 
original resources using QUERY.

At this point though one nearly wishes for QUERY to default to GET, so that a resource
that does not understand the QUERY language sent would still return the full representation. 
This type of fallback is not possible for PATCH to PUT since PUT already has content. 
But since GET never has an associated content, it may oddly enough work there... It would 
require the returned content to contain a header stating that the response is a response 
to the query in the GET body, because otherwise the client may not be able to distinguish 
the result of the QUERY from the full result returned by a server that did not understand the body. 

Hmmmm. Any thoughts on this?

Henry

[1] the LDP WG wiki page: http://www.w3.org/2012/ldp/wiki/Main_Page
    The published W3C recommendation: 
       http://www.w3.org/TR/ldp/
[2] http://www.w3.org/TR/ldp-primer/
[3] I changed the second content to be an image in order to get a bit faster to the point.
[4] http://www.w3.org/TeamSubmission/n3/ 
[5] http://www.w3.org/TR/sparql11-query/
[6] http://www.w3.org/2005/Incubator/webid/spec/identity/#overview
[7] https://tools.ietf.org/html/rfc3986#section-3.5

Social Web Architect
http://bblfish.net/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 28 Apr 2015 09:29:18 -0700

> On Apr 27, 2015, at 10:29 PM, Julian Reschke <[email-masked]> wrote:
> 

That is scoping by the message body, not by the method semantics.  The difference is
that the request target contains essential bits for routing a request within (or behind)
the origin server, so if a special route is applied to, for example, a path within the
URI, then we want the method semantics to be limited to the scope of that path.
If the scope is not limited by the method, then implementation gets very messy.
[POST might have remained limited in that way as well, but the introduction of forms
made its original semantics irrelevant, so there wasn't much point in scoping them.]

Routing can't parse the request body for routing information because that would assume
all resources are the same and the body is going to be processed instead of responding
with 401, 403, or some other error code usually determined by path.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 28 Apr 2015 20:57:55 +0200

> That is scoping by the message body, not by the method semantics.  The difference is

Yes.

> made its original semantics irrelevant, so there wasn't much point in scoping them.]

I agree with the explanation, but exactly how is this different from how 
a GET on a search engine works?

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 28 Apr 2015 13:52:19 -0700

> On Apr 28, 2015, at 11:57 AM, Julian Reschke <[email-masked]> wrote:
> 
> I agree with the explanation, but exactly how is this different from how a GET on a search engine works?

Because a GET has all of the information necessary to see the scope
within the request target.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Wed, 29 Apr 2015 08:02:51 +0200

> within the request target.

Hm, no.

Example: <https://www.google.de/search?q=site:ietf.org+roy>

Best regards, Julian

---

From: Zhong Yu <[email-masked]>
Date: Wed, 29 Apr 2015 11:46:38 -0500

I'm very confused about the whole thing, but my first question would be

- How do I bookmark or link to a SEARCH?

Zhong Yu
bayou.io

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 29 Apr 2015 09:47:20 -0700

> On Apr 28, 2015, at 11:02 PM, Julian Reschke <[email-masked]> wrote:

The www.google.de origin server (including the network of gateways that are configured
to handle its requests) knows that its /search engine is scoped to the entire Web.
It even knows what site: means within the query syntax.

This is fundamentally different from the scope being specified within the body
because the body is not usually read until after the request has been routed and
most access control has been settled.

It is, of course, possible to interpret anything in a message at any time.
It just gets messy.

Regardless, I would not want to implement SEARCH as either a generalized query interface
(a la WebDAV) or as a query on the representation that one might have received from a GET.
We have one primary method for retrieval because having a URI is the sugar that the rest
of the system needs.  Sub-resources can be identified and retrieved directly by form,
script, template, or a single level of indirection (e.g., search results).  A smart
implementation can reduce common query parameters on the client-side so that the URI
produced is much smaller than whatever the user may have typed.  Queries that actually
require 2kB of search parameters can use POST -- a cache cannot key them anyway.  

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Wed, 29 Apr 2015 19:00:26 +0200

> It even knows what site: means within the query syntax.

This may be true for google.*, but is it also the case for every other 
search form we have on the Web?

> most access control has been settled.

I disagree with the "usually" here...

Anyway, if a search can be scoped outside the request URI, in most cases 
that won't be visible to the routing component you mentioned above. So I 
don't believe that SEARCH makes things worse than they already are.

> require 2kB of search parameters can use POST -- a cache cannot key them anyway.

Which makes it interesting to have a way to let the server compute a URI 
template suitable for GET based on a search query (be it POST, SEARCH, 
whatnot)

Best regards, Julian

---

From: Wenbo Zhu <[email-masked]>
Date: Wed, 29 Apr 2015 10:02:48 -0700

On Wed, Apr 29, 2015 at 9:47 AM, Roy T. Fielding <[email-masked]> wrote:

> produced is much smaller than whatever the user may have typed.

> Queries that actually
This looks reasonable, although you mentioned earlier that POST may be made
cacheable too (as controlled by the headers, not the method).

One other worry is the idempotence .. GETs are always safe to retry but
POSTs are not. Or we need clarify that cacheable POSTS are idempotent??

---

From: Julian Reschke <[email-masked]>
Date: Wed, 29 Apr 2015 19:06:11 +0200

> - How do I bookmark or link to a SEARCH?

In general you won't, because generic tools don't support that. But they 
could be written.

Best regards, Julian

---

From: Zhong Yu <[email-masked]>
Date: Wed, 29 Apr 2015 12:16:15 -0500

> In general you won't, because generic tools don't support that. But they

I don't quite understand that statement. But without linking, I don't
see the point of this pile of things we call web stack. We might as
well just use W Windows:)

Zhong Yu
bayou.io

---

From: Zhong Yu <[email-masked]>
Date: Wed, 29 Apr 2015 12:18:31 -0500

I mean X Window; if web is just a GUI to remote servers.

---

From: "[email-masked]" <[email-masked]>
Date: Thu, 30 Apr 2015 08:12:22 +0200

> On 29 Apr 2015, at 19:16, Zhong Yu <[email-masked]> wrote:
> 

In the draft snell the option is given for the server to create a Link
realation to the representation for that query. I suppose that could be used
by future bookmarking software. 
The same would be true for the GET + Query header or body I suppose  
(I have not thought through that yet ).

One use case that I am working on is to build very highly linked JS 
applications, that can walk through linked data from one site to another
and build UIs out of the data. You can read more about that in either the 
SoLiD draft https://github.com/linkeddata/SoLiD
With more information here, videos and talks
 https://github.com/read-write-web/wiki/wiki
There is a video here that might help
 https://www.youtube.com/watch?v=z0_XaJ97rF0

So it is related to X-Windows in so far as it is UI centric. But it is a hyperdata
windowing system if you want, where links are the core of the system.

This need not be the only use of SEARCH or GET + Query ( whichever gets
the most support from this community ) of course, but it is one that
puts the web stack at the center of the UI experience.

And this makes it impossible in these apps to have SOAP like query 
resources that would replace the primacy of GET and hyper links.

Henry

Social Web Architect
http://bblfish.net/