Subject: Re: Proposed HTTP SEARCH method update

From: Phil Hunt <[email-masked]>
Date: Fri, 1 May 2015 09:12:56 -0700

One of the use cases I am having difficulty is what I will call “MATCH”.  This is a bit different from doing a system wide search as much of the discussion has focused on.  In this case, we want to test if a certain filter condition is true for a given resource and optionally return some attributes (or nothing at all).

We talked about this in developing the SCIM protocol.  We decided to go with the POST option since a SEARCH method would be long term (C below).

I’ve looked at a number of approaches on this case:

A. A GET is inappropriate because the filter condition cannot be passed on the URL.  It would be ok if it could be passed in the BODY, but I hear that’s bad.

B. A POST is problematic because it should be to create or replace the resource.  How does the server detect the intended operation?

C. A POST + a path extension (e.g., “/.search” giving   https://example.com/Users/<id>/.search) is a work around works, but seems kludgy.  Access control systems have to know that https://example.com/Users/<id>/.search is the same resource as https://example.com/Users/<id>

D. A POST to /search can also work, but makes for a more complex operation as now we are not talking about an operation based on a URL.  It also may complicate access control since the resource being queried isn’t part of the URL.

I recall talking to Julian about if-match headers…I can’t remember the pros/cons we talked about. But I do think that the semantics shouldn’t change. I can see adding if-match conditions to a SEARCH command as being useful. I think overloading these might be more complexity.

I think this is a case that more clearly shows why “SEARCH” is useful. 

Phil

@independentid
www.independentid.com
[email-masked]

---

From: ashok malhotra <[email-masked]>
Date: Fri, 01 May 2015 19:13:32 -0400

Thanks, Phil, good example.
I think you are talking about a kind of existential query such as:
"Are there any Red items?"

I had not thought of the access control issue but that, clearly, is
significant.

 > I think this is a case that more clearly shows why “SEARCH” is useful.

Others, please send additional use cases for SEARCH/QUERY so we can
strengthen the argument.  I get many anecdotal requests for SEARCH
but documented usecases would be a big help.
-- 
All the best, Ashok

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 10 May 2015 14:04:44 +0200

> On 1 May 2015, at 18:12, Phil Hunt <[email-masked] <mailto:[email-masked]>> wrote:
> 

yep .

> A. A GET is inappropriate because the filter condition cannot be passed on the URL.  It would be ok if it could be passed in the BODY, but I hear that’s bad.

There are two possibilities here, both allowed by the current spec.
 1) GET with Query: and Query-Type: header ( the second one giving the mime type of the header
 2) GET with Content-Type header and body.

There have been no criticisms of (1), only of (2) . 
(2) does not go against any of the more recent specs - it is even allowed, and probably works for most servers. It should be left open for the longer term after studies if it really is problematic and in what circumstances. But 1) is a good fallback position.

The fallback position of both is that if the server does not understand the headers or body, it returns the value it would have returned on a plain GET. This actually works: you can try it on any web server to see :-)

The main problem with a GET is when the resource is to all intents and purposes infinitely long and one really never wants to do a GET on the resource. In that case it is important to be able to run SEARCH ( or some other verb such as QUERY). Could it be that some resource just offer SEARCH/QUERY and not GET in that case?

> C. A POST + a path extension (e.g., “/.search” giving   https://example.com/Users/<id>/.search <https://example.com/Users/%3Cid%3E/.search>) is a work around works, but seems kludgy.  Access control systems have to know that https://example.com/Users/<id>/.search <https://example.com/Users/%3Cid%3E/.search> is the same resource as https://example.com/Users/<id> <https://example.com/Users/%3Cid%3E>
> 
> www.independentid.com <http://www.independentid.com/>[email-masked] <mailto:[email-masked]>

---

From: Henry Story <[email-masked]>
Date: Sun, 10 May 2015 12:03:36 +0000

yep .

---

From: Phil Hunt <[email-masked]>
Date: Mon, 11 May 2015 14:16:40 -0700

Henry,

Authors of the HTTP specs are advising passing a body with GET is a very bad idea. This is mostly for historical reasons.

Julian Reschke also felt that the use of headers for queries was also ill-advised given that a lot of firewalls may manipulate or block headers. The concern is unanticipated consequences.  E.g. the nature of the GET changes because the header or the body is missing and the server mistakenly releases information because the operation becomes treated as a plain get.  E.g. the return of a resource was to be subject to some filter condition and the client is fooled into thinking the condition was true.

There are also (whether founded or not) concerns about space limitations in headers and URLs. Though I have to wonder, if your filter is that long, should the filter be processed?  E.g. why pass a public key when you can pass its thumbprint.

By using a new HTTP method, then the failure due to lack of support for the method becomes guaranteed. A server that does not support or does not get the request (because of a firewall) will not unintentionally continue processing the request.  

Phil

@independentid
www.independentid.com
[email-masked]

> On May 10, 2015, at 5:04 AM, [email-masked] wrote:
> 
>

---

From: James M Snell <[email-masked]>
Date: Mon, 11 May 2015 16:04:31 -0700

Thank you Phil. Great way to explain.
On May 11, 2015 2:21 PM, "Phil Hunt" <[email-masked]> wrote:

> Henry,
>
> Authors of the HTTP specs are advising passing a body with GET is a very
> bad idea. This is mostly for historical reasons.
>
> Julian Reschke also felt that the use of headers for queries was also
> ill-advised given that a lot of firewalls may manipulate or block headers.
> The concern is unanticipated consequences.  E.g. the nature of the GET
> changes because the header or the body is missing and the server mistakenly
> releases information because the operation becomes treated as a plain get.
> E.g. the return of a resource was to be subject to some filter condition
> and the client is fooled into thinking the condition was true.
>
> There are also (whether founded or not) concerns about space limitations
> in headers and URLs. Though I have to wonder, if your filter is that long,
> should the filter be processed?  E.g. why pass a public key when you can
> pass its thumbprint.
>
> By using a new HTTP method, then the failure due to lack of support for
> the method becomes guaranteed. A server that does not support or does not
> get the request (because of a firewall) will not unintentionally continue
> processing the request.
>
> Phil
>
> @independentid
> www.independentid.com
> [email-masked]
>
> On 1 May 2015, at 18:12, Phil Hunt <[email-masked]> wrote:
>
> One of the use cases I am having difficulty is what I will call “MATCH”.
> This is a bit different from doing a system wide search as much of the
> discussion has focused on.  In this case, we want to test if a certain
> filter condition is true for a given resource and optionally return some
> attributes (or nothing at all).
>
>
> yep .
>
>
> We talked about this in developing the SCIM protocol.  We decided to go
> with the POST option since a SEARCH method would be long term (C below).
>
> A. A GET is inappropriate because the filter condition cannot be passed on
> the URL.  It would be ok if it could be passed in the BODY, but I hear
> that’s bad.
>
>
> (2) does not go against any of the more recent specs - it is even allowed,
> and probably works for most servers. It should be left open for the longer
> term after studies if it really is problematic and in what circumstances.
> But 1) is a good fallback position.
>
> The fallback position of both is that if the server does not understand
> the headers or body, it returns the value it would have returned on a plain
> GET. This actually works: you can try it on any web server to see :-)
>
> The main problem with a GET is when the resource is to all intents and
> purposes infinitely long and one really never wants to do a GET on the
> resource. In that case it is important to be able to run SEARCH ( or some
> other verb such as QUERY). Could it be that some resource just offer
> SEARCH/QUERY and not GET in that case?
>
>
> B. A POST is problematic because it should be to create or replace the
> resource.  How does the server detect the intended operation?
>
> C. A POST + a path extension (e.g., “/.search” giving
> https://example.com/Users/<id>/.search) is a work around works, but seems
> kludgy.  Access control systems have to know that
> https://example.com/Users/<id>/.search is the same resource as
> https://example.com/Users/<id>
>
> D. A POST to /search can also work, but makes for a more complex operation
> as now we are not talking about an operation based on a URL.  It also may
> complicate access control since the resource being queried isn’t part of
> the URL.
>
> I recall talking to Julian about if-match headers…I can’t remember the
> pros/cons we talked about. But I do think that the semantics shouldn’t
> change. I can see adding if-match conditions to a SEARCH command as being
> useful. I think overloading these might be more complexity.
>

---

From: Zhong Yu <[email-masked]>
Date: Mon, 11 May 2015 18:25:49 -0500

>  2) GET with Content-Type header and body.

I don't think you'll have *any* luck trying to modify the semantics of
GET. Your best bet is still to define a new method.

Sure, the server and the client can construct any well-formed message
and assign whatever meaning to it. It'll work most of the times. You
can even construct ill-formed messages. Heck, why use HTTP any all:)

But the question is, do others agree to this semantics too, and do you
care about their existence. Regardless of what the spec says, GET has
a widely accepted and understood semantics, any deviation from it can
and will cause interoperability problems with some deployments.

For the interest of this working group, we can nevertheless discuss
the wording of the spec with regard to this use case. The following
statement in RFC7231

> A payload within a GET request message has no defined semantics

is broad and vague, how do we apply it here? I would rather take an
extreme interpretation - A payload in a GET request is completely
meaningless, it can be dropped without affecting the semantics of the
request; therefore we should not use it for any semantics purposes if
any 3rd party (besides client and server) is involved.

Zhong Yu
bayou.io

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 17 May 2015 10:37:05 +0200

> On 11 May 2015, at 23:16, Phil Hunt <[email-masked]> wrote:
> 
> Henry,
> 

This is not an issue for the proposal. The semantics I am proposing allows the server to ignore the range request and the body without harm to the request.

That can be addressed the same way HTTP 1.1 (RFC 2616 ) addresses partial GET.
In §9.3

[[
   The semantics of the GET method change to a "partial GET" if the
   request message includes a Range header field. A partial GET requests
   that only part of the entity be transferred, as described in 
   section 14.35. The partial GET method is intended to reduce unnecessary
   network usage by allowing partially-retrieved entities to be
   completed without transferring data already held by the client.
]]

So we need to include a Range header field or equivalent.
And is written clearly in §14.35.2 "A server MAY ignore the Range header. "
In the same section we also see that:

[[
   If the server supports the Range header and the specified range or
   ranges are appropriate for the entity:

      - The presence of a Range header in an unconditional GET modifies
        what is returned if the GET is otherwise successful. In other
        words, the response carries a status code of 206 (Partial
        Content) instead of 200 (OK).

      - The presence of a Range header in a conditional GET (a request
        using one or both of If-Modified-Since and If-None-Match, or
        one or both of If-Unmodified-Since and If-Match) modifies what
        is returned if the GET is otherwise successful and the
        condition is true. It does not affect the 304 (Not Modified)
        response returned if the conditional is false.

   In some cases, it might be more appropriate to use the If-Range
   header (see section 14.27) in addition to the Range header.
]]

What I am proposing is intended to be consistent with this.
I am proposing to be guided by the same constraints imposed by HTTP here:

A request with a body that is ignored should return the full content - and
it does on all correctly implemented servers. You can try it :-)

For servers that understand the query language range request, that still needs
to be defined more clearly, the server would also return a 206.

This has the advantage of imposing very RESTful constraints on the 
query language. It makes clear that the relation between the GET with the
query and the normal GET is just that the query is a partial representation
of the first. 

What I am proposing is just an extension of partial content, with a Query 
language to be able to make more precise selections in the content.

I think this can satisfy a very large number of use cases for which the 
SEARCH verb was proposed.

> There are also (whether founded or not) concerns about space limitations in headers and URLs. Though I have to wonder, if your filter is that long, should the filter be processed?  E.g. why pass a…

passing the query in the header was a fallback position, over putting one in the body.

> By using a new HTTP method, then the failure due to lack of support for the method becomes guaranteed. A server that does not support or does not get the request (because of a firewall) will not un…

The HTTP GET query proposal is not exclusive of a good SEARCH or QUERY verb proposal.

For example SEARCH/QUERY would still useful in case the server wants to make clear that the resource is too big to do a GET on, and that only partial content queries should be used.
Here the server has a good reason to make failure of a GET the default.

>>> I think this is a case that more clearly shows why “SEARCH” is useful. 

Social Web Architect
http://bblfish.net/

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 17 May 2015 11:13:56 +0200

> On 12 May 2015, at 01:25, Zhong Yu <[email-masked]> wrote:
> 

That is good, because we don't want to modify the semantics of GET, 
just extend the types of range requests allowed by HTTP 1.1 to 
not just allow byte ranges but more semanitcally rich partical content
requests.

I note that RFC7233 on Range Requests allows these to be extended

[[
2.2.  Other Range Units

   Range units are intended to be extensible.  New range units ought to
   be registered with IANA, as defined in Section 5.1.

     other-range-unit = token
]]

> and will cause interoperability problems with some deployments.

yes, and there is no intention to change it.

> any 3rd party (besides client and server) is involved.

It says very precisely that there are "no defined semantics", which is
not the same as "never can have a defined semantics".  This is good
protocol design: don't close doors you don't need to. 

Furthermore if the server drops the body, everything functions exactly
right, just as it does when a server ignores the range request headers.

Henry

Social Web Architect
http://bblfish.net/

---

From: Phil Hunt <[email-masked]>
Date: Sun, 17 May 2015 08:46:44 -0700

The outcomes are somewhat different or at least critical. In this case, if a header or body is stripped, bad things happen. Neither the server or the client are aware. The servers default action is to process the get (now plain). 

For example if a proxy strips the header, the server never applies the filter and the client assumes everything returned is a match. 

This would be bad if the search filter was a security filter for example. Does resource henry123 have scope=update or role=admin? 

Phil

> On May 17, 2015, at 01:37, "[email-masked]" <[email-masked]> wrote:
> 
>

---

From: "[email-masked]" <[email-masked]>
Date: Wed, 20 May 2015 21:45:14 +0200

> On 17 May 2015, at 17:46, Phil Hunt <[email-masked]> wrote:
> 

In the GET + query-body-(or-header) proposal, the intermediary would have to also drop the status code. It would for example have to change the response code from 206 to 200 or vice versa.

If this is a criticism of this proposal, how is this not also a criticism of 
Range Requests  RFC 7233 ? (  http://tools.ietf.org/html/rfc7233 )

> This would be bad if the search filter was a security filter for example. Does resource henry123 have scope=update or role=admin? 

I am not sure I follow you here. We have explored declarative access control rules  by having each resource have a Link header to an acl resource. You can read up the current work here:
 
  https://www.w3.org/wiki/WebAccessControl

The ACL associated with a resource should be in a different resource. Mixing data and metadata on a resource does not seem to be a good idea. 

Social Web Architect
http://bblfish.net/