Subject: NEW PREFERENCE - return=query-result

From: Ning Dong <[email-masked]>
Date: Thu, 10 Sep 2015 21:49:07 +0000

Hi,
	Could you please review the request of adding a new value for return prefer header?

	The new value is "query-result", which is used when creating a query definition resource with POST or PUT request.
	The client would like the server to create the query definition resource, but also execute the query and return the query result.
	For example,
	POST /employees/searches HTTP/1.1
	Host: example.org
	Content-Type: application/json
	Prefer: return=query-result
	
	{
		"q": "name eq foo",
		"fields": ["name","age","startdate"],
		"orderBy": ["name","age:desc"]
	}

	This above resource defines a query (equivalent to select name, age, startdate from employees where employees.name='foo' order by name, age desc).
	Without the Prefer: return=query-result header, the server would just create a new resource and return a 201 response.
	If server honors the prefer header, then the server will not only create a new resource, but also execute the query based on the query definition.
	The response body will contain the result of the query execution, such as:
	201 Created
	Preference-Applied: return=query-result
	Location: http://example.com/employees/searches/q1
	Content-Location: http://example.com/employees/searches/q1/result

	{
	  "items": [
		{"name": "foo",
		 "age": 35,
		 "startdate": "2008-02-15"}
 	  ]
	}

 o  Preference: return

  o  Value: query-result

  o  Optional Parameters: n/a

  o  Description: It is used to indicate that result of the query execution is preferred in the response.

  o  Reference: Oracle will add a new sub type (type=query-def) in application/vnd.oracle.resource+json media type. This new sub type uses return=query-result prefer header.
     The application/vnd.oracle.resource+json media type is defined at:
     http://www.oracle.com/webfolder/technetwork/tutorials/appdevinfo/New%20REST%20Media%20Type.pdf

  o  Notes: It is related to another request to add "transient" prefer header.

	Thanks and appreciate your review.

Ning

---

From: Phil Hunt <[email-masked]>
Date: Fri, 11 Sep 2015 08:56:24 -0700

This is difficult. Two conventions are in collision.  The definition of post to create a resource and the expectation that a query returns a result. 

I would maybe tip the scales in favor of what most Oracle apis would need as a default. 

Would it be true that clients want to create stored searches by default?

Phil

> On Sep 10, 2015, at 14:49, Ning Dong <[email-masked]> wrote:
> 
> Hi,

---

From: Ning Dong <[email-masked]>
Date: Fri, 11 Sep 2015 16:28:11 +0000

Phil,
   I think it is hard to say. I think most cases client probably doesn't want to create a stored search. They just want to execute the query.

   What do you suggest to make the proposal more general and useful for other use cases? We have a very specific use case and very specific way of solving the issue. It might not fly for other people, I guess.

   Thanks.

Ning

---

From: James M Snell <[email-masked]>
Date: Fri, 11 Sep 2015 10:10:53 -0700

To be honest, I'm entirely  -1 on a preference for query-result. If
you want something like this, use PUT or POST to create the stored
query, then create a new resource that you can either use GET or
SEARCH (http://tools.ietf.org/html/draft-snell-search-method-00) on.
In my opinion, `query-result` would entirely be an abuse of the
preference mechanism.

---

From: Ning Dong <[email-masked]>
Date: Fri, 11 Sep 2015 17:34:58 +0000

Yes, that is right. We'll need two requests though: one to create the query definition resource, and one to get the result.

Thanks.

Ning

---

From: Phil Hunt <[email-masked]>
Date: Fri, 11 Sep 2015 13:04:24 -0700

Agree. If we had search then I would say post should create a resource by default. 

But without search I think many are stuck with using post to search given concerns about passing PII and other restricted info on url with GET. 

Phil

> On Sep 11, 2015, at 10:10, James M Snell <[email-masked]> wrote:
>

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 12 Sep 2015 07:37:01 +1000

If a resource can be coded to have two behaviours (and branch on the
'prefer' header), why not just code two resources?

It seems much cleaner to me to have one URI that always stores a query and
returns 201, and another that always executes immediately and returns 200,
when POSTed to.

They're both valid uses of POST.
On 12/09/2015 6:08 AM, "Phil Hunt" <[email-masked]> wrote:

> But without search I think many are stuck with using post to search given
> concerns about passing PII and other restricted info on url with GET.
>
> Phil
>
> >> This is difficult. Two conventions are in collision.  The definition of
> post to create a resource and the expectation that a query returns a result.
> >>
> >>>   The new value is "query-result", which is used when creating a query
> definition resource with POST or PUT request.
> >>>   The client would like the server to create the query definition
> resource, but also execute the query and return the query result.
> >>>   For example,
> >>>   This above resource defines a query (equivalent to select name, age,
> startdate from employees where employees.name='foo' order by name, age
> desc).
> >>>   Without the Prefer: return=query-result header, the server would
> just create a new resource and return a 201 response.
> >>>   If server honors the prefer header, then the server will not only
> create a new resource, but also execute the query based on the query
> definition.
> >>> o  Reference: Oracle will add a new sub type (type=query-def) in
> application/vnd.oracle.resource+json media type. This new sub type uses
> return=query-result prefer header.

---

From: Phil Hunt <[email-masked]>
Date: Fri, 11 Sep 2015 14:54:01 -0700

If you mean have multiple endpoints I agree. 

Another possibility that has been suggested to me is using content-type to signal processing behavior. But this might be a good example as to why it is not the best. It strikes me as more confusing when the content is the same but using content type to signal processing instructions. It then gets complex if you want to signal different permutations and combinations of processing behaviors. 

"post creates a resource" becomes nice and consistent if we move reporting processing functions out of POST(things that do not change state) into a new method like SEARCH. 

Phil

> On Sep 11, 2015, at 14:37, Matthew Kerwin <[email-masked]> wrote:
> 
> If a resource can be coded to have two behaviours (and branch on the 'prefer' header), why not just code two resources?
> 
> It seems much cleaner to me to have one URI that always stores a query and returns 201, and another that always executes immediately and returns 200, when POSTed to.
>

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 12 Sep 2015 10:06:41 +1000

> If you mean have multiple endpoints I agree.

Yep, that was what I meant.

[...]

>
> "post creates a resource" becomes nice and consistent if we move
reporting processing functions out of POST(things that do not change state)
into a new method like SEARCH.

If we say "post modifies a resource" (including append, delete, etc.) then
I half agree; but POST has been fuzzy for a long time, and no amount of
decree from us will change people's behaviour. (I.e. many people using POST
for SEARCH will continue to do so.)

If SEARCH is to be the unambiguous read-only-POST, you'd need an
unambiguous writing-POST method too. That way things like response status
codes could be clearly reasoned for each (especially things like 200, 303,
maybe 404.)

That said, I have no problem with a fuzzy POST, be it mixing POST and
SEARCH, or using a different POSTed-to endpoint for each action (including
'?action=...' query parameters), or embedding action instructions in the
POST body (SOAP anyone?).

The prefer(ence-applied) mechanism seems like a convenient fourth channel,
but I'm with James that it's not the right use of 'prefer.'

Cheers

---

From: Phil Hunt <[email-masked]>
Date: Fri, 11 Sep 2015 17:59:42 -0700

I agree past usage becomes a huge barrier. 

I think though that for those designing new protocols/apis on top of http, the model they can produce is arguably simpler without having to mix new channels and  re-appropriating content type to signal functionality etc.

I could go either way on an update only method. We already have put and patch. For that matter there is already lots of overlap between post and patch and it doesn't seem to be a problem. 

I also like the idea of service provider gateways/balancers being able to distinguish read vs write via the method as it can be used to enhance scale in some cases. E.g.,by routing read transactions to different servers. 

Phil

> On Sep 11, 2015, at 17:06, Matthew Kerwin <[email-masked]> wrote:
> 
> 
> If we say "post modifies a resource" (including append, delete, etc.) then I half agree; but POST has been fuzzy for a long time, and no amount of decree from us will change people's behaviour. (I.e. many people using POST for SEARCH will continue to do so.)
> 
> If SEARCH is to be the unambiguous read-only-POST, you'd need an unambiguous writing-POST method too. That way things like response status codes could be clearly reasoned for each (especially things like 200, 303, maybe 404.)
> 
> That said, I have no problem with a fuzzy POST, be it mixing POST and SEARCH, or using a different POSTed-to endpoint for each action (including '?action=...' query parameters), or embedding action instructions in the POST body (SOAP anyone?).
> 
> The prefer(ence-applied) mechanism seems like a convenient fourth channel, but I'm with James that it's not the right use of 'prefer.'
> 
> Cheers

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 12 Sep 2015 13:06:24 +1000

On 12/09/2015 9:14 AM, "Ning Dong" <[email-masked]> wrote:
>
> I submitted two requests, and both of them are related to search. If we
have the SEARCH verb, then we don't have this problem.
>
> For the case where client wants to execute an ad hoc query against a
collection, [...]
> We don't want to save the query definition though since it won't be
useful. We really just want the server to use the query definition in the
payload, execute query and return the result.
>
> A second use case is a query definition might be reused later, but we
want the query to be executed when the query definition resource is created
to save a server round trip. That is the return=query-result proposal. This
use case is achievable with two requests: one to create the query
definition resource, and another to execute the query.
> Ning

​The problem I'm having is that you're mixing things up strangely. The
endpoint definition seems to be: POST here to store a procedure. On top of
that you layer a preference for "also execute it and return the result."
Ok, I can understand that, and maybe 'query-result' is different enough
from 'representation' to make sense.

But with 'transient' you seem to instead be saying: the endpoint definition
is to store a query, and immediately execute it and return the result. On
top of that you're layering a preference to not store the query? It seems
quite convoluted, or at least not well engineered, and smells a bit like
server-side caching of requests.

If I'm reading it wrong, and it's actually the *same* endpoint (i.e. store
the query, and maybe return an immediate result) then 'transient' isn't a
preference, it's an instruction to alter the base behaviour of the endpoint
-- an instruction which should be signalled, not with a preference, but
with a mechanism like those I mentioned earlier.

Cheers

---

From: James M Snell <[email-masked]>
Date: Mon, 14 Sep 2015 11:01:42 -0700

The "transient" preference looks just fine to me.

On Mon, Sep 14, 2015 at 10:59 AM, Ning Dong <[email-masked]> wrote:
> Matt,
>     Thanks for the comments.
>
>     The proposed "transient" preference is a preference in the sense that
> the server still can ignore it. In that case, the server would just create
> the query definition. The client side will do the following:
>     1. POST /searches (with body=query definition)
>         Server returns something like:
>         201 CREATED
>         Location: /searches/foobar
>     2. GET /searches/foobar/result
>         to get the query executed and return the query result
>     3. DELETE /searches/foobar
>         A responsible client needs to delete the resource since it won't be
> used any more.
>
>     I agree that the "transient" preference alters the server behavior. I
> think any preference header alters the server behavior in some way but I
> think your concern is that this alters the most basic behavior of the server
> (the basic part is creating the resource).
>
>     We have been exploring different options of supporting the adhoc query
> in a "standard" way. We could do the 3 steps way, but it is a lot of
> overhead.
>     We could enhance out specific media type and include "transient"
> information inside the query definition itself in the payload. It works only
> for our query definitions and won't work for other media types.
>     (If we had SEARCH verb, it would solve our problem, but it has not been
> approved yet.)
>
>     I have been thinking whether this can be make more generic and applied
> to broader scope instead of this narrow use case. Yes it is kind of related
> to caching. Following that line of thought, would it make sense to do
> something like?
>     Prefer: time-to-live=0
>     or
>     Prefer: time-to-live=short (we can have other values like medium, long)
>
>     BTW, this is the first time for me to send a request and discuss
> something in this mailing list. Please let me know if there is a better
> communication channel so that I don't pollute everybody's mailbox.
>
>     Thanks a lot.
>
> Ning
>
>
> On 9/11/2015 8:06 PM, Matthew Kerwin wrote:
>
>
>> A second use case is a query definition might be reused later, but we want
>> the query to be executed when the query definition resource is created to
>> save a server round trip. That is the return=query-result proposal. This use
>> case is achievable with two requests: one to create the query definition
>> resource, and another to execute the query.
>>
>> Thanks.
>>
>> Ning
>>
>
> The problem I'm having is that you're mixing things up strangely. The
> that you layer a preference for "also execute it and return the result." Ok,
> I can understand that, and maybe 'query-result' is different enough from
> 'representation' to make sense.
> -- an instruction which should be signalled, not with a preference, but with
> a mechanism like those I mentioned earlier.
> Cheers
>
>

---

From: Ning Dong <[email-masked]>
Date: Mon, 14 Sep 2015 17:59:55 +0000

Matt,
    Thanks for the comments.

    The proposed "transient" preference is a preference in the sense that the server still can ignore it. In that case, the server would just create the query definition. The client side will do the following:
    1. POST /searches (with body=query definition)
        Server returns something like:
        201 CREATED
        Location: /searches/foobar
    2. GET /searches/foobar/result
        to get the query executed and return the query result
    3. DELETE /searches/foobar
        A responsible client needs to delete the resource since it won't be used any more.

    I agree that the "transient" preference alters the server behavior. I think any preference header alters the server behavior in some way but I think your concern is that this alters the most basic behavior of the server (the basic part is creating the resource). 

    We have been exploring different options of supporting the adhoc query in a "standard" way. We could do the 3 steps way, but it is a lot of overhead. 
    We could enhance out specific media type and include "transient" information inside the query definition itself in the payload. It works only for our query definitions and won't work for other media types.
    (If we had SEARCH verb, it would solve our problem, but it has not been approved yet.)

    I have been thinking whether this can be make more generic and applied to broader scope instead of this narrow use case. Yes it is kind of related to caching. Following that line of thought, would it make sense to do something like?
    Prefer: time-to-live=0
    or
    Prefer: time-to-live=short (we can have other values like medium, long)

    BTW, this is the first time for me to send a request and discuss something in this mailing list. Please let me know if there is a better communication channel so that I don't pollute everybody's mailbox. 

    Thanks a lot.

Ning

On 9/11/2015 8:06 PM, Matthew Kerwin wrote:
> 
> On 12/09/2015 9:14 AM, "Ning Dong" < <mailto:[email-masked]>[email-masked] <mailto:[email-masked]>> wrote:
> >
> > I submitted two requests, and both of them are related to search. If we have the SEARCH verb, then we don't have this problem.
> >
> > For the case where client wants to execute an ad hoc query against a collection, [...]
> > We don't want to save the query definition though since it won't be useful. We really just want the server to use the query definition in the payload, execute query and return the result.
> >
> > A second use case is a query definition might be reused later, but we want the query to be executed when the query definition resource is created to save a server round trip. That is the return=query-result proposal. This use case is achievable with two requests: one to create the query definition resource, and another to execute the query.
> >
> > Thanks.
> >
> > Ning
> >
> 
> ​ The problem I'm having is that you're mixing things up strangely. The endpoint definition seems to be: POST here to store a procedure. On top of that you layer a preference for "also execute it and return the result." Ok, I can understand that, and maybe 'query-result' is different enough from 'representation' to make sense. But with 'transient' you seem to instead be saying: the endpoint definition is to store a query, and immediately execute it and return the result. On top of that you're layering a preference to not store the query? It seems quite convoluted, or at least not well engineered, and smells a bit like server-side caching of requests. If I'm reading it wrong, and it's actually the *same* endpoint (i.e. store the query, and maybe return an immediate result) then 'transient' isn't a preference, it's an instruction to alter the base behaviour of the endpoint -- an instruction which should be signalled, not with a preference, but with a mechanism like those I mentioned earlier. Cheers