Subject: How to express no matching results in HTTP SEARH method?

From: Sawood Alam <[email-masked]>
Date: Tue, 3 Nov 2020 21:40:17 -0500

Hi,

What would be an appropriate status code if the target resource of the
SEARCH method wants to provide an entity to convey that the search
operation yielded no results? In traditional GET-style searching, one could
return a 404, would that still be an option (I doubt it as it would mean
the target resource itself that performs the search is not found)?

The specification seems to suggest usage of 204, but in that case no
content can be returned in the response, which may be desired in some
cases. Also, 204 is a success response, which means a client might repeat
the request, while a 4xx response would suggest that there is no point in
repeating the request without making changes to it.

Best,

--
Sawood Alam

---

From: Julian Reschke <[email-masked]>
Date: Wed, 4 Nov 2020 13:59:09 +0100

Am 04.11.2020 um 03:40 schrieb Sawood Alam:
> operation yielded no results? In traditional GET-style searching, one
> could return a 404, would that still be an option (I doubt it as it
> would mean the target resource itself that performs the search is not
> found)?

That would be my interpretation as well.

> content can be returned in the response, which may be desired in some

...in which case, you wouldn't have to use a 204.

> cases. Also, 204 is a success response, which means a client might
> repeat the request, while a 4xx response would suggest that there is no
> point in repeating the request without making changes to it.

If the request target exists, and accepts and processes the query, the
response ought to be a 2xx, no?

Best regards, Julian

---

From: Sawood Alam <[email-masked]>
Date: Wed, 4 Nov 2020 08:24:59 -0500

> response ought to be a 2xx, no?

The question remains open, "how to express a successful SEARCH response
with an unsuccessful search result when returning an entity body is
desired?" Does it warrant a new 2xx status code? I think this will be a
common situation for many people who would be willing to migrate their
GET-style searching where they could use a 404 to express it. Some
discussion and guidelines in the spec on this matter would be useful for a
more coherent adoption.

Best,

--
Sawood Alam

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 04 Nov 2020 13:24:57 +0000

--------

> response ought to be a 2xx, no?

Any librarian will tell you that "Found" is much simpler than
"Not found", which has a lot of possible semantics:

	Not found, though it should have been.

	Not found, and will never be found.

	Not found, but will certainly be found sometime in the future [($time)].

	Not found, and cannot possibly be found until ($time)

	Not found here, but try asking ($uri, [$query_params])

	Not even looking, you have been asking too many queries recently

	Not telling, (temporary/permanent/authorization needed)

	Found, but you cant have it (temporary/permanent/authorization needed)

	...

It might be a good idea to hash out a couple of general mechanisms
in the spec, to provide mechanisms for traffic/load-engineering,
at the very least something like "Dont-repeat-this-SEARCH: <seconds>" ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 4 Nov 2020 14:30:29 +0100

Am 04.11.2020 um 14:24 schrieb Sawood Alam:
> ... > The question remains open, "how to express a successful SEARCH response

"unsuccessful" means "no results", right?

> discussion and guidelines in the spec on this matter would be useful for
> a more coherent adoption.
> ...

I don't see why this needs a new status code.

You can use both 204 and 200, in the latter case, the media type of the
response describes how it is to be interpreted.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 4 Nov 2020 14:36:19 +0100

> at the very least something like "Dont-repeat-this-SEARCH: <seconds>" ?

That's all very interesting, but why is this relevant now, but not for
existing uses of POST we want to replace?

(also: <https://greenbytes.de/tech/webdav/rfc6585.html#status-429>)

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 04 Nov 2020 13:56:22 +0000

--------

> existing uses of POST we want to replace?

Because this is the century of the fruitbat, and we're trying to do a better
job than back in the dark ages where things were just slapped together ?

Because SEARCH has much narrower and therefore more manageable semantics
than POST, which can literally be and do anything ?

---

From: Julian Reschke <[email-masked]>
Date: Wed, 4 Nov 2020 15:18:55 +0100

> than POST, which can literally be and do anything ?

My understanding is that we essentially want to define something that is
the same as POST, with the additional property of being safe by
definition. Nothing more specific than that.

Best regards, Julian

---

From: James M Snell <[email-masked]>
Date: Wed, 4 Nov 2020 06:35:47 -0800

That certainly has been the intent to this point, yes, and it's what
I'd like to see. Anything else can be done as a separate piece of
work.

For instance, for something like a "don't repeat this search" for
N-seconds, one can make use of 425 Too Early responses, or 429 Too
Many Requests responses with a Retry-After header. There's no reason
that existing mechanisms would not work, or to think that something
specific to SEARCH needs to be hashed out.

---

From: Roberto Polli <[email-masked]>
Date: Wed, 4 Nov 2020 16:12:25 +0100

Hi Julian,

Il giorno mer 4 nov 2020 alle ore 15:22 Julian Reschke
<[email-masked]> ha scritto:
> ... we essentially want to define something that is

Q:  POST is cacheable (under conditions). I was curious why SEARCH is
not cacheable: this can be useful...

R.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 4 Nov 2020 17:09:03 +0100

> not cacheable: this can be useful...

Cacheability is an open TODO. I agree that it's supposed to be at least
as cacheable as POST :-).

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 04 Nov 2020 16:14:41 +0000

--------
> as cacheable as POST :-).

Please specify that SEARCH results must *always* have a Cache-Control.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 4 Nov 2020 17:47:00 +0100

> Please specify that SEARCH results must *always* have a Cache-Control.

Or better, that cacheable results must have it. The rationale here is
that as usual, all those who don't care will forget to put it and will
have their search working fine, and things will break when inserting a
cache. Usual stuff "your reverse-proxy doesn't support my server".
Better be clear from the start that it's OK not to have the header but
then one must not expect caching. Because you'll have to deal with it
anyway.

Willy

---

From: Austin Wright <[email-masked]>
Date: Wed, 4 Nov 2020 13:49:49 -0700

> On Nov 3, 2020, at 19:40, Sawood Alam <[email-masked]> wrote:
> 
> Hi,
> 
> What would be an appropriate status code if the target resource of the SEARCH method wants to provide an entity to convey that the search operation yielded no results? In traditional GET-style searching, one could return a 404, would that still be an option (I doubt it as it would mean the target resource itself that performs the search is not found)?

A search result with zero results is still a resource that exists, so the status code should be 200. This is also true for `GET /search?q={query}`.

If the client expects at least one result, but none are returned, it doesn’t need a status code to figure out something’s up.

This is different from looking up a single resource by its URI. If an HTML form says to use `GET /item?id={id}` to look up a product, the response will be the actual product itself (not a query result), so either it exists at that URI (200), or it doesn’t (404).

It’s the difference between saying “Make me a list of all the books on the second shelf” (an empty list is still a list, 200) and “Get me the first book on the second shelf” (That’s an error if there’s no book there, 404).

> 
> The specification seems to suggest usage of 204, but in that case no content can be returned in the response, which may be desired in some cases. Also, 204 is a success response, which means a client might repeat the request, while a 4xx response would suggest that there is no point in repeating the request without making changes to it.

A 204 is identical to sending a 200 response with Content-Length: 0

If you want to suggest that the response is not going to change in the near future, use caching headers.

> Best,

Cheers,

Austin.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 4 Nov 2020 23:30:39 +0100

> A 204 is identical to sending a 200 response with Content-Length: 0

?

https://tools.ietf.org/html/rfc7231#section-6.3.5

And it’s not like the spec is confabulating here:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/204

Grüße, Carsten

---

From: Austin Wright <[email-masked]>
Date: Wed, 4 Nov 2020 18:53:49 -0700

> On Nov 4, 2020, at 15:30, Carsten Bormann <[email-masked]> wrote:
> 

What’s the question, exactly? 

Maybe I’m being unclear. In responses without a Content-Length or Transfer-Encoding header, normally the response message only ends when the TCP connection is terminated. The exception is a 204; the next response begins immediately when the headers end.

Cheers,

Austin.

---

From: Henry Story <[email-masked]>
Date: Thu, 5 Nov 2020 04:20:31 +0100

> On 4 Nov 2020, at 14:56, Poul-Henning Kamp <[email-masked]> wrote:
> 
> --------
> Julian Reschke writes:

Structurally I think the following is true:

- POST creates a new resource - so it can have consequences like filling 
    a shopping cart or enrolling in the army
- GET fetches a representation without the client being bound by anything more
   than having seen the result. 

SEARCH (when restricted to one resource) is just a more  efficient GET,
just like PATCH is a more efficient PUT.

Henry

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 5 Nov 2020 04:48:50 +0100

On Wed, Nov 04, 2020 at 06:53:49PM -0700, Austin Wright wrote:
> > And it's not like the spec is confabulating here:
> > 
> What's the question, exactly? 
> 
> Maybe I'm being unclear. In responses without a Content-Length or
> Transfer-Encoding header, normally the response message only ends when the
> TCP connection is terminated. The exception is a 204; the next response
> begins immediately when the headers end.

Absolutely. 204 is a 200 with no payload. Its use is mainly for methods
which don't expect a response, but like any 2xx it can be simplified to
the equivalent 200 by agents which don't care about specific statuses,
thus it simplifies as a 200 with no payload.

Willy

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 5 Nov 2020 12:06:35 +0100

On Wed, Nov 04, 2020 at 08:24:59AM -0500,
 Sawood Alam <[email-masked]> wrote 
 a message of 105 lines which said:

> The question remains open, "how to express a successful SEARCH
> response with an unsuccessful search result when returning an entity
> body is desired?" Does it warrant a new 2xx status code? I think
> this will be a common situation for many people who would be willing
> to migrate their GET-style searching where they could use a 404 to
> express it.

Note that there was a very similar discussion during the
standardization of DNS-over-HTTPS (RFC 8484). The decision was that
200 is the proper code even if the DNS request was unsuccessful. To
quote the RFC "For example, a successful 2xx HTTP status code is used
even with a DNS message whose DNS response code indicates failure,
such as SERVFAIL or NXDOMAIN."

The rationale being "HTTP succeeded, so 200, even if DNS failed".

Replace DNS by SEARCH and I think the reasoning still holds.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 5 Nov 2020 12:35:03 +0100

> The rationale being "HTTP succeeded, so 200, even if DNS failed".

That's IMHO the "tunnel other protocol over HTTP" approach which we
should avoid.

> Replace DNS by SEARCH and I think the reasoning still holds.

If the SEARCH request fails because the query is malformed, it should
return a 4xx. If the SEARCH request suceeds, but there is search result,
a 2xx is the right answer.

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 5 Nov 2020 09:25:57 -0800

> On Nov 5, 2020, at 3:06 AM, Stephane Bortzmeyer <[email-masked]> wrote:
> 

That rationale would be bogus. They are just being lazy.

....Roy

---

From: Ben Schwartz <[email-masked]>
Date: Thu, 5 Nov 2020 12:34:40 -0500

This was extensively debated during the DoH process.  I think the outcome
in RFC 8484 reasonably applies here too: if a search is successfully
performed, then HTTP has succeeded.  If the search result would benefit
from some explanatory metadata, that can be handled in the response body,
and the response format should be rich enough to express it.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 5 Nov 2020 10:38:33 -0800

> On Nov 5, 2020, at 9:34 AM, Ben Schwartz <[email-masked]> wrote:
> 
> This was extensively debated during the DoH process.  I think the outcome in RFC 8484 reasonably applies here too: if a search is successfully performed, then HTTP has succeeded.  If the search result would benefit from some explanatory metadata, that can be handled in the response body, and the response format should be rich enough to express it.

I don't care how extensively it was debated within some other WG.
It's not their call. In both cases, that rationale is bogus.

HTTP is an application-layer protocol and the responses must
reflect those semantics within the status code whether the
resource is a file, DNS entry, or search result.
It allows clients to be independent of the resource types
and intermediaries to work without application knowledge.

There are some searches that will return zero results and
be considered a success. That's fine -- it depends on how the
search is phrased. Zero results is not a "not found" error when
the search is "find some among many". 404 would only occur
if the request target is the error.

In contrast, failure codes on unsuccessful searches, like bad
query syntax, bad gateways, and timeouts (409, 412, 422, 5xx,
etc.), are not successful responses and must be reflected in
the HTTP status code. If the spec says otherwise, then that
spec is in error beyond its own scope.

Specifications that claim to be based on HTTP must adhere to
HTTP's requirements. Specifications that claim something bogus
have a conflict that they must address, either by fixing their
spec or by not claiming they are based on HTTP.

....Roy

---

From: Mike Bishop <[email-masked]>
Date: Thu, 5 Nov 2020 19:37:43 +0000

Roy, you seem to be disagreeing, but then presenting a detailed argument for
the same position you're disagreeing with.  If the search was successfully
performed on a resource, and the result of the search is an empty set,
that's a successful operation (2XX).  If the search could not be performed
for some reason, that's an error (4XX, 5XX).

I'm confused.  Can you clarify why you don't believe that aligns with "if a
search is successfully performed, then HTTP has succeeded"?

(I disagree with you slightly about "find some among many," because the
resource to be searched could be present but empty.  I think the
differentiator is whether the targeted resource exists, regardless of what
it contains.)

-----Original Message-----
From: Roy T. Fielding <[email-masked]> 
Sent: Thursday, November 5, 2020 1:39 PM
To: Ben Schwartz <[email-masked]>
Cc: Stephane Bortzmeyer <[email-masked]>; Sawood Alam
<[email-masked]>; Julian Reschke <[email-masked]>; ietf-http-wg
<[email-masked]>
Subject: Re: How to express no matching results in HTTP SEARH method?

in RFC 8484 reasonably applies here too: if a search is successfully
performed, then HTTP has succeeded.  If the search result would benefit from
some explanatory metadata, that can be handled in the response body, and the
response format should be rich enough to express it.

HTTP is an application-layer protocol and the responses must reflect those
semantics within the status code whether the resource is a file, DNS entry,
or search result.
It allows clients to be independent of the resource types and intermediaries
to work without application knowledge.

There are some searches that will return zero results and be considered a
success. That's fine -- it depends on how the search is phrased. Zero
results is not a "not found" error when the search is "find some among
many". 404 would only occur if the request target is the error.

In contrast, failure codes on unsuccessful searches, like bad query syntax,
bad gateways, and timeouts (409, 412, 422, 5xx, etc.), are not successful
responses and must be reflected in the HTTP status code. If the spec says
otherwise, then that spec is in error beyond its own scope.

Specifications that claim to be based on HTTP must adhere to HTTP's
requirements. Specifications that claim something bogus have a conflict that
they must address, either by fixing their spec or by not claiming they are
based on HTTP.

....Roy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 5 Nov 2020 12:46:01 -0800

> On Nov 5, 2020, at 11:37 AM, Mike Bishop <[email-masked]> wrote:
> 

That wasn't what I was responding to. The example given was a response
that was not a successful search: "SERVFAIL or NXDOMAIN" having a status
of 2xx. That's wrong.

> search is successfully performed, then HTTP has succeeded"?

The general statement of "when a search is successfully performed we
would expect a 2xx response" is true to the extent that what the
application is actually doing is searching for something wherein
"none found" is a successful response, like most queries that
respond with a list of resources that match.

But SEARCH (the method) isn't limited to searching for something
according to some criteria. Maybe it should be. Maybe not. I don't
think it matters, so long as the protocol is self-descriptive and
doesn't respond success when the request actually failed by its
own semantics.

> differentiator is whether the targeted resource exists, regardless of what

This isn't a helpful way to think. What we want from status codes
at the HTTP level is an indication of what recipients can do to
"fix this" even when they are not aware of the specific application
being processed over HTTP. A success doesn't need to be fixed.

For example, BigCo pays X megabucks to ensure that all valid queries
are processed within 50ms, including severe penalties when they don't.
The X has megaservers to respond to megaclients, but still doesn't
control probability and can't afford to buy a separate set of
megaservers for every specific application. So, it sprays all
SEARCH requests to N recipient servers in parallel and forwards
the first successful response. X cannot do that if it has to look
inside the body of a response to make an application-specific decision
about whether that response contains a successful SEARCH response.
Likewise, it might have an auditing requirement to store invalid
queries for some period of time, which it could do in 1 place
if the response indicates validity or in N places if it does not.

The easy thing for application developers to do is to always
send a 200 and make excuses within the content. If that was
generally considered to be an okay thing for one use of HTTP,
it would soon be for all uses of HTTP, and a whole bunch of
things that allow HTTP traffic to scale start failing.

....Roy