Subject: Appropriate use of HTTP status codes for application health checks

From: matt <[email-masked]>
Date: Thu, 23 Feb 2017 00:54:11 +0000

Hello,

My colleagues and I are involved in a debate about the proper usage of HTTP return codes for application health pages.

For instance, you have a /health page that returns JSON listing your application’s dependencies as either “Up” or “Down”

Some suggest that it is acceptable for your /health page to return an unassigned 5xx or 503 if the /health page returns successfully, but the page results indicate the application is not healthy. Spring Boot <https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-1.1-Release-Notes#healthindicators> has done this. Although I have reservations about 503 since your request for the page was handled successfully.

Other contend that your /health page should always return a 200 regardless of whether the page results is indicative of application health or not. 

As a layman I can see the argument for both sides, and it seems both practices have been used in the past. I perused the RFCs but I don’t feel like I found the ‘silver bullet’ answer on this.

Kindly,

Matt

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 23 Feb 2017 19:36:44 +1000

Hi Matt,

On 23 February 2017 at 10:54, matt <[email-masked]> wrote:

> Hello,
>
> My colleagues and I are involved in a debate about the proper usage of
> HTTP return codes for application health pages.
>
> For instance, you have a /health page that returns JSON listing your
> application’s dependencies as either “Up” or “Down”
>
> Some suggest that it is acceptable for your /health page to return an
> unassigned 5xx or 503 if the /health page returns successfully, but the
> page results indicate the application is not healthy. Spring Boot
> <https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-1.1-Release-Notes#healthindicators> has
> done this. Although I have reservations about 503 since your request for
> the page was handled successfully.
>
> Other contend that your /health page should *always* return a 200
> regardless of whether the page results is indicative of application health
> or not.
>
> As a layman I can see the argument for both sides, and it seems both
> practices have been used in the past. I perused the RFCs but I don’t feel
> like I found the ‘silver bullet’ answer on this.
>
> Kindly,
>
> Matt
>

The trick with REST (assuming that's what you're going for here) is
defining the resources.

If the resource is a status page, then it can return 200 to say that
the *status
*is available, even if the available status is "unavailable".

If the resource is the dependency itself (or a model or representation
thereof), it can return a 5xx or whatever to say that the *dependency* is
unavailable.

In general, I say: do what feels right, and document it clearly.

Of course, I'm the biggest REST expert on this list, so I'm 100% correct.

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 23 Feb 2017 22:53:07 +1300

On 23/02/2017 1:54 p.m., matt wrote:
> Hello,
> 
> 
> 
> My colleagues and I are involved in a debate about the proper usage
> of HTTP return codes for application health pages.
> 
> 
> 

The action being performed as far as HTTP semantics are concerned is not
a health check - it is simply "fetch".

As such the status code refers to the "JSON file" thing not its
contents. HTTP does not care what that "JSON file" thing means to your
application, its just some opaque bytes to be located and delivered.

> unassigned 5xx or 503 if the /health page returns successfully, but

HTTP defines that all unknown 5xx codes are equivalent to 500 status.

For HTTP agents outside your application that usually means a retry with
different server is required, until a 2xx/3xx status is found or no
alternative servers can be identified.

So a 5xx status "working" in terms of your application health check is
actually an excentional event depending on a narrow set of circumstances;
 no other server/IPs available, and
 no alternative routes across the network to reach it.

> the page results indicate the application is not healthy. Spring Boot
> <https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-1.1-Release-Notes#healthindicators>
> has done this. Although I have reservations about 503 since your
> request for the page was handled successfully.
> 

Rightly so.

> Other contend that your /health page should always return a 200
> regardless of whether the page results is indicative of application
> health or not.
> 
> practices have been used in the past. I perused the RFCs but I don’t
> feel like I found the ‘silver bullet’ answer on this.
> 

The "always 200" is a bit strict. All 2xx and 3xx status mean successful
*fetch*, with various grades of meaning to that success.

IMHO a better efficient way for a polling system is to use 204 as "All
okay", and 200 as "some problem(s)". No bandwidth wasted with payload on
the common Up status, and ability to deliver details about the outage on
the Down status.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 23 Feb 2017 11:24:31 +0100

Hi Amos,

> the common Up status, and ability to deliver details about the outage on

In fact it's common to see health check applications return 5xx for a
very simple reason, the front equipment performing the check (often a
load balancer) has to deal with these situations anyway, and most use
cases just want to return "completely up" or "completely dead". But I
agree that when you want to support the gray area in between, it's much
better to support intermediary codes. FWIW haproxy also supports a
special case of 404 to mean "closing soon, no more requests please" so
that admins can simply touch/rm a file in a docroot. That's just to say
that there are many valid use cases and tha common sense adapted to what
components *reliably* support is often the best here.

Cheers,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 27 Feb 2017 10:21:47 +1100

Hi Matt,

My .02 -

What's most important when choosing a status code is what generic semantics you want to expose to *generic* HTTP software --  client libraries, proxies, load balancers, etc. They'll use them to do things like  retry requests, assign default caching policies, invoke error recovery, etc.

So, there are arguments for a 500 or a 200 here, depending on what you want generic HTTP software to do. I think most people would go for 500 (if only for the reason that Willy mentioned; some load balancers expect it).

If you choose 5xx, *don't* create a new one -- there isn't any point in application-specific status codes (see above). 500 is perfectly good for it; if more nuance or detail is necessary, you can carry that in headers and/or the body.

Cheers,

> On 23 Feb 2017, at 11:54 am, matt <[email-masked]> wrote:
> 
> Hello,
> 
> 
> 
> Some suggest that it is acceptable for your /health page to return an unassigned 5xx or 503 if the /health page returns successfully, but the page results indicate the application is not healthy. Spring Boot has done this. Although I have reservations about 503 since your request for the page was handled successfully.
> 
> 
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 27 Feb 2017 17:38:49 +1300

> components *reliably* support is often the best here.

For an individual health-check you are right. But that is not the
use-case matt has.

The use-case in question is for the response coming from some aggregator
process, which uses health-checks as its input/data. One status code
summarizing the situation of N endpoints.  No 4xx or 5xx is going to be
adequate for that, simply because of what the 400 and 500 defaults mean
to the general HTTP ecosystem.

The individual endpoints being health-tested, sure a 4xx/5xx is usually
best. Squid uses 503 to respond to *all* queries received during teh
final seconds of shutdown so retries can be done for them. Your 404 as a
final status is of most use for active health-check probes to an
individual endpoint.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 27 Feb 2017 07:19:37 +0100

> to the general HTTP ecosystem.

I totally get your point but I see a big difference between what would
be perfect and what components can do. For example for over a decade
haproxy was not able to consider anything but a status code, and because
of this there have been many people who implemented 500 as a response to
aggregated tests just for this (now it's more flexible). And I've had to
deal with other products which could only use this as well.

Also, even for an aggregated test, you may end up with real 5xx errors
because of timeouts or failure to deal with unexpected responses, so
the LB still has to deal with this case normally.

So I'd summarize it like this when seen from the front component :

   - 200 => status is OK
   - <something> => status is faulty (partially or totally)
   - 5xx => a technical error appeared during the processing

Given the 5xx has to be dealt with, if there is no need for a clear
distinction between a failure in the health check component and a
faulty test, the 5xx will work fine. If it's needed to make a
distinction (eg: all responses are logged), then something else
would be better (including some 2xx as you proposed).

Cheers,
willy