Subject: #285: Strength of requirements on Accept re: 406

From: Mark Nottingham <[email-masked]>
Date: Wed, 22 Jun 2011 10:21:15 +1000

<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/285>

The definition of the 406 status code allows servers to override the clients' preferences:

"""
Note: HTTP/1.1 servers are allowed to return responses which are not acceptable according to the accept header fields sent in the request. In some cases, this might even be preferable to sending a 406 response. User agents are encouraged to inspect the header fields of an incoming response to determine if it is acceptable.
"""

(editorial nit: I think that should be "according to the Accept and Accept-* header fields...")

However, this isn't really reflected in the Accept-* header requirements; e.g., for Accept:

"""
If an Accept header field is present, and if the server cannot send a response which is acceptable according to the combined Accept field value, then the server SHOULD send a 406 (Not Acceptable) response.
"""

For Accept-Charset, we currently have:

"""
If no "*" is present in an Accept-Charset field, then all character encodings not explicitly mentioned get a quality value of 0.
"""

.... which implies that they're unacceptable.

Accept-Encoding has:

"""
If the Accept-Encoding field-value is empty, then only the "identity" encoding is acceptable.

If an Accept-Encoding field is present in a request, and if the server cannot send a response which is acceptable according to the Accept-Encoding header field, then the server SHOULD send an error response with the 406 (Not Acceptable) status code.

If no Accept-Encoding field is present in a request, the server MAY assume that the client will accept any content coding.  In this case, if "identity" is one of the available content-codings, then the server SHOULD use the "identity" content-coding, unless it has additional information that a different content-coding is meaningful to the client.
"""

Finally, for Accept-Language, we currently defer to RFC4647, but 2616 said:

"""
If no language-range in the field matches the tag, the language quality factor assigned is 0. [...] If an Accept-Language header is present, then all languages which are assigned a quality factor greater than 0 are acceptable.
"""

My understanding has always been that it's always permissible to send a response in a format that isn't explicitly called out in the Accept header, UNLESS Accept explicitly contains */*;q=0 -- and even then, the client needs to be able to handle an unexpected response format. This is because you can't require servers (especially HTTP/1.0 servers, for which there aren't hard requirements) to understand the Accept-* headers. 

If that's the case, I think we should soften these requirements in the Accept-* headers and also align them. Straw-man text for Accept:

"""
If an Accept header field is present, but the server cannot send a response which is acceptable according to its field value, the server can either send a response in another format, or a 406 (Not Acceptable) response. However, the server SHOULD NOT send a format that matches a media-range with a qvalue of "0".
"""

Thoughts? It's pretty clear that the smallest impact change would be to just change the definition of 406 to align with the text in the Accept-* headers, but I don't think that reflects reality.

--
Mark Nottingham   http://www.mnot.net/

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 22 Jun 2011 21:14:39 +1200

> If no Accept-Encoding field is present in a request, the server MAY assume that the client will accept any content coding.  In this case, if "identity" is one of the available content-codings, then…

"If no Accept-Encoding field is present in a request, the server MAY 
assume that the client will accept any content coding. "

This seems highly dangerous to me.  IMO it would be extremely foolhardy 
for a server to send back content gzipped when there was no 
Accept-Encoding header at all.

Surely a sensible requirement would be that in the absense of an 
indication that a client can handle any content-encoding other than 
identity, the identity encoding MUST be used.

Otherwise we place a requirement on all clients to add Accept-Encoding: 
identity to all requests, which is mindless bloat.

Or am I reading this wrong?

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 22 Jun 2011 19:18:55 +1000

On 22/06/2011, at 6:58 PM, Adrien de Croy wrote:

> "If no Accept-Encoding field is present in a request, the server MAY assume that the client will accept any content coding. "
> 
> This seems highly dangerous to me.  IMO it would be extremely foolhardy for a server to send back content gzipped when there was no Accept-Encoding header at all.
> 
> Surely a sensible requirement would be that in the absense of an indication that a client can handle any content-encoding other than identity, the identity encoding MUST be used.
> 
> Otherwise we place a requirement on all clients to add Accept-Encoding: identity to all requests, which is mindless bloat.
> 

Some background here:
  http://www.stevesouders.com/blog/2010/07/12/velocity-forcing-gzip-compression/

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Wed, 22 Jun 2011 14:20:47 +0200

On 2011-06-22 11:18, Mark Nottingham wrote:
> ...
> ...

If I read this correctly, this is a workaround for broken intermediaries 
(and maybe IE6 behind proxies).

Do we really need to address this in the spec? And does it affect 
anything beyond Accept-Encoding?

Are we sure that 406 is supposed to be about all "Accept-*" headers?

Maybe a compromise would be to mention that some intermediaries are 
known to eat "Accept-Encoding", and thus origin servers can pick 
Content-Codings of their choice if they are sure the recipient will be 
able to process them.

Best regards, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 23 Jun 2011 09:59:09 +1200

Some versions of WinGate removed Accept-Encoding.

If the server ignored that, and sent say gzipped back, it would cause 
problems, esp with caching.

I think it's a really bad idea to ignore it.  It would have been removed 
for a reason.

Some plug-in filters also remove it (e.g. they can't handle certain 
encodings, but need to scan the data).  So ignoring again causes problems.

In general I think ignoring Accept-Encoding or sending an encoded 
response when there was no Accept-Encoding received is a very bad idea.

Adrien

On 23/06/2011 12:20 a.m., Julian Reschke wrote:
> If I read this correctly, this is a workaround for broken 
> intermediaries (and maybe IE6 behind proxies).
>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 22 Jun 2011 15:09:27 -0700

> This seems highly dangerous to me.  IMO it would be extremely foolhardy for a server to send back content gzipped when there was no Accept-Encoding header at all.

See the definition of the field.  No Accept-Encoding means send me ANY encoding.
An empty Accept-Encoding field-value means "no encoding".

Accept-Encoding was defined two years after Cotent-Encoding was *deployed*.
There was simply no other possible definition that would work.

....Roy

---

From: Brian Pane <[email-masked]>
Date: Wed, 22 Jun 2011 16:11:44 -0700

On Wed, Jun 22, 2011 at 2:59 PM, Adrien de Croy <[email-masked]> wrote:
> I think it's a really bad idea to ignore it.  It would have been removed for
> a reason.
>
> In general I think ignoring Accept-Encoding or sending an encoded response
> when there was no Accept-Encoding received is a very bad idea.

>From a server-side perspective, though, allowing a client or
intermediary to quadruple the server operator's egress network transit
cost by removing the Accept-Encoding seems undesirable - I'm inclined
to call it a vulnerability, in fact.

Thus my preference would be to add something like "implementations
MUST accept a Content-Encoding of 'gzip' for requests and responses,"
but unfortunately that would cause some formerly HTTP/1.1-compliant
implementations to suddenly be noncompliant, so it's out of scope for
HTTPbis.  For the next version of HTTP, though, I think it's
worthwhile to consider requiring that implementations support one or
more specified compression mechanisms.

-Brian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 23 Jun 2011 09:27:18 +1000

I'll point out that this text was in 2616:
  http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3
and also in 2068 (section 14.3).

So, we've gotten this far without the Web blowing up, and as the link below points out, people are relying on this behaviour; i.e., if we change the requirement, we'll make servers (e.g., google.com) non-conformant. 

On 22/06/2011, at 10:20 PM, Julian Reschke wrote:

> If I read this correctly, this is a workaround for broken intermediaries (and maybe IE6 behind proxies).
> 
> Do we really need to address this in the spec? And does it affect anything beyond Accept-Encoding?
> 
> Maybe a compromise would be to mention that some intermediaries are known to eat "Accept-Encoding", and thus origin servers can pick Content-Codings of their choice if they are sure the recipient will be able to process them.
> 
> Best regards, Julian

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 23 Jun 2011 11:41:39 +1200

OK, I can see how the definition kinda works in 2616, but there's a problem.

what happens when someone rolls out a new encoding?

That means you need to update ALL clients now to exclude it if it's not 
supported, if they were relying on lack of the header.

Otherwise you need to add Accept-Encoding to all requests = bloat.

What about HTTP/1.0 clients?  Surely they don't send Accept-Encoding, 
and a server can't use Content-Encoding in responses?

Just seems a bit backward to me to have the default (no header) mean 
everything, including the unknown.

If C-E had already been rolled out, maybe those deployments should have 
been fixed.  Too late now I guess so we live with the bloat.  What it 
means I guess though is that even if a filter that previously would have 
stripped the A-E header instead inserts A-E: identity, it still can't 
rely on getting that back as the server gets to make the final decision, 
it's only a should requirement to send back a 406.

Makes for fun when a proxy down-grades a request (wrt A-E) which then 
gets bounced by the Origin Server because of the downgrade.

Which then means proxies really shouldn't mess with A-E, and need to 
support all encodings, including all the filters in their processing chain.

Adrien

On 23/06/2011 10:09 a.m., Roy T. Fielding wrote:
>> "If no Accept-Encoding field is present in a request, the server MAY assume that the client will accept any content coding."
>>

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 23 Jun 2011 18:50:49 +1200

On 23/06/11 11:41, Adrien de Croy wrote:
> What about HTTP/1.0 clients? Surely they don't send Accept-Encoding, and
> a server can't use Content-Encoding in responses?
>

Indicating that proxies SHOULD only upgrade. Adding encodings they can 
translate down to one of the known clients acceptable ones.

> support all encodings, including all the filters in their processing chain.

For a _proxy_ this is not a problem. For a content filter it likely is.

Then again you can look at the rollout of sdch and xpress encodings as a 
case study on encodings the middleware is not knowing about.

AYJ

---

From: Yves Lafon <[email-masked]>
Date: Thu, 23 Jun 2011 05:12:22 -0400 (EDT)

> supported, if they were relying on lack of the header.

Well, there is a MAY there, so if the server wants to always serve a CE 
not supported by the majority of clients, that's its (bad) choice.

> Otherwise you need to add Accept-Encoding to all requests = bloat.

Most people inspect the UA string in that case (but omit to send a Vary: 
header... if there are different encodings available).

> What about HTTP/1.0 clients?  Surely they don't send Accept-Encoding, and a 
> server can't use Content-Encoding in responses?
>
> If C-E had already been rolled out, maybe those deployments should have been 
> fixed.  Too late now I guess so we live with the bloat.  What it means I 
> guess though is that even if a filter that previously would have stripped the 
> A-E header instead inserts A-E: identity, it still can't rely on getting that 
> back as the server gets to make the final decision, it's only a should 
> requirement to send back a 406.
>
> Makes for fun when a proxy down-grades a request (wrt A-E) which then gets 
> bounced by the Origin Server because of the downgrade.
>
> Which then means proxies really shouldn't mess with A-E, and need to support 
> all encodings, including all the filters in their processing chain.

Proxies don't need to know about the A-E/C-E, unless they want to inspect 
the payload. (as opposed to TE/T-E), they just shouldn't mess with it...

>>>> If an Accept-Encoding field is present in a request, and if the server 
>>>> cannot send a response which is acceptable according to the 
>>>> Accept-Encoding header field, then the server SHOULD send an error 
>>>> response with the 406 (Not Acceptable) status code.
>>>> 
>>>> If no Accept-Encoding field is present in a request, the server MAY 
>>>> assume that the client will accept any content coding.  In this case, if 
>>>> "identity" is one of the available content-codings, then the server 
>>>> SHOULD use the "identity" content-coding, unless it has additional 
>>>> information that a different content-coding is meaningful to the client.
>>>> """
>>>> 
>>> 
>>> assume that the client will accept any content coding."
>>> 

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 28 Jun 2011 15:41:03 +1000

Back to the original issue here.

I think it can be broken down like this:

1) Accept's definition needs to be aligned with the definition of 406 to allow the server to send something other than what's in Accept. See proposal at the bottom of this e-mail.

2) Accept-Charset explicitly says that if it's present, anything not mentioned gets a quality value of '0'. Accept-Language implies that (depending on how you read it). In both cases, if servers are disallowed from sending anything not acceptable, this implies that these headers are mandatory. I'd argue that the requirements should be relaxed in a manner similar to the proposal for Accept, to match reality.

3) Either way, more needs to be said about this in p3 5.1 Server-Driven negotiation.

I'm happy to write a proposal along these lines; anyone with concerns about going in this direction?

Cheers,

---

From: Karl Dubost <[email-masked]>
Date: Wed, 29 Jun 2011 15:15:42 -0400

> If an Accept header field is present, and if the server cannot send a response which is acceptable according to the combined Accept field value, then the server SHOULD send a 406 (Not Acceptable) r…

Rails do not take into account Accept headers
https://github.com/rails/rails/commit/2f4aaed7b3feb3be787a316fab3144c06bb21a27
And it seems PHP Recess is doing the same (to check)

I do not know if IE still does that (to check) but
"Hit F5 to refresh that page, and IE will probably go back to sending */* again."
— http://blogs.msdn.com/b/ieinternals/archive/2009/07/01/9811694.aspx

-- 
Karl Dubost - http://dev.opera.com/
Developer Relations & Tools, Opera Software