Subject: Re: #285: Strength of requirements on Accept re: 406

From: Mark Nottingham <[email-masked]>
Date: Sun, 17 Jul 2011 12:19:48 +1000

Proposal:

* Change text in Accept definition to:

"""
If an Accept header field is present in a request, but the server cannot send a response which is acceptable, then the server can either send a response in another format, or a 406 (Not Acceptable) response. 
"""

* Change text in Accept-Charset to:

"""
If an Accept-Charset header field is present in a request, but the server cannot send a response which is acceptable, then the server can either use another character encoding, or send a 406 (Not Acceptable) response. 
"""

* Change text in Accept-Encoding to:

"""
If an Accept-Encoding field is present in a request, but the server cannot send a response which is acceptable, then the server SHOULD send a response without any encoding (i.e., the "identity" encoding).
"""

* Add this paragraph to 5.1 Server-Driven Negotiation:

"""
Server-driven negotiation allows the user agent to specify its preferences, but it cannot expect responses to always honour them. For example, the origin server might not implement server-driven negotiation, or it might decide that sending a response that doesn't conform to them is better than sending a 406 (Not Acceptable) response.

Many of the mechanisms for expressing preferences use quality values to declare relative preference. See Section 6.4 of [Part1] for more information. 
"""

On 28/06/2011, at 3:41 PM, Mark Nottingham wrote:

> Back to the original issue here.
> 
> I think it can be broken down like this:
> 
> 1) Accept's definition needs to be aligned with the definition of 406 to allow the server to send something other than what's in Accept. See proposal at the bottom of this e-mail.
> 
> 2) Accept-Charset explicitly says that if it's present, anything not mentioned gets a quality value of '0'. Accept-Language implies that (depending on how you read it). In both cases, if servers are disallowed from sending anything not acceptable, this implies that these headers are mandatory. I'd argue that the requirements should be relaxed in a manner similar to the proposal for Accept, to match reality.
> 
> 3) Either way, more needs to be said about this in p3 5.1 Server-Driven negotiation.
> 
> I'm happy to write a proposal along these lines; anyone with concerns about going in this direction?
> 
> Cheers,
> 
> 
> On 22/06/2011, at 10:21 AM, Mark Nottingham wrote:
> 
>> <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/285>
>> 
>> The definition of the 406 status code allows servers to override the clients' preferences:
>> 
>> """
>> Note: HTTP/1.1 servers are allowed to return responses which are not acceptable according to the accept header fields sent in the request. In some cases, this might even be preferable to sending a 406 response. User agents are encouraged to inspect the header fields of an incoming response to determine if it is acceptable.
>> """
>> 
>> (editorial nit: I think that should be "according to the Accept and Accept-* header fields...")
>> 
>> However, this isn't really reflected in the Accept-* header requirements; e.g., for Accept:
>> 
>> """
>> If an Accept header field is present, and if the server cannot send a response which is acceptable according to the combined Accept field value, then the server SHOULD send a 406 (Not Acceptable) response.
>> """
>> 
>> For Accept-Charset, we currently have:
>> 
>> """
>> If no "*" is present in an Accept-Charset field, then all character encodings not explicitly mentioned get a quality value of 0.
>> """
>> 
>> .... which implies that they're unacceptable.
>> 
>> Accept-Encoding has:
>> 
>> """
>> If the Accept-Encoding field-value is empty, then only the "identity" encoding is acceptable.
>> 
>> If an Accept-Encoding field is present in a request, and if the server cannot send a response which is acceptable according to the Accept-Encoding header field, then the server SHOULD send an error response with the 406 (Not Acceptable) status code.
>> 
>> If no Accept-Encoding field is present in a request, the server MAY assume that the client will accept any content coding.  In this case, if "identity" is one of the available content-codings, then the server SHOULD use the "identity" content-coding, unless it has additional information that a different content-coding is meaningful to the client.
>> """
>> 
>> Finally, for Accept-Language, we currently defer to RFC4647, but 2616 said:
>> 
>> """
>> If no language-range in the field matches the tag, the language quality factor assigned is 0. [...] If an Accept-Language header is present, then all languages which are assigned a quality factor greater than 0 are acceptable.
>> """
>> 
>> 
>> My understanding has always been that it's always permissible to send a response in a format that isn't explicitly called out in the Accept header, UNLESS Accept explicitly contains */*;q=0 -- and even then, the client needs to be able to handle an unexpected response format. This is because you can't require servers (especially HTTP/1.0 servers, for which there aren't hard requirements) to understand the Accept-* headers. 
>> 
>> If that's the case, I think we should soften these requirements in the Accept-* headers and also align them. Straw-man text for Accept:
>> 
>> """
>> If an Accept header field is present, but the server cannot send a response which is acceptable according to its field value, the server can either send a response in another format, or a 406 (Not Acceptable) response. However, the server SHOULD NOT send a format that matches a media-range with a qvalue of "0".
>> """
>> 
>> Thoughts? It's pretty clear that the smallest impact change would be to just change the definition of 406 to align with the text in the Accept-* headers, but I don't think that reflects reality.
>> 
>> --
>> Mark Nottingham   http://www.mnot.net/
>> 
>> 
>> 
>> 
> 
> --
> Mark Nottingham   http://www.mnot.net/
> 
> 
> 
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Jul 2011 19:32:25 +0200

On 2011-07-17 04:19, Mark Nottingham wrote:
> ...

Works for me; proposed patch in 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/285/285.diff>.

I wasn't entirely sure where to put the new text in 5.1; right now the 
whole subsection reads like this...:

5.1.  Server-driven Negotiation

    If the selection of the best representation for a response is made by
    an algorithm located at the server, it is called server-driven
    negotiation.  Selection is based on the available representations of
    the response (the dimensions over which it can vary; e.g., language,
    content-coding, etc.) and the contents of particular header fields in
    the request message or on other information pertaining to the request
    (such as the network address of the client).

    Server-driven negotiation is advantageous when the algorithm for
    selecting from among the available representations is difficult to
    describe to the user agent, or when the server desires to send its
    "best guess" to the client along with the first response (hoping to
    avoid the round-trip delay of a subsequent request if the "best
    guess" is good enough for the user).  In order to improve the
    server's guess, the user agent MAY include request header fields
    (Accept, Accept-Language, Accept-Encoding, etc.) which describe its
    preferences for such a response.

    Server-driven negotiation has disadvantages:

    1.  It is impossible for the server to accurately determine what
        might be "best" for any given user, since that would require
        complete knowledge of both the capabilities of the user agent and
        the intended use for the response (e.g., does the user want to
        view it on screen or print it on paper?).

    2.  Having the user agent describe its capabilities in every request
        can be both very inefficient (given that only a small percentage
        of responses have multiple representations) and a potential
        violation of the user's privacy.

    3.  It complicates the implementation of an origin server and the
        algorithms for generating responses to a request.

    4.  It might limit a public cache's ability to use the same response
        for multiple user's requests.

    Server-driven negotiation allows the user agent to specify its
    preferences, but it cannot expect responses to always honour them.
    For example, the origin server might not implement server-driven
    negotiation, or it might decide that sending a response that doesn't
    conform to them is better than sending a 406 (Not Acceptable)
    response.

    Many of the mechanisms for expressing preferences use quality values
    to declare relative preference.  See Section 6.4 of [Part1] for more
    information.

    HTTP/1.1 includes the following header fields for enabling server-
    driven negotiation through description of user agent capabilities and
    user preferences: Accept (Section 6.1), Accept-Charset (Section 6.2),
    Accept-Encoding (Section 6.3), Accept-Language (Section 6.4), and
    User-Agent (Section 9.9 of [Part2]).  However, an origin server is
    not limited to these dimensions and MAY vary the response based on
    any aspect of the request, including aspects of the connection (e.g.,
    IP address) or information within extension header fields not defined
    by this specification.

       Note: In practice, User-Agent based negotiation is fragile,
       because new clients might not be recognized.

    The Vary header field (Section 3.5 of [Part6]) can be used to express
    the parameters the server uses to select a representation that is
    subject to server-driven negotiation.

...feedback appreciated,

Julian

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 19 Jul 2011 12:03:51 +1200

Hi

just an observation.

An example in the prose below refers to a server wanting to vary the 
response based on say the client IP.

How would it specify that in a Vary header?

Since that's not actually a request header, I don't see how the server 
can emit a vary header which refers to it.

We would need some standardised tokens to represent such things, 
otherwise caching intermediaries cannot store and use the correct metadata.

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com
WinGate 7 beta out now - http://www.wingate.com/getlatest/

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Jul 2011 10:08:03 +1000

Vary: *

On 19/07/2011, at 10:03 AM, Adrien de Croy wrote:

> 
> Hi
> 
> just an observation.
> 
> An example in the prose below refers to a server wanting to vary the response based on say the client IP.
> 
> Since that's not actually a request header, I don't see how the server can emit a vary header which refers to it.
> 
> We would need some standardised tokens to represent such things, otherwise caching intermediaries cannot store and use the correct metadata.
> 
> Regards
> 
> Adrien
> 
> 
>> Works for me; proposed patch in <http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/285/285.diff>.
>> 
>> I wasn't entirely sure where to put the new text in 5.1; right now the whole subsection reads like this...:
>>

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 19 Jul 2011 12:18:27 +1200

well I guess that means that the Vary header can only refer to request 
headers or "*", and therefore cannot refer to all the attributes that 
were used to select the representation.

I realise this is probably a vanishingly small benefit, but I wonder if 
there is any use for a concept of well-known tokens for such things, or 
concept of virtual request headers, which amount to the same thing.  E.g 
any request can be deemed to contain the headers

ClientIp: <ipaddress>
TransportProtocol: <transport protocol>
etc

then it can be referred to by Vary, cache-control etc, and intermediate 
caches can then do something useful with content which varies on it, 
rather than the server just blanket forcing revalidation with Vary: *

ClientIp is obviously difficult when it comes to proxies, as the O-S 
won't see the client IP unless it is forwarded in another header (such 
as X-F-F).  So actually maybe giving the example of the client IP is 
misleading in the prose.

Apologies if this is pedantism taken too far.  There could be some 
interesting possibilities with considering this is all.

Adrien

On 19/07/2011 12:08 p.m., Mark Nottingham wrote:
>>> Works for me; proposed patch in<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/285/285.diff>.
>>>

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Jul 2011 10:25:05 +1000

It's been discussed before, and there might be some future work on a more fine-grained replacement for Vary. But not as a work item under this charter :)

Cheers,

On 19/07/2011, at 10:18 AM, Adrien de Croy wrote:

> 
> well I guess that means that the Vary header can only refer to request headers or "*", and therefore cannot refer to all the attributes that were used to select the representation.
> 
> I realise this is probably a vanishingly small benefit, but I wonder if there is any use for a concept of well-known tokens for such things, or concept of virtual request headers, which amount to the same thing.  E.g any request can be deemed to contain the headers
> 
> ClientIp: <ipaddress>
> then it can be referred to by Vary, cache-control etc, and intermediate caches can then do something useful with content which varies on it, rather than the server just blanket forcing revalidation with Vary: *
> 
> ClientIp is obviously difficult when it comes to proxies, as the O-S won't see the client IP unless it is forwarded in another header (such as X-F-F).  So actually maybe giving the example of the client IP is misleading in the prose.
> 
> Apologies if this is pedantism taken too far.  There could be some interesting possibilities with considering this is all.
> 
> Adrien
> 
>

---

From: "Eric J. Bowman" <[email-masked]>
Date: Mon, 18 Jul 2011 19:43:46 -0600

> otherwise caching intermediaries cannot store and use the correct

Well, that's one approach.  Another would be to design the system to
work within the constraints of the protocol -- I'd use Digest auth,
Vary: Authorization and stash the client IP in username.

-Eric

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 19 Jul 2011 14:56:25 +1200

> Vary: Authorization and stash the client IP in username.

 My first response was to think "Cache-Control: private" to prevent 
 storing it for service to multiple clients.
 With possible proxy-revalidate as a backup if they did cache it.

 Amos

---

From: Julian Reschke <[email-masked]>
Date: Tue, 19 Jul 2011 09:16:32 +0200

> <http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/285/285.diff>.

-> <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1335>