Subject: RFC7234: Can a request body form part of a "cache key"?

From: Graham Leggett <[email-masked]>
Date: Wed, 27 Jul 2016 17:18:49 +0200

Hi all,

RFC7234 states the following with respect to cache keys for methods other than GET:

"However, it is also possible to cache [...] responses to methods other than GET if the method's definition allows such caching and defines something suitable for use as a cache key.”

Could an RFC7234 compliant cache use a request body as part of the cache key, if it was so defined for a particular method?

Concrete example: If a cache cached a PROPFIND request keyed against a request body and URL, would that be RFC7234 compliant? Same for a SEARCH request?

Regards,
Graham
—

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 27 Jul 2016 10:13:05 -0700

> On Jul 27, 2016, at 8:18 AM, Graham Leggett <[email-masked]> wrote:
> 
> Hi all,
> 

Yes.

> Concrete example: If a cache cached a PROPFIND request keyed against a request body and URL, would that be RFC7234 compliant? Same for a SEARCH request?

Yes (assuming the PROPFIND response is marked cacheable) and Yes (ditto).
The problem is ensuring that the body is sufficiently limited so that it
won't be unique per request.

Caches that do not support this kind of caching should simply pass the
request/response through unharmed. Actual mileage may vary.

Cheers,

....Roy

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 27 Jul 2016 20:56:13 +0000

maybe need something like

Vary: Request-Body

?

Adrien

------ Original Message ------
From: "Roy T. Fielding" <[email-masked]>
To: "Graham Leggett" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 28/07/2016 5:13:05 AM
Subject: Re: RFC7234: Can a request body form part of a "cache key"?

>>  "However, it is also possible to cache [...] responses to methods 
>>other than GET if the method's definition allows such caching and 
>>defines something suitable for use as a cache key.”
>>
>>  Could an RFC7234 compliant cache use a request body as part of the 
>>cache key, if it was so defined for a particular method?
>
>Yes.
>
>>  Concrete example: If a cache cached a PROPFIND request keyed against 
>>a request body and URL, would that be RFC7234 compliant? Same for a 
>>SEARCH request?
>

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Jul 2016 23:37:33 +0200

> maybe need something like

I'd say this is implied anyway.

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 27 Jul 2016 17:26:39 -0600

> I'd say this is implied anyway.

RFC 7231 appears to imply the opposite: It explicitly allows GET
requests with bodies while not placing any request-body-related
restrictions on their response cachability and sharing AFAICT.
Similarly, there are instructions for caching POST responses that do not
mention request body importance (and nearly implying its irrelevance by
mentioning GET hits for POST-cached responses).

Alex.

---

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 28 Jul 2016 01:00:15 +0000

------ Original Message ------
From: "Alex Rousskov" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Cc: "Julian Reschke" <[email-masked]>; "Adrien de Croy" 
<[email-masked]>
Sent: 28/07/2016 11:26:39 AM
Subject: Re: RFC7234: Can a request body form part of a "cache key"?

>requests with bodies

"A payload within a GET request message has no defined semantics;
sending a payload body on a GET request might cause some existing
implementations to reject the request."

Same for HEAD.
Does anyone allow bodies on GET?  We block them I think.

I've also seen bodies on CONNECT...

Adrien

>while not placing any request-body-related

---

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 28 Jul 2016 01:25:00 +0000

bodies on GET is a can of worms.

how can GET be idempotent if we allow bodies on GET and say they have no 
defined semantics within HTTP?

Is GET with Content-Length: 0 semantically different to GET with no 
Content-Length header since the body has no defined semantics in http?

Is it therefore appropriate to strip Content-Length: 0 from GET 
requests?

Seems like the potential for request smuggling is large with bodies on 
GET.

Shouldn't we just ban them?  I'm sure this has been discussed at length 
on here many times before.

Maybe at least be a little stronger in the language?

e.g. clients SHOULD not send bodies on GET.

Adrien

------ Original Message ------
From: "Adrien de Croy" <[email-masked]>
To: "Alex Rousskov" <[email-masked]>; 
"[email-masked]" <[email-masked]>
Cc: "Julian Reschke" <[email-masked]>
Sent: 28/07/2016 1:00:15 PM
Subject: Re: RFC7234: Can a request body form part of a "cache key"?

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 28 Jul 2016 08:32:54 +0200

> Content-Length header since the body has no defined semantics in http?

Normally it is the same except that it indicates that the sender considers
there's an empty body instead of no body, and that it might be aware of
this possibility to send a body and is probably able to put something else
there. Just like servers expect to see a content-length 0 for POSTs even
if there's nothing to send because they want to see an empty body instead
of no body. By the way 7230 says this :

   A user agent SHOULD send a Content-Length in a request message when
   no Transfer-Encoding is sent and the request method defines a meaning
   for an enclosed payload body.  For example, a Content-Length header
   field is normally sent in a POST request even when the value is 0
   (indicating an empty payload body).  A user agent SHOULD NOT send a
   Content-Length header field when the request message does not contain
   a payload body and the method semantics do not anticipate such a
   body.

> Is it therefore appropriate to strip Content-Length: 0 from GET requests?

I don't see why it would not be. The framing remains exactly the same,
you're just telling the server that you're not aware of any body. Don't
forget that there's no such thing as as content length. What matters is
the size of the payload body, which in terms of framing is always there
right after the headers except when you establish a tunnel or when you
get a response which explicitly doesn't support a body (response to HEAD,
204, 304). For the semantics however the body is not always there. The
body size can be communicated in different ways, including the use of
Transfer-Encoding. It is suggested somewhere in 7230 that content-length
should be avoided when not expected, so I think that what your proxy is
doing probably slightly improves interoperability. Just found it here :

   A user agent SHOULD NOT send a
   Content-Length header field when the request message does not contain
   a payload body and the method semantics do not anticipate such a
   body.

> Seems like the potential for request smuggling is large with bodies on GET.

Yep, but not with content-length 0. Request smuggling is a framing issue.

> Shouldn't we just ban them?  I'm sure this has been discussed at length on
> here many times before.

Its been discussed in the past. Banning them causes more harm than not banning
them because it creates expectations on recipient that such a body doesn't
exist so it doesn't need to be checked. That's precisely what introduces
request smuggling. It's better to consider that content-length or
transfer-encoding are used for the framing and apply similarly to all known
and unknown methods. By the way this subject is covered here in 7231 :

   Since message parsing (Section 3.3 of [RFC7230]) needs to be
   independent of method semantics (aside from responses to HEAD),
   definitions of new methods cannot change the parsing algorithm or
   prohibit the presence of a message body on either the request or the
   response message.  Definitions of new methods can specify that only a
   zero-length message body is allowed by requiring a Content-Length
   header field with a value of "0".

> e.g. clients SHOULD not send bodies on GET.

I tend to think that the way it is done right now ensures that nobody is
willing to make shortcuts in their parsers and that's important. However
what we could have done (I thought we had it but can't find it) would be
to mention that a server MAY reject a GET with a body. That's more
dissuasive for clients and that still indicates that the server is not
allowed to be lazy on this.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 28 Jul 2016 08:35:40 +0200

> allowed to be lazy on this.

Just found it after sending, that's at the end of 4.3.1 in #7231 :

   A payload within a GET request message has no defined semantics;
   sending a payload body on a GET request might cause some existing
   implementations to reject the request.

It's less strong than what I thought however, I think we could suggest
servers to reject these requests if they don't intend to process the
body (nor simply to parse it).

Willy

---

From: Julian Reschke <[email-masked]>
Date: Thu, 28 Jul 2016 09:00:33 +0200

> restrictions on their response cachability and sharing AFAICT.

I don't see how this means that you could ignore the body should you 
decide to cache it.

> mentioning GET hits for POST-cached responses).

"Responses to POST requests are only cacheable when they include 
explicit freshness information (see Section 4.2.1 of [RFC7234]). 
However, POST caching is not widely implemented.

For cases where an origin server wishes the client to be able to cache 
the result of a POST in a way that can be reused by a later GET, the 
origin server MAY send a 200 (OK) response containing the result and a 
Content-Location header field that has the same value as the POST's 
effective request URI (Section 3.1.4.2)." -
<https://greenbytes.de/tech/webdav/rfc7231.html#rfc.section.4.3.3.p.4>, 
paragraph break added by me.

So these are two distinct things; the POST itself could be cacheable 
*and* the POST response can be decorated to imply that it can be used 
for subsequent GET requests.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 28 Jul 2016 15:29:34 +0200

> On 28 Jul 2016, at 8:35 AM, Willy Tarreau <[email-masked]> wrote:
> 

Pretty much.

The intent was to clarify that all requests can have bodies syntactically, because we have a (somewhat) generic message parsing model; HTTP software that doesn't care about the method (e.g., a proxy) shouldn't have to know about it to parse a message. 

However, GET itself defines no semantics, so it's useless to send one. That means that various software can and will assume that such a thing is useless if they *do* pay attention to the method, and may refuse it -- including things like firewalls, WAFs, etc.

Practically speaking, you're going to have a problem if you use a body on a GET with systems you don't own (i.e., at any scale). It may work as a pre-arranged, pairwise thing, but that's not really a primary use case for HTTP (and you're effectively creating a private GET by doing so).

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 28 Jul 2016 10:07:39 -0600

> I don't see how this means that you could ignore the body should you

What is not prohibited is allowed. If caching a GET response while
paying no attention to the GET request body is not prohibited, then it
is allowed. If it is allowed, then "Vary: Request-Body" is not implied
in the GET transaction context.

> for subsequent GET requests.

I am talking about the latter. The explicit and detailed instructions on
how to make a POST response cachable do not say anything about variance
on the POST request body. Thus, "Vary: Request-Body" is not implied in a
POST transaction context.

We can all join in a happy hand-waving exercise while talking about
"intent", "practice", and blocking valid messages we dislike, but until
somebody finds a _protocol_ requirement that clearly implies "Vary:
Request-Body", there is no such implication.

Alex.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 28 Jul 2016 18:18:58 +0200

> in the GET transaction context.

Then I'd say we should open a bug for RFC 7234bis.

> ...

Best regards, Julian

---

From: Graham Leggett <[email-masked]>
Date: Thu, 28 Jul 2016 18:20:22 +0200

> request/response through unharmed. Actual mileage may vary.

What would be a reasonable way to configure a RFC7234 compliant cache that wants to cache methods other than GET?

Would it be reasonable to explicitly specify the methods the cache is willing to cache, and have “GET” as the default? for example:

  CacheMethods GET PROPFIND SEARCH

What happens if the cache is misconfigured by the admin to cache something that is marked uncacheable by an RFC, for example:

  CacheMethods PUT DELETE

Would an RFC7234 compliant cache explicitly fail such a configuration?

Regards,
Graham
—

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 28 Jul 2016 10:39:51 -0600

On 07/28/2016 12:32 AM, Willy Tarreau wrote:
> there's an empty body instead of no body

In other words, it may be semantically different [for the sender and
possibly the ultimate recipient of the message].

> I don't see why it would not be. 

Your own [correct] explanation above shows why it would not be.
Stripping Content-Length:0 from GET requests may strip vital sender's
information.

Was it a good idea for the sender to use Content-Length:0 to carry vital
information? Of course not! However, from protocol point of view, "let's
strip Content-Length: 0 from GET requests" is equivalent to "let's
violate HTTP [payload forwarding requirements]".

If we acknowledge that empty payload may be semantically significant for
the sender, we should not encourage its removal.

>    a payload body and the method semantics do not anticipate such a

So, if a received request message has an [empty] payload body, we ought
to assume that the presumably compliant user agent meant to send it!
Note that the above requirement does not say "non-empty payload". In
other words, it tells user agents to avoid Content-Length [in GETs and
such] unless they do want to send [any] payload.

Cheers,

Alex.

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 28 Jul 2016 10:54:57 -0600

> Then I'd say we should open a bug for RFC 7234bis.

Yes, if you want to require something that was meant to be required but
actually was not, then amending the protocol is the way to go. It may be
difficult to justify that new requirement in this case though!

Thank you,

Alex.

---

From: David Morris <[email-masked]>
Date: Thu, 28 Jul 2016 10:29:22 -0700 (PDT)

It is impossible for me to imagine a use case where the body associated
with GET wouldn't have the same significance to caching as parameters
included on the URL. Hence, I believe this is a bug in the specification
and not a new requirement. To me this was an obvious requirement.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 28 Jul 2016 10:55:56 -0700

> On Jul 28, 2016, at 9:07 AM, Alex Rousskov <[email-masked]> wrote:
> 

I don't know how you came up with that twisted logic. GET is defined
as having no semantics for any body if it is present.  As such, an
implementation would be broken to do anything other than to parse and
discard that body if received.  Actually using the body to affect the
request would imply it has semantics.

There is no such thing as "Vary: Request-Body".  That isn't a field name,
would not be useful in Vary, and would have no effect on caching (other
than disabling it for caches that panic on vary).

> Request-Body", there is no such implication.

Feel free to implement it either way.  The semantics for POST are quite
loose, so the notion that there will be any independent interoperability
here is quite fanciful.  However, there is still an opportunity for specific
services where knowledge about what that POST means is known to the cache.

The same applies to PROPFIND and SEARCH, but those methods are defined by
other specs that are free to define semantics for the body.

....Roy

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 28 Jul 2016 12:03:04 -0600

On 07/28/2016 11:29 AM, David Morris wrote:

> included on the URL. Hence, ...

I can easily imagine such use cases. For example, "give me resource X
and, if you support prefetch, consider using my request body as a list
of resources I might request next".

If we use your decision logic, then my ability to imagine a use case
proves that body ignorance is not a bug but a useful feature!

Alex.

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 28 Jul 2016 12:37:26 -0600

> I don't know how you came up with that twisted logic. 

Sorry, but I see nothing twisted about it. It feels very
straightforward, actually, but I am obviously biased.

> GET is defined
> as having no semantics for any body if it is present.

HTTP RFCs do not define GET body semantics. That does not mean an
implementation may not assign its own semantics to that body. Naturally,
such an implementation would still have to rely on standard HTTP
mechanisms to correctly deliver that body to an origin server [which
understands that custom semantics].

> As such, an
> discard that body if received.

A specific origin server implementation that recognizes the body (based
on the URI, headers, or otherwise) may interpret it according to that
body semantics. A proxy implementation should forward that body intact
[unless the proxy is deployed to police/sanitize traffic]. If HTTP wants
to require some other/specific behavior, it should say so explicitly. It
had an opportunity to do that, but did not. It only said that some
servers _might_ reject GETs with bodies.

> Actually using the body to affect the request would imply it has semantics.

Agreed.

> than disabling it for caches that panic on vary).

Yes, and I hope everybody on this thread knows that it is just a
convenient notation to say "this [cachable] response variant depends on
the request body as a whole".

> services where knowledge about what that POST means is known to the cache.

Agreed! Your suggestion matches my assertion that there is no
protocol-level "Vary: Request-Body" implication in this area. Caching
implementations need to proceed with great caution, but they cannot be
called non-compliant if they do not assume "Vary: Request-Body".

> other specs that are free to define semantics for the body.

... but regardless of how they define that semantics, if they want that
request body to affect cachability of the response, they have to use
standard HTTP mechanisms to do so. For example, if they want the "Vary:
Request-Body"-like semantics, they could require a combination of
Content-Checksum request header and a "Vary: Content-Checksum" response
header.

Alex.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 28 Jul 2016 21:00:32 +0200

> Content-Checksum request header and a "Vary: Content-Checksum" response

Alex,

it's pretty clear that you're playing the role of the spec lawyer here. 
That's fine: it is needed to spot bugs in specs.

That said: it *is* a bug in the spec if people can read it as caching a 
PROPFIND while ignoring the body will ever be meaningful.

So let's open a ticket (will do tomorrow unless somebody is faster) and 
move on.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 28 Jul 2016 21:37:10 +0200

> and not a new requirement. To me this was an obvious requirement.

Given that it's clearly stated that the GET body (if any) has no defined
semantics, it doesn't make sense to expect caches to consider changes on
something with no semantics to decide whether or not to provide a response from
their cache. It's like showing someone a hand-written text using an alphabet
they don't know and saying them "turn right at the first roadsign you see
with this written on it". But the person doesn't read this alphabet, so she
doesn't know when different forms or line lengths are relevant or not and
indicate a difference or not. Here it's the same. Should your cache compare
bytes ? Should it assume %-encoding ? Should it ignore blank lines ? Should
it ignore the CR before an LF ? Should it ignore everything ? Every choice is
possible and valid until some semantics are defined, and here they are not.
At the very least you would need to define these semantics on the cache
itself before taking such a decision.

Regards,
Willy

---

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 28 Jul 2016 21:33:40 +0000

the text in the RFC is

"A payload within a GET request message has no defined semantics;"

I guess it comes down to interpretation as to whether you take that to 
mean

a) this document defines no semantics for the body on GET request 
messages
b) this document prohibits any semantic meaning being applied to the 
body on GET request messages

If you take the former interpretation then we have this situation.

If you take the latter, then I think it should be re-worded to be more 
specific, since it has implications like a proxy being free to strip 
bodies off GET request messages etc.

Adrien

------ Original Message ------
From: "Roy T. Fielding" <[email-masked]>
To: "Alex Rousskov" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>; "Julian Reschke" 
<[email-masked]>; "Adrien de Croy" <[email-masked]>
Sent: 29/07/2016 5:55:56 AM
Subject: Re: RFC7234: Can a request body form part of a "cache key"?

>>  On Jul 28, 2016, at 9:07 AM, Alex Rousskov 
>><[email-masked]> wrote:
>>

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 29 Jul 2016 07:26:02 +0200

On Thu, Jul 28, 2016 at 09:33:40PM +0000, Adrien de Croy wrote:
> I guess it comes down to interpretation as to whether you take that to mean
> 
> a) this document defines no semantics for the body on GET request messages
> b) this document prohibits any semantic meaning being applied to the body on
> GET request messages
> 
> specific, since it has implications like a proxy being free to strip bodies
> off GET request messages etc.

"has no defined semantics" implies "according to this spec". The spec
does not prohibit anything, it even suggests that some servers may
reject this body, meaning "use at your own risk". As soon as you want
to use that, you decide about what semantics you'll use.

A proxy like yours, designed to be used between end-users and the internet
and to increase security could safely strip such contents, because it will
be difficult for users and servers to agree on non-documented semantics. A
proxy designed to be the most transparent and interoperable possible, not
focusing specifically on security and possibly being deployed as a gateway
between internal components could be blamed for doing this if someone locally
decides to apply some semantics to a GET body. But that's always a risk when
using out-of-spec features ;-)

Willy

---

From: Julian Reschke <[email-masked]>
Date: Fri, 29 Jul 2016 08:02:51 +0200

> So let's open a ticket (will do tomorrow unless somebody is faster) and

-> <https://github.com/httpwg/http11bis/issues/17>

Best regards, Julian

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 29 Jul 2016 19:30:31 +1200

On 29/07/2016 4:20 a.m., Graham Leggett wrote:
> On 27 Jul 2016, at 7:13 PM, Roy T. Fielding wrote:
> 

Not easily. Each method has its own requirements or prohibitions about
caching. That includes a prohibition against caching unknown methods.

The criteria for most methods is quite complicated at times, so no
simple cache method X will be sufficient unless the proxy is coded to
know what to check for when caching method X. In which case you may as
well cache that method by defaut as well, and allow non-caching to be
the configurable aspect.

That is how we do it in Squid. Caching everything we can by default.
Just providing various ways for admin to limit what gets stored, or
manipulate the heuristic algorithms for revalidating things already stored.

Amos