Subject: #904: Content on GET requirement strength

From: Mark Nottingham <[email-masked]>
Date: Fri, 16 Jul 2021 17:06:35 +1000

<https://github.com/httpwg/http-core/issues/904>

Currently, we have consensus around the following text in Semantics:

> A client SHOULD NOT generate content in a GET request. Content received in a GET request has no defined semantics, cannot alter the meaning or target of the request, and might lead some implementations to reject the request and close the connection because of its potential as a request smuggling attack (Section 11.2 of [Messaging]).

<https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#GET>

In her review, our AD pointed out that it's good practice to explain the conditions under which a SHOULD NOT be violated -- generally, if we can't do so, it's a sign that the requirement isn't well stated.

In discussing this, we haven't yet been able to come up with such an explanation. So, the current proposal is to replace the text above with:

> A client MUST NOT generate content in a GET request. Content received in a GET request has no defined semantics, cannot alter the meaning or target of the request, and might lead some implementations to reject the request and close the connection because of its potential as a request smuggling attack (Section 11.2 of [Messaging]).

Thoughts? If folks disagree, suggestions about how to qualify the SHOULD NOT would be appreciated (see the issue for some options already discussed). 

Alternatively, if we can't come to relatively quick consensus on MUST NOT or a qualification for SHOULD NOT, we could consider leaving it as an unqualified SHOULD NOT - the AD comment isn't blocking. Not terribly good editorial practice, but I'm pretty sure we've done worse somewhere.

Cheers,

P.S. The same adjustment would be made on HEAD and DELETE, since we've tried to align their language regarding this behaviour.

--
Mark Nottingham   https://www.mnot.net/

---

From: Mike Bishop <[email-masked]>
Date: Fri, 16 Jul 2021 14:33:53 +0000

I'm supportive of your perspective that GET (which is already defined to place no semantic value on the body) is not compatible HTTP when sent with content.  However, the http-core docs' practice has been to avoid making existing uses newly non-conformant.  This is the third generation of these specs, and late in the process at that.

As I've added on the issue, I feel like this is converging on a "MUST NOT unless."  There is a narrowly scoped situation in which this can be done -- prior arrangement with all parties -- and it's otherwise prohibited.  That captures both the strength of our guidance not to do this, but also the reality that some people already do and presumably don't plan to stop.

That said, I don't feel like this is a hill to die on.  The spec says this is not good behavior, and explains why.  The explanation of the SHOULD NOT could simply be the existence of resources which define private semantics to GET content.  Clients interacting with such resources will have to choose whether to bow to the API definition.

-----Original Message-----
From: Mark Nottingham <[email-masked]> 
Sent: Friday, July 16, 2021 3:07 AM
To: HTTP Working Group <[email-masked]>
Subject: #904: Content on GET requirement strength

Cheers,

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 16 Jul 2021 18:11:35 +0200

On 2021-07-16, at 16:33, Mike Bishop <[email-masked]> wrote:
> 
> prior arrangement with all parties -- and it's otherwise prohibited.

Why is this one different from other places where the standard is making clear mandates and there is still behavior out there that violates those?
Are we downgrading all MUSTs to SHOULDs as soon as we see deviating behavior between consenting implementations in the wild?

Grüße, Carsten

---

From: David Benjamin <[email-masked]>
Date: Fri, 16 Jul 2021 12:37:03 -0400

Is the spec's explanation of the implications here even sufficient? In
addition to the request getting potentially rejected, might it also lead to
an actual request smuggling attack if the server *doesn't* reject the
request but, instead, interprets it differently?

Given the security risk, the current text doesn't seem strong enough to me.

On Fri, Jul 16, 2021 at 12:15 PM Carsten Bormann <[email-masked]> wrote:

> Why is this one different from other places where the standard is making
> clear mandates and there is still behavior out there that violates those?
> Are we downgrading all MUSTs to SHOULDs as soon as we see deviating
> behavior between consenting implementations in the wild?
>
> Grüße, Carsten
>
>
>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 16 Jul 2021 18:41:37 +0200

Am 16.07.2021 um 18:11 schrieb Carsten Bormann:
> ...

Adding history:

In RFC 2616, the spec had no normative requirement not to send GET
request bodies.

RFC 7230 added a statement about the payload having no defined syntax,
still no normative requirement.

The current draft, after two LCs, now says:

"A client SHOULD NOT generate content in a GET request. Content received
in a GET request has no defined semantics, cannot alter the meaning or
target of the request, and might lead some implementations to reject the
request and close the connection because of its potential as a request
smuggling attack (Section 11.2 of [Messaging])."

This discussion is about either adding more prose to the SHOULD NOT, or
further *upgrading* it to a MUST NOT.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Jul 2021 18:47:53 +0200

> further *upgrading* it to a MUST NOT.

I'd add that the algorithm described to determine the presence of a
message body and its length covers all use case, and that I would find
it particularly dangerous to consider that absolutely all methods including
unknown ones use the standard algorithm *except* GET.

I find the "SHOULD NOT" well balanced to discourage attempting to use that
over unknown infrastructure components without suddenly declaring all
compatible implementations non-compliant. As proposed there, adding an
"unless negotiated by other means" or something along this would be nice
though.

Willy

---

From: Asbjørn Ulsberg <[email-masked]>
Date: Fri, 16 Jul 2021 22:43:05 +0200

fre. 16. jul. 2021 kl. 18:40 skrev David Benjamin <[email-masked]>:

> addition to the request getting potentially rejected, might it also lead to an
> actual request smuggling attack if the server doesn't reject the request but,
> instead, interprets it differently?

Indeed. As far as I can tell, there's also a potential security and
privacy issue in having an intermediate cache reuse the response from
a prior query because it (rightly) does not consider the request body
to be significant when keying the cache entry. This can both lead to
unauthorized access to data as well as a possible cache poisoning
vector.

> Given the security risk, the current text doesn't seem strong enough to me.

Agreed. But I think it may be worth reflecting on why GET with a body
is an issue worth discussing both here on the mailing list and in the
HTTP spec itself. Enough implementations of this anti-pattern over the
years have made it evident (at least to me) that there is a need for a
safe HTTP method sporting a request body. The most common use case is
performing complex queries where the query string either reveals too
much information, causes length issues, or is just too messy. To
accommodate these use cases, an I-D for a safe method with body has
been initiated:

https://datatracker.ietf.org/doc/draft-ietf-httpbis-safe-method-w-body/

With such a method in the implementer's toolbox, I'm pretty certain a
MUST NOT requirement would be easy to swallow. However, since the I-D
is still far from completion and there are no standardized
alternatives to GET with body, a more elaborate explanation of
potential security and privacy risks and stronger language with the
current SHOULD NOT requirement seems appropriate.

-- 
Asbjørn Ulsberg           -=|=-        [email-masked]
«He's a loathsome offensive brute, yet I can't look away»

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 17 Jul 2021 11:50:29 +0200

On 2021-07-16, at 22:43, Asbjørn Ulsberg <[email-masked]> wrote:
> To
> MUST NOT requirement would be easy to swallow.

Indeed.

> However, since the I-D

https://datatracker.ietf.org/wg/httpbis/about/

doesn’t indicate anything about the trajectory the SEARCH draft mentioned above is in, so I’m speaking from the experience of having done a similar thing in RFC 8132 (there, the same method is called FETCH):  It should not be long (6 months?  Surely < 12 months?) until that document is published, while the referencing documents are probably intended to be good for a decade or so.  So the perspective here should be that it’s done.

Grüße, Carsten

---

From: Rafal Pietrak <[email-masked]>
Date: Sat, 17 Jul 2021 13:39:13 +0200

Hello everybody,

Pls forgive me cutting in, where I'm not particularly competent, but
I've checked the link below, and read it from a perspective of a
"simple" home-page web writer. The comments bellow come from reading it
specifically from such perspective (i.e: what it gives web-page-authors).

The draft seam to define SEARCH as "something" that HTTP server may
receive "from the wild". When reading the proposal from
"home-page-author" perspective, one MUST assume, that all the SEARCH
requests will "get back" to the server as a result of interpretation of
just received by the browser a "home-page". IMHO this perspective is
missing from the draft.

I admit, that when fixed (as below), the SEARCH method could provide
functionality I was attempting to achieve with my cookie-draft.

>From 'home-page-writer' perspective, we assume, that the entire
selection of possible SEARCH requests, that a client browser could
submit to that server is provided to that browser by that server on the
just retrieved home-page (html-home-page).

If so, the questions are:
1. will it be possible to send to the browsers a "home-page" with some
of the <form method="POST"> turned into <form method="SEARCH">? It looks
like YES, but I'm not quite sure. So, at least a "none nominative"
example would be nice to have.
2. will it be possible to change some (or all, if provision just "some"
is unreasonable difficult) of the "<a href=''>", "<img src=>", "<link>",
etc, etc, from launching a GET requests, into launchinig a SEARCH
requests instead? This is not addressed by the draft, yet IMHO it would
be nice to have.
3. the draft mentions 'accept-search' being used by the server to
indicate it can accept SEARCH method. But from (1) and (2) above, it's
equally important, that the browser is able to notify server, that the
web-page returned can contain SEARCH requests ... like in <form
methos="SEARCH"> in (1) above. So, "accept-search" should also be send
to servers by complying browsers.

Then again, when server receives "accept-search" it may request browser
to turn "all the possible" href/src/source links from GET into SEARCH.
something like "meta-search" for the <HEAD> could do that.

That "meta-search" also could supplement SEARCH triggered by "a href"
(or others) with additional data.

the SEARCH draft mentions, that processing of SEARCH by the server will
not change state of the server. From those words I was missing an
emphesise, that in consequence, web browser submitting a SEARCH should
not alert user like they do when user requests "page-refresh", while
that page arrived as a response to POST request. Such clearification
would be desirable

I think there may be a problem with 3xxx redirect response as defined in
current version of the draft: "alterative URI from which the search
results can be retrieved using an HTTP GET request".

The problem may be at the point where one MUST change method from SEARCH
to GET. This may prove difficult when not all the SEARCH params fit into
GET request. So, such change request should probably be returned by the
server only under specific circumstances, and so as specific/uniq
redirect code; while "simple redirection" should NOT change methods
(SEARCH->GET).

-- 
Rafał Pietrak

---

From: Julian Reschke <[email-masked]>
Date: Sat, 17 Jul 2021 14:46:10 +0200

> just received by the browser a "home-page". IMHO this perspective is

It's not the only perspective.

> example would be nice to have.

That would require a change from the HTML spec.

> requests instead? This is not addressed by the draft, yet IMHO it would

No. GET remains the primary method for information retrieval.

> to servers by complying browsers.

Actually, the signal for the *method* is the "Allow" response header
field. "accept-search" would be about the supported formats.

> something like "meta-search" for the <HEAD> could do that.

Why would you want that?

> that page arrived as a response to POST request. Such clearification

That follows from the defintion of being a "safe" method.

> redirect code; while "simple redirection" should NOT change methods

A server would only do that if it actually can define a URI for the
final SEARCH result.

Best regards, Julian

---

From: Rafal Pietrak <[email-masked]>
Date: Sat, 17 Jul 2021 15:49:02 +0200

Hi,

> Am 17.07.2021 um 13:39 schrieb Rafal Pietrak:
[------------]
> It's not the only perspective.

That's for sure.

I've only got an impression, that this particular perspective was not

[-------------]
> That would require a change from the HTML spec.

OK. Somehow I've assumed, that GET/POST and HTML-FORM are "kind-of
inter-woven" with each other. Probably I've got the assumption a bit too
far.

Still, if there pops up another transport method (HTTP/SEARCH). It would
be nice for the payload specs to follow.

> No. GET remains the primary method for information retrieval.

Yes. But when another doors open, why keep it half closed. The SEARCH as
it is, could be beneficial for some, but with little extention it may be
far better for much more of us.

I just ment the above - see the potential.

[------------]
>> something like "meta-search" for the <HEAD> could do that.

to be able to supplement more parameters to GET (by replacing it behind
screens  with a SEARCH) with additional information (required for
successfully fetching the href/src link), without javascripting.

I can imagine, that it could also be a place to explicitly declare "on
the page" the list of cookies that this server expects with
GET/POST/SEARCH from this page. Such far reaching modification could
allow server some additional control over cookies at client browser
without javascritp ever touching them. ... like server saying: "from
this page, send only this list of cookies, no other".

I'm stressing "additional information", because:
1. SEARCH looks like a method with "capacity" similar to the POST method
2. I would expect GET getting obsoleted in time, with such capable
contender as SEARCH. (despite the fact, that it's currently the "primary
means")

[--------------]
> A server would only do that if it actually can define a URI for the

the problem is (can be) at the client side. I'm assuming here, that
initial SEARCH was originally "cooked" like a POST with large list of
parameters in multipart body. Now, if the client gets a redirect to GET,
all that parameters must end up at GET argument line. I expect, that
could be a problem in some corner cases.

-- 
Rafał Pietrak

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 23 Jul 2021 11:56:54 +1000

We seem to have converged on a resolution here:
  https://github.com/httpwg/http-core/pull/906

Please take a look and if you have questions, comments or objections, let us know.

Cheers,

> On 16 Jul 2021, at 5:06 pm, Mark Nottingham <[email-masked]> wrote:
>

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 23 Jul 2021 17:05:12 +1200

> Please take a look and if you have questions, comments or objections, let us know.

On this topic, Squid-6 is updated to enforce rejection of payloads for 
these methods in messages labeled as HTTP/1.0 or older.

Use of at least HTTP/1.1 and appropriate framing headers are required 
for these methods to transfer a payload through Squid.

Amos Jeffries
The Squid Software Foundation

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 23 Jul 2021 07:18:09 +0200

Hi Amos,

On Fri, Jul 23, 2021 at 05:05:12PM +1200, Amos Jeffries wrote:
> On this topic, Squid-6 is updated to enforce rejection of payloads for these
> methods in messages labeled as HTTP/1.0 or older.
> 
> Use of at least HTTP/1.1 and appropriate framing headers are required for
> these methods to transfer a payload through Squid.

Great, I think it's a very reasonable approach and I intend to do
the same (as well as enforcing close on T-E with 1.0 or with C-L
according to #879) in haproxy.

Willy