Subject: HTTPbis -10 drafts published

From: Julian Reschke <[email-masked]>
Date: Tue, 13 Jul 2010 08:18:10 +0200

Hi,

we have published a snapshot as draft-ietf-httpbis-*-10 yesterday.

All changes should be mentioned in the changes sections:

<http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-10#appendix-D.11>

<http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-10#appendix-C.11>

<http://tools.ietf.org/html/draft-ietf-httpbis-p3-payload-10#appendix-E.11>

<http://tools.ietf.org/html/draft-ietf-httpbis-p4-conditional-10#appendix-C.11>

<http://tools.ietf.org/html/draft-ietf-httpbis-p5-range-10#appendix-D.11>

<http://tools.ietf.org/html/draft-ietf-httpbis-p6-cache-10#appendix-C.11>

<http://tools.ietf.org/html/draft-ietf-httpbis-p7-auth-10#appendix-C.11>

Furthermore, 
<http://tools.ietf.org/html/draft-ietf-httpbis-method-registrations-04> 
was updated to include recently published RFCs describing the methods 
PATCH, BIND, UNBIND and REBIND.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 14 Jul 2010 15:32:32 +1000

Thanks, Julian.

Everyone, please review these changes; the intent is to close these issues with this round of drafts:
  http://trac.tools.ietf.org/wg/httpbis/trac/report/15

Cheers,

On 13/07/2010, at 4:18 PM, Julian Reschke wrote:

> Furthermore, <http://tools.ietf.org/html/draft-ietf-httpbis-method-registrations-04> was updated to include recently published RFCs describing the methods PATCH, BIND, UNBIND and REBIND.
> 
> Best regards, Julian
> 

--
Mark Nottingham     http://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 14 Jul 2010 08:50:33 +0200

Hi,

> <http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-10#appendix-C.11>

This one is still rather vague about the CONNECT method (7.9).
It says :

   This specification reserves the method name CONNECT for use with a
   proxy that can dynamically switch to being a tunnel (e.g., SSL
   tunneling [RFC2817]).

It's still not easy to implement HTTP-compliant intermediaries based on
this, especially since the RFC above is cited as an example of usage,
and it does not seem very obvious to check for the TLS RFC to find the
semantics of an HTTP method.

There is also an important aspect to consider which are not specified
even in RFC2817 : what to do with the body if any is specified in the
request or in the response ? (I've seen some proxies return
"Content-length: 0" in response to a CONNECT, maybe they've encountered
issues with some old user-agents). While a proxy won't probably care, a
reverse-proxy based load balancer sitting in front of a proxy cares a bit
more. For instance, let's suppose the reverse proxy is forwarding a
request body advertised in the content-length header, as allowed by
httpbis-p1#3.3. Suddenly, the proxy responds with "200 established".
Should the reverse proxy silently consume the remaining data, should it
pass them to the proxy assuming they will be considered as part of the
tunnelled data ? Should it break the connection since after all it's the
client's fault ? When it happens in the response, it may cause the reverse
proxy to forward the response body until the advertised number of bytes is
reached, and only then the tunnel would be established. But the server
might be waiting for some upstream tunnelled data before sending more.

So shouldn't we simply state that both the request using a CONNECT method
and its response which indicates the tunnel is established MUST NOT
contain a body and that clients, intermediates and proxies MAY reject
such messages ?

Since TLS is not the only protocol using CONNECT anymore (even WebSocket
intends to use it), I suggest that we move the description from the
RFC2817 to httpbis-p2 :

  5.2 Requesting a Tunnel with CONNECT

   A CONNECT method requests that a proxy establish a tunnel connection
   on its behalf. The Request-URI portion of the Request-Line is always
   an 'authority' as defined by URI Generic Syntax [2], which is to say
   the host name and port number destination of the requested connection
   separated by a colon:

      CONNECT server.example.com:80 HTTP/1.1
      Host: server.example.com:80

   Other HTTP mechanisms can be used normally with the CONNECT method --
   except end-to-end protocol Upgrade requests, of course, since the
   tunnel must be established first.

   For example, proxy authentication might be used to establish the
   authority to create a tunnel:

      CONNECT server.example.com:80 HTTP/1.1
      Host: server.example.com:80
      Proxy-Authorization: basic aGVsbG86d29ybGQ=

   Like any other pipelined HTTP/1.1 request, data to be tunneled may be
   sent immediately after the blank line. The usual caveats also apply:
   data may be discarded if the eventual response is negative, and the
   connection may be reset with no response if more than one TCP segment
   is outstanding.

  5.3 Establishing a Tunnel with CONNECT

   Any successful (2xx) response to a CONNECT request indicates that the
   proxy has established a connection to the requested host and port,
   and has switched to tunneling the current connection to that server
   connection.

   It may be the case that the proxy itself can only reach the requested
   origin server through another proxy.  In this case, the first proxy
   SHOULD make a CONNECT request of that next proxy, requesting a tunnel
   to the authority.  A proxy MUST NOT respond with any 2xx status code
   unless it has either a direct or tunnel connection established to the
   authority.

   An origin server which receives a CONNECT request for itself MAY
   respond with a 2xx status code to indicate that a connection is
   established.

   If at any point either one of the peers gets disconnected, any
   outstanding data that came from that peer will be passed to the other
   one, and after that also the other connection will be terminated by
   the proxy. If there is outstanding data to that peer undelivered,
   that data will be discarded.

Any thoughts ?

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 14 Jul 2010 17:01:39 +1000

Hi Willy,

>   proxy that can dynamically switch to being a tunnel (e.g., SSL

I was just looking at a couple of bookmarked e-mails regarding CONNECT, and thinking that taking responsibility for it from 2817 is something we should discuss in Maastricht; we already update it for other reasons, and several people have complained about this over the years.

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 14 Jul 2010 09:28:13 +0200

Hi,

> <http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-10#appendix-D.11>

In this one we would need to add a bit of clarifications about the
value domains that may be used for integers. I have identified 2
situations where we can encounter integer overflows, with either
just impossibility to transfer data, or more important, security
issues :

- 6.2.1. Chunked Transfer Coding
  chunk-size     = 1*HEXDIG

- 9.2. Content-Length
  Content-Length-v = 1*DIGIT

In both cases, we don't indicate any minimal size requirement.
It looks like both values are generally assumed to be 31 or 32
bits depending on signedness. BTW, I've once been unable to
download a DVD image from a site because the announced content
length was wrong due to integer overflow. I don't remember what
server it was though :-/

I think it is difficult to ask every implementation to support
at least XXX bits, but we should at least add as a strict
requirement that any agent along the chain must detect integer
overflows when parsing or printing these values, and must abort
processing if this happens.

My main concern here is a security issue. If an intermediary
supports more bits than a server, it may forward a larger body
than the server believes is sent, and push "invisible" requests
to the server. For instance :

   POST /harmless HTTP/1.1
   Host: victim
   Content-length: 4294967302         # = 2^32 + 6

   Hello
   POST /login.jsp HTTP/1.1
   Host: victim
   Content-length: 20

   user=foo1&pass=bar1
   POST /login.jsp HTTP/1.1
   Host: victim
   Content-length: 20

   user=foo2&pass=bar2

etc...

The reverse-proxy reads "Content-length: 4294967302", parses it as
a 64 bit value and starts forwarding that many bytes. The server
parses it as a 32 bit value, overflows and expects 6 bytes
(4294967302 mod 2^32 = 6). It then reads the "Hello\n" message and
takes the next pending data as a new valid request.

The same issue happens if the intermediary parses the value as an
unsigned 32bit value while the server parses it as a signed 32 bit
value, which will be negative for lengths >= 2^31. #9.2 states
that "Any Content-Length greater than or equal to zero is a valid
value". The risk I see here is that this is translated in the code
as "Consider the content-length only if >= 0", resulting in no
body for negative values. The same problem as above then appears.

I'm just realizing something else. Some implementers might be
tempted to use strtoul() or equivalent to reliably parse these
values and detect errors. The definition of those values with
1*DIGIT suggests that leading zeroes are allowed. Special care
must be taken to ensure that the decoding is forced to base 10,
otherwise a number such as 0123 may be automatically interpreted
as the octal value for the decimal 83.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Wed, 14 Jul 2010 09:42:11 +0200

> bits depending on signedness. BTW, I've once been unable to

How did you come to that conclusion?

> length was wrong due to integer overflow. I don't remember what

That's a server bug. If the content length exceeds what the server can 
express in Content-Length, it should switch to chunked encoding.

> processing if this happens.

I think that requirement is already implicit; if you can't parse a 
length, don't pretend you can.

> ...

They are.

> as the octal value for the decimal 83.

That's true, but again, that's a quality of implementation issue.

We've had similar discussions before: it's not clear that considerations 
like these belong into the actual HTTP spec. On the other hand, a 
separate document discussing issues like these could certainly be 
written (IETF Informational, or even somewhere else?).

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 14 Jul 2010 10:16:25 +0200

> How did you come to that conclusion?

Observations, tests and implementations. When testing some products,
it's often very instructive to try to send them too large content-length
values or chunked encodings. And implementations have to choose a type
for their value anyway. It's common to find "int" or "unsigned int"
there. Even in haproxy, I chose "unsigned int" (32 bit) for the chunk
size, and positive 63-bit value for the content length. At least I've
been very careful to detect all overflows everywhere, but this is also
because I'm sensibilized to the security implications. Other implementers
may simply consider that "that large is enough and if it fails above that
it does not matter".

> express in Content-Length, it should switch to chunked encoding.

Agreed, and that might also be why I've only seen that once ;-)

> length, don't pretend you can.

I agree with that, the issue is being aware that you can't parse.
I've seen HTTP parser code in enterprises... Believe me, sometimes
you're thankful to the application server below which already does
some cleaning. Most people are not aware of integer overflows and
will only think with "large enough" quantities. Above that is
infinite. The fact that an apparently large number decodes as a
small one is non-sense for some. Basically, you'd find things such
as :

   l = atol(get_header("content-length"));

And once you tell the guy that there is no error check and he even
supports negative values, he would simply switch to :

   l = strtoul(get_header("content-length"), &err, 0);

(which now supports octal decoding).

Since there are already some precisions in the draft, such as "any
value >= 0 is valid", I think it makes sense to add a few words at
some places to enforce certain things. Something like saying that
Content-length is a DECIMAL representation might ring a bell to the
implementer.

> That's true, but again, that's a quality of implementation issue.

I completely agree. But if we're purist, it's noted nowhere that
the values are expressed in decimal form and must be decoded as
such ;-) The only thing that is indicated is that a DIGIT represents
a decimal 0-9.

> written (IETF Informational, or even somewhere else?).

I think we could have something generic somewhere. But there are a
few points in HTTP which, if poorly implemented, may have important
side effects. A single sentence such as "Implementations MUST detect
integer overflows and integer parsing errors" for the Content-Length,
Bytes-range and chunk size is not too much and can help getting more
reliable and interoperable implementations.

Best regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 14 Jul 2010 10:55:14 +0200

Hi,

> <http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-10#appendix-D.11>

Here I have found something that ought to be clarified concerning
the Connection header :

9.1. Connection
   HTTP/1.1 proxies MUST parse the Connection header field before a
   message is forwarded and, for each connection-token in this field,
   remove any header field(s) from the message with the same name as the
   connection-token.  Connection options are signaled by the presence of
   a connection-token in the Connection header field, not by any
   corresponding additional header field(s), since the additional header
   field may not be sent if there are no parameters associated with that
   connection option.

   Message headers listed in the Connection header MUST NOT include end-
   to-end headers, such as Cache-Control.

The last sentence is already very important, but some side effects
remain on some implementations, because it is not stated that only
headers that were given by the client must be removed. If you take
Apache 2.2 as a proxy for instance, by default it will add an
"X-Forwarded-For" header when forwarding the connection to the
server, to indicate the client's address. If the client says
"Connection: X-Forwarded-For", then this header is removed from the
output and the server does not get the client's address. I've not
checked the code, but I think this is because the header cleaning
happen just before forwarding the connection, and after header
addition. This can permit a client to alter the semantics of the
communication between a proxy and a server, possibly bypassing some
filtering or hiding its activities. I think that adding a sentence
such as the following would be fine :

   If an HTTP/1.1 proxy intends to modify or add headers to the
   message being forwarded, it may only do so after the headers
   above have been removed.

I've also tried to remove "Connection" (which is a hop-by-hop header)
with Apache and fortunately it did not work. Out of curiosity I tried
with "Content-Length" and "Host" and they did not work either.
However, when I try to remove "Transfer-Encoding" (hop-by-hop) on a
POST with empty body, I observe that "Content-Length: 0" is not added
to the request, which is harmless (the request will be rejected due
to missing content length).

I've just found that other end-to-end headers such as "cache-control",
"content-encoding" etc... can still be removed via Apache, possibly
leading to differences in contents interpretation between the proxy
and the server (eg: when content filtering is performed), though this
is purely implementation-specific. However, implementation-specific
differences or limitations sometimes indicate a difficulty in fully
understanding or respecting a standard.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Wed, 14 Jul 2010 11:09:15 +0200

> may simply consider that "that large is enough and if it fails above that

Well, if it fails in the right way it's ok (such as failing with a 500), 
simply being a known limitation. It's only problematic if the error 
condition isn't detected and the implementation just proceeds with bad data.

> Agreed, and that might also be why I've only seen that once ;-)

Oh, I've seen those, too (did lots of >2GB testing with WebDAV many 
years ago). Back then, the Java servlet API didn't properly support 
content lengths > 2GB. Hopefully this has been fixed by today.

>     l = strtoul(get_header("content-length"),&err, 0);
>
> value>= 0 is valid", I think it makes sense to add a few words at

We had that discussion before (in the context of Javascript, see 
<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/161>). The ABNF 
already says it.

It's very hard to optimize the spec for people who don't read it 
properly, nor their language/API documentation.

> such ;-) The only thing that is indicated is that a DIGIT represents

I'd prefer not to be purist.

> reliable and interoperable implementations.

I don't believe that at all, sorry. People who get this wrong today get 
it wrong because they are sloppy programmers, not because of what the 
spec says.

There are many many things the spec *could* mention. Expecting 
overflows. Expecting parse errors. Treating absent parameters when the 
ABNF disallows them. Treating multiple header instances when only one is 
allowed. This is open-ended.

It doesn't *need* to be in *this* spec. But that doesn't mean that it 
would be a bad idea to work on a document like that.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 14 Jul 2010 11:41:21 +0200

> condition isn't detected and the implementation just proceeds with bad data.

Exactly !

> <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/161>). The ABNF 

I've checked before reporting the issue but did not notice it :-(
I've checked again, and all I see is "1*DIGIT". I've checked RFC5234
too just in case, but it does not state either how to parse digits.

> properly, nor their language/API documentation.

To be honnest, here in my opinion we're in the "well known" area.
I think I'm trying to read the specs enough to find what I must do
and what I might receive based on what others might send, but if I
had a doubt about the base of a content-length for instance, after
a check in the spec I wouldn't find the response and I would have
to check what existing implementations do.

But seeing your patch above, and given the number of places where
"decimal" was explicitly stated, I think that a single sentence
along with the ABNF could state "All 1*DIGIT character sequences
are to be interepreted as decimal values".

> it wrong because they are sloppy programmers, not because of what the 

Well, one of the difficulties with HTTP is that no limit to anything
is specified. That's what makes it that open, but also what causes
so many arbitrary choices. I regularly hear questions such as "what's
the max length a URL can take ?" or "what's the max length of a header ?".
When I reply there's no such limit, people are embarrassed and have to
resort to the "large enough" principle, which generally means using a
type which can hold values that cannot be reached. I agree the spec
cannot correct such behaviours, but when some things are well-known
and some errors not uncommon, it does not cost much to help implementers
not do the same mistakes again.

> would be a bad idea to work on a document like that.

I see, maybe something like "best practices for reliable HTTP implementations" ?

Regards,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 15 Jul 2010 10:24:48 +1000

http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-10#section-4.1.2 :

>    HTTP does not place a pre-defined limit on the length of a request-
>    target.  A server MUST be prepared to receive URIs of unbounded
>    length and respond with the 414 (URI Too Long) status if the received
>    request-target would be longer than the server wishes to handle (see
>    Section 8.4.15 of [Part2]).
> 
>    Various ad-hoc limitations on request-target length are found in
>    practice.  It is RECOMMENDED that all HTTP senders and recipients
>    support request-target lengths of 8000 or more OCTETs.

Cheers,

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 15 Jul 2010 12:45:19 +1200

that's quite an interesting scenario

if a proxy were to receive a request message with say

Connection: content-type

in it, what do you think should the proxy do?

a) ignore it (not remove Content-Type)
b) reject the message (client attempted exploit)
c) something else

it may be clear enough for Content-Type, but what about some other 
header (e.g. header not known about by the proxy).  Should we have a 
requirement that a proxy should reject any message that has a token in 
the Connection header that is not a known hop-by-hop header?

regards

Adrien

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 15 Jul 2010 10:50:28 +1000

What kind of exploit? Remember that Connection only affects the headers in *this* message; i.e., in your scenario, the client couldn't affect the response headers.

On 15/07/2010, at 10:45 AM, Adrien de Croy wrote:

> it may be clear enough for Content-Type, but what about some other header (e.g. header not known about by the proxy).  Should we have a requirement that a proxy should reject any message that has a token in the Connection header that is not a known hop-by-hop header?
> 
> regards
> 
> Adrien
> 
>

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 15 Jul 2010 13:12:50 +1200

the client would be affecting the headers in the message forwarded by 
the proxy to the next hop.

Actually it makes me query whether it is even necessary to process the 
Connection header in this way, since all hop-by-hop headers must not be 
forwarded by intermediaries anyway (they can be re-added if the 
intermediary understands them).

All this covers therefore is a client-specified way to specify removal 
of headers the proxy doesn't recognise as hop-by-hop, which it seems is 
a bad idea?

Regards

Adrien

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 14 Jul 2010 18:17:45 -0700

> in it, what do you think should the proxy do?

Delete the content-type header, as required by HTTP/1.1.

> it may be clear enough for Content-Type, but what about some other header (e.g. header not known about by the proxy).  Should we have a requirement that a proxy should reject any message that has a…

That would be the complete opposite of the reason we have the
Connection header -- to indicate what headers are hop-by-hop.
We needed it precisely to indicate hop-by-hop extensions.

There is absolutely no risk in following the instruction
exactly as indicated.  The sender already has control over the
bits being sent, and your proxy should be enforcing its constraints
on what to forward *after* the message is processed for forwarding.

....Roy

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 15 Jul 2010 13:50:09 +1200

ok, but what about from RFC2616 S 14.10

there are only 4 types of header (AFAICT)

general-header
request-header
response-header
entity-header

any unknown header is an extension header, and is treated as an entity 
header.

All entity headers are end to end are they not?

Therefore it's illegal to specify any entity header in the Connection 
header.

Adrien

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 15 Jul 2010 12:25:33 +1000

Allowing specification of new hop-by-hop headers allows deployment of new connection-oriented features. 

On 15/07/2010, at 11:12 AM, Adrien de Croy <[email-masked]> wrote:

> 
> 
> the client would be affecting the headers in the message forwarded by the proxy to the next hop.
> 
> Actually it makes me query whether it is even necessary to process the Connection header in this way, since all hop-by-hop headers must not be forwarded by intermediaries anyway (they can be re-added if the intermediary understands them).
> 
> All this covers therefore is a client-specified way to specify removal of headers the proxy doesn't recognise as hop-by-hop, which it seems is a bad idea?
> 
> Regards
> 
> Adrien
>

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 14 Jul 2010 19:53:37 -0700

On Jul 14, 2010, at 6:50 PM, Adrien de Croy wrote:

> any unknown header is an extension header, and is treated as an entity header.

No.  It is parsed as an entity-header.

> All entity headers are end to end are they not?

Anything in the spec that says something is end-to-end or
hop-by-hop will be rewritten soon.  Fields are hop-by-hop
when they are listed in Connection (or are Connection).
All others are end-to-end.

> Therefore it's illegal to specify any entity header in the Connection header.

That does not change how they are handled.

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 15 Jul 2010 06:31:26 +0200

> All others are end-to-end.

So probably the spec should enumerate the well-known headers
that cannot be hop-by-hop and that must not be listed in
Connection. At the moment the spec only says "All other
headers defined by HTTP/1.1 are end-to-end headers", which
is not much practical for implementers, considering the size
of the spec and the high risk of missing a few ones.

And that leaves the question open about what to do if those
headers are found. The worst cases I thought about (Host,
Content-Length) don't work with Apache at least, but they
may cause real trouble on proxies which blindly apply the
spec.

Willy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 15 Jul 2010 12:05:12 -0700

> may cause real trouble on proxies which blindly apply the

They should not cause any problems, at least no more so than
receiving invalid messages.  If the software can't handle weird
things then it doesn't belong on the Internet.

We don't need to protect implementations from their own stupidity.
It is far better for the protocol to be self-descriptive, so that
we can extend it over time.

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 15 Jul 2010 21:33:06 +0200

> things then it doesn't belong on the Internet.

It's not that simple. I have an example of a customer who uses front
Apache reverse-proxies to perform security controls and to only
let "clean" requests pass through. Those Apaches also add some
headers to the requests being forwarded to the servers for logging
purposes, and it is the only way to reach the servers. Due to that
implementation mistake, it is possible (and I've tested) for the
client to make the reverse proxy remove those headers it had just
added, so that the end point finally does not get the information
it should have unconditionally got.

Now, if we want to be fair, there are two points here which are
causing that issue :

  - apache's header removal does not happen in the appropriate
    order.

  - apache is used as a reverse-proxy (and is often used that
    way) but it follows a proxy behaviour instead of a gateway
    behaviour. But I suspect that when they began, the difference
    was not clear yet between the two.

> We don't need to protect implementations from their own stupidity.

I'd not call that stupidity. The spec is huge and very complete,
and as such it's not easy to get it right first, especially since
some information are spread all over it (eg: "all other headers
defined by HTTP/1.1"). It's got *a lot* better since RFC2616, but
still it's sometimes hard to know whether your implementation is
compliant or not, depending how a paragraph is read, and I have a
lot of respect for the persons who spend time trying to follow it
as closely as possible.

Well, maybe those could be some points to add to a separate draft to
try to improve the robustness of new implementations by suggesting
how to handle a bunch of corner cases.

Regards,
Willy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 15 Jul 2010 16:28:04 -0700

> it should have unconditionally got.

Yes, but why is that a problem?  First, the process adding headers
should have already removed the Connection header received -- otherwise
it isn't doing its job.  Second, even without fixing that bug, the
result is fail safe -- the proxy won't be able to forward what it
generated.

>  - apache's header removal does not happen in the appropriate

The order depends on when the module does its stuff, not on
something inherent in Apache.  It is the security-checking module's
responsibility to do the removal earlier (or schedule its additions
later) if that is desired.

>    was not clear yet between the two.

It depends on which module is used for that purpose, but yes
the mod_proxy stuff makes for a poor gateway.  The difference
was well known at the time -- I should have vetoed the reverse
proxy features when they were added (they belong in a separate
module).

> how to handle a bunch of corner cases.

It wouldn't be confusing at all if it were not for all the
extra requirements that gave types to headers (like hop-by-hop).
They make it sound like there is some header-aware engine going
through and checking the types.  There is no such thing in a well
written intermediary -- every decision should be based on a user
config or the self-descriptive message.

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Jul 2010 06:55:49 +0200

> result is fail safe -- the proxy won't be able to forward what it

It's not a problem from an HTTP point of view, the request is valid. It's
a problem because some mandatory information which is unconditionally set
by the proxy regardless of the client's request can still be removed by
the client (eg: X-Forwarded-For, Host, X-Forwarded-Host, ...). When the
next hop server takes a decision based on these info, this can become
dangerous (here the main issue was not getting the client's IP in the
server logs). We could for instance imagine an HTTP/1.0 compatible
server which does virtual hosting and which gives access to the base dir
of the virtual servers when no Host was specified. This is just an example,
I'm not saying this is what happens. The issue is mostly that the client
can control some aspects of the other side connection in an unexpected
manner.

Someone else (Adrien ?) suggested it might be problematic with caches.
I'm wondering what can happen if the client does that on Accept-Language
for instance. Sometimes the cache will index the headers from the real
request, but since the server won't get it it may return a different
version of the document, which will be cached associated with different
request headers than what generated the response. Same for the Host
field, eventhough it's less likely that the request will be accepted.

> later) if that is desired.

I *think* that x-forwarded-* are managed by the mod_proxy module
itself, though I may be wrong since I don't know apache well enough.
Thus, I'm not sure we could fix this in the config by just moving
LoadModule directives around.

> proxy features when they were added (they belong in a separate

I agree.

> through and checking the types.

Yes, indeed, that's how I parse it. My understanding is that as an
implementer, I should read the whole doc, and write down all header
names see at least once, then eliminate from that list the ones
listed as hop-by-hop.

> There is no such thing in a well

In fact, in my opinion on a real proxy (I mean an outgoing proxy),
it should not be much of a problem if the user can control what
goes out (except maybe for what goes into the cache). I tried to
imagine what could happen if the user sent a GET+content-length+
and got it removed to unveil a second request, etc... so that it
could bypass some mandatory filtering, but I don't see how it
could use that to gain unwanted capabilities.

On the reverse proxy side it's different because what we expect
from a gateway is to perform strong checks before feeding the
servers with safe requests. But here the issue is not caused by
what the spec says (since gateways are not concerned by the rule
on the Connection header), but rather by the use of a component
for the wrong job.

I still just have a few doubts for the side effects on caching proxies,
but I'm not skilled in this area and don't know well what part of the
request may have an effect on what is really cached (eg: Accept-Language,
Byte-Range, ...)

Best regards,
Willy

---

From: "Thomson, Martin" <[email-masked]>
Date: Fri, 16 Jul 2010 13:08:39 +0800

> It's not a problem from an HTTP point of view, the request is valid. It's

I don't see why you are belabouring the point.  Especially when it is predicated on a software error.

If the client wants a header to go to the origin server, don't use Connection.  If a proxy wants to ensure a particular header is set in a particular way, set that header.

Yes, you can probably screw up all sorts of things by doing all sorts of dumb things.  It's pointless compromising the integrity of a perfectly sound specification so that you can club a few idiots over the head with it.  That's a self-defeating principle - the idiots will simply find another dumb thing to do that you couldn't predict.

Your solution here is to submit a bug report.

--Martin

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 16 Jul 2010 17:16:13 +1200

just to belabour the point

RFC2616 S 14.10

This should either

a) be removed from HTTPbis if we are to take the approach suggested by 
Roy; or
b) be elaborated further to specify what an agent should do upon 
receiving a message that violates this MUST level requirement

Adrien

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 15 Jul 2010 22:39:01 -0700

>   to-end headers, such as Cache-Control.

Ah, excellent example of a nonsense requirement.  Note how it is phrased
as an existence test rather than a requirement on senders (not to send)
and on recipients (what the heck we are supposed to do when we receive it).

> This should either
> 
> a) be removed from HTTPbis if we are to take the approach suggested by Roy; or
> b) be elaborated further to specify what an agent should do upon receiving a message that violates this MUST level requirement

Well, given that any header sent from the one end and received at the
other end is an end-to-end header, and that the only thing that
makes a header other than Connection a "hop-by-hop" header is the
fact that it is listed in Connection (this was part of my original
HTTP/1.1 proposals regarding keep-alive), we should remove that
requirement because it requires omnipotence on the part of implementations
(they must know the purpose of all future headers) and serves no
useful purpose (because the requirement to not forward any header
named in Connection will override this pseudo-requirement that
says the field value MUST NOT exist).

In fact, I already made this comment in ticket #60, but it was
part of an editorial discussion on a separate issue.

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Jul 2010 08:35:01 +0200

> If the client wants a header to go to the origin server, don't use Connection.  If a proxy wants to ensure a particular header is set in a particular way, set that header.

I have no problem with that, but while it's easy to read it that way
once you got trapped, you should at least admit that it's not easy
to guess such side effects when blindly applying the spec :

   The "Connection" general-header field allows the sender to specify
   options that are desired for that particular connection and MUST NOT
   be communicated by proxies over further connections.
   ...
   HTTP/1.1 proxies MUST parse the Connection header field before a
   message is forwarded and, for each connection-token in this field,
   remove any header field(s) from the message with the same name as the
   connection-token.

So if the proxy decides to force a header to be set despite this header
being listed in Connection, it looks like it violates the spec.

As long as we'll have unclear situations where implementers have to
chose between strictly respecting the spec or doing the right thing,
we'll get such implementation issues. Of course these issues can be
fixed afterwards, but don't you think that the spec's purpose is also
not to direct implementers in the wrong direction ?

Also :

Almost all headers are supposed to be end-to-end. In fact, the hop-by-hop
ones are the ones enumerated in this header. If a client lists Content-Length
in Connection, does that make it a hop-by-hop header ? And if so, how should
a proxy proceed ? It's sort of an egg-and-chicken game. We're saying the proxy
"hey, I'm sending you a header I want you to remove, and since it's listed in
my Connection header, you are supposed to believe me and there is no risk that
the request you forward is not valid".

Maybe a single sentence could be added to clarify all things :

    HTTP/1.1 proxies MUST parse the Connection header field before a
    message is forwarded and, for each connection-token in this field,
    remove any header field(s) from the message with the same name as the
-   connection-token.
+   connection-token, unless that header was set by the proxy itself, eg.
+   for logging purposes or message integrity.

> Yes, you can probably screw up all sorts of things by doing all sorts of dumb things.  It's pointless compromising the integrity of a perfectly sound specification so that you can club a few idiots…

I'm amazed by the number of times I read that implementers are idiots or
stupid. If you tell them to implement vague things they will all do stupid
things. The people who define the spec have a very clear understanding of
its semantics, so there are a lot of obvious things to them. Implementers
have to learn these semantics by reading the spec. If the spec is unclear
or self-contradictory, you'll get as many incompatible implementations as
you have readers. From my understing, the goal of this WG is to spot unclear
areas which can lead to wrong implementations and to clear them. When we
find wrong implementations which can be explained by how the spec is read,
I think we should at least wonder if there is no other way to explain the
rules and make them more explicit.

> Your solution here is to submit a bug report.

That's what I intend to do, but the authors will have the right to
say "hey we've strictly followed the spec".

Willy