Subject: clarify some MUST requirements in HTTPbis part 1 section 3.3

From: Dmitry Kurochkin <[email-masked]>
Date: Tue, 29 Nov 2011 06:05:14 +0400

Hello.

HTTPbis part 1 section 3.3 has the following requirements:

  If more than one Transfer-Encoding header field is present in a
  message, the multiple field-values MUST be combined into one
  field-value, according to the algorithm defined in Section 3.2, before
  determining the message-body length.

  If a message is received that has multiple Content-Length header
  fields (Section 8.2) with field-values consisting of the same decimal
  value, or a single Content-Length header field with a field value
  containing a list of identical decimal values (e.g., "Content-Length:
  42, 42"), indicating that duplicate Content-Length header fields have
  been generated or combined by an upstream message processor, then the
  recipient MUST either reject the message as invalid or replace the
  duplicated field-values with a single valid Content-Length field
  containing that decimal value prior to determining the message-body
  length.

It is not clear whether proxies that do not reject the message MUST
replace the duplicated field-values in the forwarded message.  In other
words, are non-transforming proxies required to either reject or fix the
message?

Should the specs require connection closure to prevent message smuggling
in naive implementations that respond with an error to "reject the
message" but then keep the connection open/persistent?  The specs are
explicit about this in item #3 where they talk about "framing" error,
but not here, where essentially the same framing error is suspected if
the message is rejected.

Also, should the specs be more specific regarding what connection
(upstream or downstream) proxy is required to close?

  If this is a response message received by a proxy, the proxy MUST
  discard the received response, send a 502 (Bad Gateway) status code as
  its downstream response, and then close the connection.

Above can be interpreted that the downstream connection must be closed.
But the proxy should only close the upstream connection.  Moreover, the
"then" word is wrong there as there is no need to keep that connection
open while the 502 response is being sent.  The order of those actions
should be up to the proxy to decide.

The fixed text may read:

  If this is a response message received by a proxy, the proxy MUST
  discard the received response, close the upstream connection, and send
  a 502 (Bad Gateway) status code response as its downstream response.

Regards,
  Dmitry

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 29 Nov 2011 18:45:56 +1300

> replace the duplicated field-values in the forwarded message.  In other

How is "MUST ... replace the duplicated field-values with a single valid 
Content-Length field" unclear?

> words, are non-transforming proxies required to either reject or fix the

Yes. "MUST either ... or ...."

Depends on your philosophy IMO. If you want to be hardline and prefer 
reject over header alterations its allowed.

> but not here, where essentially the same framing error is suspected if

Good point.

The text you quoted above from paragraph 4 is only talking about cases 
where the values are identical (ie correct), just duplicated.

    GET / HTTP/1.1
    Content-Length: 0
    Content-Length: 0
    Content-Length: 0,0
    Host: example.com

There is no request smuggling here. Other requests may be pipelined 
before or after it perfectly fine under the semantics of treating each 
requests independently.

The smuggling risk only rises where conflicting or incorrect values are 
sent, which is the case described in item #3.

AYJ

---

From: Dmitry Kurochkin <[email-masked]>
Date: Tue, 29 Nov 2011 10:35:18 +0400

Hi Amos.

> Content-Length field" unclear?

I am concerned about the "prior to determining the message-body length"
part.  It may be interpreted as the proxy must remove duplicate header
values for it's internal use only (i.e. to determine the message-body
length) and can forward unmodified message with duplicate headers.  Same
for TE header.  In other cases (e.g. item #2), the spec is explicit:
modify before forwarding downstream.

> sent, which is the case described in item #3.

Indeed.  So even if proxy does not remove the duplicate CL values, it
still must correctly determine the message length per item #4 because
item #3 does not apply here.  In this case, "prior to determining the
message-body length" part in the paragraph 4 seems unneeded.

Regards,
  Dmitry

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 29 Nov 2011 07:38:22 +0100

Hi Amos,

> reject over header alterations its allowed.

I understand Dmitry's point. I think he's using a proxy which is more a
gateway in fact, which does very few things and just tries to stay in sync
with the stream and to avoid alterations as much as possible. That's what
is one in IPS, load balancers or any product which needs to read HTTP
without being intrusive. I've had the same experience with haproxy, where
HTTP was added as minimal processings on top of TCP before being improved
with time.

My personal rule of thumb on this is that if you don't touch anything, you
can't make things worse ; if you find that there is a real danger in what
you see, you should close ; if you find that what you read is not perfect
but should cause no risk (at worst just fail), then it's better not to
alter the stream.

In his situation, there is a difference between the interpretation of the
content-length and modifying it before forwarding it. If his product is
just forwarding the TCP stream and not producing the headers, he's not
violating the HTTP specs (otherwise routers would be too). However if he
detects something abnormal he can't understand, it would seem quite dangerous
to ignore it and let it pass.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 29 Nov 2011 07:43:43 +0100

Hi Dmitry,

On Tue, Nov 29, 2011 at 10:35:18AM +0400, Dmitry Kurochkin wrote:
(...)
> message-body length" part in the paragraph 4 seems unneeded.

I think the "prior to determining..." part comes from a general handling
of multiple headers. For numerical values, it is not a problem. However
some headers will have a different meaning when folded. For instance, an
Expires header that would be split between the day of week and the rest
could be valid only once folded :

    Expires: Tue
    Expires: 29 Nov 2011 06:40:52 GMT
vs
    Expires: Tue, 29 Nov 2011 06:40:52 GMT

In my opinion, having such guidelines in the spec is better than leaving
it to the implementer to guess them. For those who're unsure, they'll
write more reliable code. If you know what you're doing and are able to
guess that in your case the "prio to" is unneeded, then that's fine.

Regards,
Willy

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 29 Nov 2011 20:51:34 +1300

I don't see it that way.  If the intermediary owns the TCP connection to 
the client, it is the emitter.

HTTP explicitly prohibits sending multiple Content-Length headers.

Therefore an intermediary must not send more than one.  Regardless of 
what it receives.

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 29 Nov 2011 11:48:48 +0100

Hi Adrien,

> Therefore an intermediary must not send more than one.  Regardless of 

It's not only a matter of whether it owns the TCP connection or not, but
also what it does with it. A basic TCP proxy owns the connections too and
becomes the emitter, still it cannot claim to be an HTTP agent and even
less to be able to fix complex protocol issues.

In my opinion, the difference lies on who produces the contents. If a proxy
builds a complete request from a list of headers, it has no excuse for not
producing them right. If a proxy just buffers TCP payload until it sees a
double CRLF, makes a few checks there and lets them go, we cannot expect it
to fix such issues.

Of course everything is not that black and white but it is easy to spot
some valid uses of every variations and to act responsibly in every case.

Regards,
Willy

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 29 Nov 2011 23:55:56 +1300

I can understand why someone would code some software that way, but I 
don't think it's correct to call that a proxy.  At least it's not HTTP 
compliant.

It could be argued that strategy of proxy operation is simply not valid.

Regards

Adrien

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 29 Nov 2011 14:32:09 +0100

> don't think it's correct to call that a proxy.  At least it's not HTTP 

I 100% agree with you, that's why I stated that it was more of a gateway than
a proxy.

> It could be argued that strategy of proxy operation is simply not valid.

Agreed too! The term "proxy" is often used when connections are
re-established (TCP proxies) but I regularly repeat that in the HTTP
terminology that's not enough to be called a proxy. An HTTP proxy is a
well-defined entity that cannot simply be assumed by a TCP proxy.

Regards,
Willy

---

From: Alex Rousskov <[email-masked]>
Date: Tue, 29 Nov 2011 08:58:04 -0700

> well-defined entity that cannot simply be assumed by a TCP proxy.

The scope of this thread question is not a TCP proxy. The core of the
concern is a regular HTTP proxy. Some HTTP proxies prefer to forward
message headers "as is" when possible rather than always recreating the
forwarded message from internal metadata. Such proxies may still be
compliant and interoperable if the specs are clear when it comes to
forwarded message manipulation.

The current requirements do not make it clear whether a compliant HTTP
proxy may, for example, leave "Content-Length: 100,100" in the message
it forwards while correctly interpreting that header as "Content-Length:
100". The uncertainty exists because the requirement uses the word
"replace" but talks about message interpretation (which does _not_
require changing the forwarded message).

  Requirement: MUST replace the duplicated field-values with ...
  Scope or timing: prior to determining the message-body length.

Possible solutions:

1) If the intent is to require that proxies fix messages they forward,
then the requirement should be polished and split into two:

  MUST interpret the duplicated field-values as ...
  MUST replace such Content-Length header(s) with ...

2) If the intent is to require correct interpretation without mandating
message fixes, then the requirement should be polished by changing
"replace with" to "interpret as".

What was the intent? Do we want to require all proxies to fix malformed
Content-Length headers in forwarded messages?

Thank you,

Alex.