Subject: Re: clarify some MUST requirements in HTTPbis part 1 section 3.3

From: Mark Nottingham <[email-masked]>
Date: Thu, 1 Dec 2011 21:30:45 +1100

I think that's where we left it.

This style of implementation often changes the stream; e.g.,

  Connection: close
to 
  Connxxtion: close

and similar. Likewise, implementations like those discussed could do:

  Content-Length: 100,100
to
  Content-Length: 100    <-- whitespace

Cheers,

On 30/11/2011, at 2:58 AM, Alex Rousskov wrote:

> On 11/29/2011 06:32 AM, Willy Tarreau wrote:
>> On Tue, Nov 29, 2011 at 11:55:56PM +1300, Adrien de Croy wrote:
>>> 
>>> 
>>> 
>>> I can understand why someone would code some software that way, but I 
>>> don't think it's correct to call that a proxy.  At least it's not HTTP 
>>> compliant.
>> 
>> I 100% agree with you, that's why I stated that it was more of a gateway than
>> a proxy.
>> 
>>> It could be argued that strategy of proxy operation is simply not valid.
>> 
>> Agreed too! The term "proxy" is often used when connections are
>> re-established (TCP proxies) but I regularly repeat that in the HTTP
>> terminology that's not enough to be called a proxy. An HTTP proxy is a
>> well-defined entity that cannot simply be assumed by a TCP proxy.
> 
> 
> The scope of this thread question is not a TCP proxy. The core of the
> concern is a regular HTTP proxy. Some HTTP proxies prefer to forward
> message headers "as is" when possible rather than always recreating the
> forwarded message from internal metadata. Such proxies may still be
> compliant and interoperable if the specs are clear when it comes to
> forwarded message manipulation.
> 
> The current requirements do not make it clear whether a compliant HTTP
> proxy may, for example, leave "Content-Length: 100,100" in the message
> it forwards while correctly interpreting that header as "Content-Length:
> 100". The uncertainty exists because the requirement uses the word
> "replace" but talks about message interpretation (which does _not_
> require changing the forwarded message).
> 
>  Requirement: MUST replace the duplicated field-values with ...
>  Scope or timing: prior to determining the message-body length.
> 
> Possible solutions:
> 
> 1) If the intent is to require that proxies fix messages they forward,
> then the requirement should be polished and split into two:
> 
>  MUST interpret the duplicated field-values as ...
>  MUST replace such Content-Length header(s) with ...
> 
> 2) If the intent is to require correct interpretation without mandating
> message fixes, then the requirement should be polished by changing
> "replace with" to "interpret as".
> 
> 
> What was the intent? Do we want to require all proxies to fix malformed
> Content-Length headers in forwarded messages?
> 
> 
> Thank you,
> 
> Alex.
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 01 Dec 2011 09:17:44 -0700

>   Content-Length: 100    <-- whitespace

Absolutely, but the question paused by this thread remains: Are proxies
_required_ to fix messages they forward? Or are they just required to
interpret the messages they receive correctly? The HTTPbis specs do not
give a clear answer, IMO, and should be adjusted one way or the other.

I can propose specific wording changes if it helps but (a) I do not want
to spend time on that if there is consensus that the current wording is
good enough and (b) I need to know which way to polish (to require fixes
or remove the implication that they are required).

Thank you,

Alex.

---

From: Patrick McManus <[email-masked]>
Date: Thu, 01 Dec 2011 17:32:46 -0500

On Thu, 2011-12-01 at 09:17 -0700, Alex Rousskov wrote:

> _required_ to fix messages they forward?

proxies are clients, and I'd like to think the spec aspires to requiring
all clients to send well formed messages. so I would favor mandatory
cleanup.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 2 Dec 2011 00:14:57 +0100

Hi Patrick,

> all clients to send well formed messages. so I would favor mandatory

I see it from a different angle. In my opinion, proxies are not client
in that they don't author requests but forward what they understood from
what they received. It's much easier to be a clean sender when you're a
client than when you're a proxy. If we brought the multiple content-length
issue here, it's precisely because proxies were facing it. There are many
situations where proxies are expected to perform some cleanup based on
protocol compliance, and expected to act differently based on client or
server bugs that end-users expect them to ignore.

I have a recent example. About 2 months ago, I got a report that a haproxy
user was sometimes seeing haproxy reject bad requests. Upon investigation,
it appeared that a buggy client software was cumulating headers between
each request to the point where some headers were sent 37 times before
being blocked due to too large a request. Very few users reached this
limit and were affected. Fortunately for them, haproxy is not able to
fold multiple headers when forwarding, otherwise they would have come to
a complete stop because the server-side software's parser was unable to
handle a comma in the affected header.

I'm not saying that it's better not to fix, and in other circumstances,
the opposite could have been true. I'm just saying that in this specific
case it was fortunate that the forwarded stream was as little mangled as
possible. My principle has always been to strictly control inputs, then
don't touch what you don't absolutely need to. From my experience, it
has always resulted in the least breakage.

However, just like a client, a proxy which is adding or mangling headers
has absolutely no excuse for emitting them in a wrong format.

Regards,
Willy

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 02 Dec 2011 14:40:28 +1300

we're talking here about the specification for HTTP and Content-Length

I don't see how we can write prose into HTTP that permits any agent to 
send more than one Content-Length header under any circumstances and 
still be compliant with the spec, no matter what you call the software 
that does it or how it operates.

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com
WinGate 7 is released! - http://www.wingate.com/getlatest/

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 2 Dec 2011 08:27:31 +0100

Hi Adrien,

> that does it or how it operates.

The ambiguity raised by Alex is that the spec talks about how headers
should be constructed and interpreted, while in his case he's also
forwarding some data which happen to be headers, without constructing
them. That's typically the case for transparent proxies ("does not modify
the request or response beyond what is required for proxy auth").

I don't see either how we could add text to cover those specific cases
for gateways or proxies which don't know all headers and their associated
rules. Once again, it's not about how to correctly write a header you're
building from scratch, but how to spot anomalies in headers you don't
necessarily care about and how to perform the cleanup without breaking
anything.

Perhaps we should try to spot a few essential guidelines for transparent
proxy authors. I think it is reasonable to emit a few MUST and a few
SHOULD which are compatible with the software/hardware capabilities
(for instance: SHOULD fix XXX and MUST reject a few dangerous patterns
if unable to fix). But we really cannot expect some proxies to know and
apply all the complex rules which clients must respect, that's irrealistic
when you consider they're forwarding headers they don't understand, and
sometimes just bytes. Or we can create a new category ("dumb gateways" or
"dumb proxies"). But in my understanding, that's already what transparent
proxies are supposed to be.

Regards,
Willy

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 02 Dec 2011 10:21:42 -0700

> that does it or how it operates.

We cannot write such prose, but folks who interpret "send" differently
from "forward" do not see that as an obstacle. We can write prose to
clarify whether "forward" is "send".

For example, when somebody starts complaining that the proxy is not
compliant because it is sending two Content-Length headers, the proxy
developer may claim that the proxy is not sending two Content-Length
headers but just forwarding two end-to-end headers received from the
previous hop. They would say that the proxy has no business fixing
end-to-end headers. I have heard many such arguments, usually backed by
a "do no harm" principle.

If there is consensus that any HTTP proxy is responsible for all
messages it forwards (i.e., send = forward), then HTTPbis should clearly
state that once and for all. For example:

  In the absence of requirements specific to proxies, when forwarding
  an upstream message, an HTTP proxy MUST comply with requirements
  for HTTP clients. Consequently, if the upstream message is
  malformed, a proxy MUST NOT forward it "as is".

If we state that, an HTTP proxy would have to make a choice when dealing
with a "broken" incoming message:

  1) reject the malformed message
  2) forward a "fixed" message
  3) forward raw bytes, becoming a TCP tunnel

Currently, many interpret HTTP specs as if there is another option:

  4) forward the malformed message "as is"
     while interpreting it correctly internally.

Thank you,

Alex.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 2 Dec 2011 21:39:13 +0100

>      while interpreting it correctly internally.

Actually, it's even more common to see this :

    5) ignore 90% of headers you don't care about and for which you
       don't know how to determine the validity, and apply 4) when
       you care about the header.

I don't see how we can impose 2) for the 90% headers that a proxy
does not understand and just ignores.

In my opinion it would be fair to ask that if a proxy internally uses a
header and it has to resort to specific tricks to interprete it correctly,
then it should at least fix it before forwarding it since it is supposed
to know the rules which apply to this header.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Fri, 02 Dec 2011 21:50:42 +0100

On 2011-12-02 21:39, Willy Tarreau wrote:
> ...

The reason being that some header fields are special and essential for 
message framing/routing.

If HTTP/1.1 had been defined from scratch, Host and Content-Length 
probably wouldn't be in the same part of the message as other metadata...

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 2 Dec 2011 22:11:04 +0100

Hi Julian,

> The reason being that some header fields are special and essential for 

I perfectly agree on this point. What it also means is that a proxy which
has to respect message framing will have to interprete those headers and
will then know their semantics and how to fix them, which does not seem
to contradict the previous proposal. Still it could be argued that not
trying to fix them will avoid making things worse (which unfortunately is
too true in some environments).

> probably wouldn't be in the same part of the message as other metadata...

Indeed, as well as Connection and Transfer-Encoding. Still they exist now
and the fact we're naming them implies they have a more important role than
the other ones. Maybe something like this would make things clearer for
proxy authors (with a better wording) :

  When a proxy faces a malformed header, it must either reject the message
  or fix the header before forwarding it. This applies to all headers the
  proxy cares about for its processing, and in any case to Connection,
  Content-Length, Host and Transfer-Encoding.

Regards,
Willy

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 02 Dec 2011 14:12:14 -0700

>        you care about the header.

Actually, RFC 2616 does define what a valid header is. Any header not
explicitly defined in that RFC still has to obey the rules for extension
headers. The proxy does not have to follow any other rules to stay
compliant.

> does not understand and just ignores.

Per my proposed wording, those headers do not need fixing: Any extension
header can be forwarded "as is" while staying compliant. There are some
nuances related to handling of multiple extension headers with same
names, but the same "as is" rule works for those as well.

If some specification defines an extension HTTP header that requires
special forwarding beyond RFC 2616 rules, that specification itself is
broken.

Thank you,

Alex.

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 02 Dec 2011 14:50:13 -0700

On 12/02/2011 02:11 PM, Willy Tarreau wrote:

> Maybe something like this would make things clearer for

Since the concept of "caring about the header" is not (and cannot be)
defined and "fixing the header" essentially means emitting a header that
complies with the specs, I believe the above is pretty much equivalent to:

Now, whether the above was the intent of RFC 2616, and (if yes) whether
HTTPbis is allowed to make that intent explicit, I do not know.

If we restrict ourselves to Content-Length handling, the above becomes:

  When forwarding an upstream message with an invalid Content-Length
  header, a proxy MUST NOT send a message with an invalid
  Content-Length header to the next hop.

followed by the current HTTPbis rules about appropriate reactions to
some specific kinds of malformed Content-Length headers.

Cheers,

Alex.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 2 Dec 2011 23:02:25 +0100

> headers. The proxy does not have to follow any other rules to stay

Some trickier situations have become apparent since 2616. One of them
being the multiple Content-Length. Each of them taken individually is
OK. Having them together is wrong, still it happens and causes folding,
resulting in "100, 100" appearing in the value. How to handle multiple
or empty Host headers is another issue. Another example, if you receive
"Connection: Transfer-Encoding" with both a Transfer-Encoding and a
Content-Length headers. You're supposed to use Transfer-Encoding to
determine the transfer length and ignore the Content-Length, to
remove the Transfer-Encoding header and to forward the message. The
next hop will only use the Content-Length that you ignored. And I
think that since you're a proxy implementer, you're well used to
facing such issues and to have to take decisions at one point.

> names, but the same "as is" rule works for those as well.

It's not that black and white. For instance in haproxy I don't use the
Host header, I leave it unaltered so that the next hop gets the real
value. So if there are crappy values there, they will pass through.
It is possible that in your proxy you heavily rely on Host and that
you find it obvious to fix it.

My current practice is to forward everything "as-is" but to be strict on
checks I need to get my work done : if I can correctly parse the header
when applying the rules, then the next hop will be able to do so and that's
fine. Otherwise I reject the message. It still straightens my hair to let
a Content-Length header leave with multiple values but it's particularly
expensive to fix every corner case. Still, I'm ready to fix a few really
dangerous ones, especially if I need to handle them in a specific way.
But I will not do a massive cleanup.

> special forwarding beyond RFC 2616 rules, that specification itself is

It's not that simple. There are issues which only happen when forwarding.
For instance, I once used to add "Connection: x-forwarded-for" to my
requests to some reverse-proxies to show that my IP address was not
passed to the origin server. This is a typical example of something
which only affects forwarding. X-Forwarded-For is not part of a spec
but you can do that with other headers and observe surprizing results.

It's not just a matter of broken specs, just that HTTP has evolved much
farther than initially designed for, inevitably resulting in nasty corner
cases. That's what http-bis is trying to address after all, so I think
that your request for clarifying "Send" vs "Forward" is welcome !

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 2 Dec 2011 23:06:12 +0100

>   malformed, a proxy MUST NOT forward it "as is".

This is still as much ambiguous as what it currently is IMHO, because
everyone will have a good reason to consider that some fixing is not
under his responsibility.

(...)
> some specific kinds of malformed Content-Length headers.

I like this one much better, it sets the focus on a clear action even
if it implies additional specific code in products.

Cheers,
Willy

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 02 Dec 2011 16:12:44 -0700

On 12/02/2011 03:02 PM, Willy Tarreau wrote:
> resulting in "100, 100" appearing in the value.

I am aware of these issues, but I do not see why they make the situation
we are discussing any "trickier". Multiple Content-Length header fields
are not valid per RFC 2616 rules. So are multiple Content-Length header
field values. And so is folding of Content-Length headers. All of that
is already covered by RFC 2616.

What is not fully covered, is what the proxy (and other agents) are
supposed to do when they receive invalid values. That is what HTTPbis is
trying to address. The current HTTPbis text is not clear when it comes
to proxy reaction to what HTTPbis considers "fixable" or "correctly
interpretable" header values.

> you find it obvious to fix it.

Please note that you seem to divide headers into "the ones the proxy
understands" and "the ones the proxy does not understand". Since such
classification is beyond HTTP specification boundaries, it does not help
and, indeed, it creates a lot of gray areas on its own.

I divide headers into "violating HTTP specs" and "not violating HTTP specs".

Now back to your example with the Host header, which I find very
valuable for this discussion!

An HTTP proxy that talks to another proxy does not need to interpret or
understand the Host header value. Can we require that proxy to validate
the Host header against HTTP specs? I do not think so. While such
validation would certainly be possible, it would be a waste of resources
in most cases. Thus, the "proxy must not forward invalid headers" is not
a good rule! Keeping the classification black and white helped us reach
that conclusion.

I believe there are many similar cases. For example, it would be wrong
to require all proxies to validate Date headers.

Are there headers that any HTTP proxy has to validate? Yes, correctly
handling all headers that deal with message framing is required for any
HTTP proxy to work correctly. Which of those headers are end-to-end and,
hence, can be "forwarded"? I can think of only the Content-Length header
but please correct me if I there are others.

Given the above, we can restrict the rule to:

> but you can do that with other headers and observe surprizing results.

Sorry, I do not see how your example contradicts what I said. Correct
handling of the Connection header is specified in RFC 2616. Connection
header value may include extension headers so adding X-Forwarded-For to
Connection is fine as far as RFC 2616 is concerned. A compliant proxy
receiving such a Connection header would have to remove the
X-Forwarded-For header from the received request, even if it knows
nothing about that particular extension.

Cheers,

Alex.

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 02 Dec 2011 16:30:34 -0700

> under his responsibility.

The existence of good reasons to violate a MUST does not make that MUST
ambiguous. However, it does prove that the MUST itself is wrong and
should not be included in HTTPbis.

> if it implies additional specific code in products.

Glad we have converged on something usable that we both can agree on! I
suspect the above wording can be improved, but it does resolve the
current ambiguity.

Thank you,

Alex.

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 3 Dec 2011 11:28:32 +1100

Hi Dmitry,

On 29/11/2011, at 1:05 PM, Dmitry Kurochkin wrote:

> Hello.
> 
> HTTPbis part 1 section 3.3 has the following requirements:
> 
>  If more than one Transfer-Encoding header field is present in a
>  message, the multiple field-values MUST be combined into one
>  field-value, according to the algorithm defined in Section 3.2, before
>  determining the message-body length.
> 
>  If a message is received that has multiple Content-Length header
>  fields (Section 8.2) with field-values consisting of the same decimal
>  value, or a single Content-Length header field with a field value
>  containing a list of identical decimal values (e.g., "Content-Length:
>  42, 42"), indicating that duplicate Content-Length header fields have
>  been generated or combined by an upstream message processor, then the
>  recipient MUST either reject the message as invalid or replace the
>  duplicated field-values with a single valid Content-Length field
>  containing that decimal value prior to determining the message-body
>  length.
> 
> It is not clear whether proxies that do not reject the message MUST
> replace the duplicated field-values in the forwarded message.  In other
> words, are non-transforming proxies required to either reject or fix the
> message?

My reading is that they must. Roy?

> Should the specs require connection closure to prevent message smuggling
> in naive implementations that respond with an error to "reject the
> message" but then keep the connection open/persistent?  The specs are
> explicit about this in item #3 where they talk about "framing" error,
> but not here, where essentially the same framing error is suspected if
> the message is rejected.

I don't think that's required here, because the values are duplicates (as discussed).

> Also, should the specs be more specific regarding what connection
> (upstream or downstream) proxy is required to close?

There is no upstream connection at this point; in this context, the proxy hasn't started to forward the request yet.

>  If this is a response message received by a proxy, the proxy MUST
>  discard the received response, send a 502 (Bad Gateway) status code as
>  its downstream response, and then close the connection.
> 
> Above can be interpreted that the downstream connection must be closed.
> But the proxy should only close the upstream connection.  Moreover, the
> "then" word is wrong there as there is no need to keep that connection
> open while the 502 response is being sent.  The order of those actions
> should be up to the proxy to decide.
> 
> The fixed text may read:
> 
>  If this is a response message received by a proxy, the proxy MUST
>  discard the received response, close the upstream connection, and send
>  a 502 (Bad Gateway) status code response as its downstream response.

I think replacing "then" with "also" in the text is sufficient.

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 3 Dec 2011 07:53:22 +0100

Hi Alex,

> I divide headers into "violating HTTP specs" and "not violating HTTP specs".

The issue I have with this is that for me, violating the spec simply implies
not doing a MUST or doing a MUST NOT. There are a huge number of such rules
in the spec, many of them irrelevant to most proxies. And by ignoring these
rules, the proxies will violate the spec by forwarding wrong contents. Your
example of the Date header is perfect. It's a general header with a MUST for
the format, still a number of proxies don't care about it and will not check
it. By forwarding a wrong one, they will violate the spec. The issue I'm
seeing is that by making it "normal" to violate the spec for a number of uses, 
critical rules are not necessarily spotted. That's why I like your proposal
which specifically focuses on the Content-Length header.

> nothing about that particular extension.

The problem is the order in which it is done. It is said "before a message
is forwarded". Some proxies remove the headers they have added themselves.
So you have a first layer of reverse proxy in front of your servers, which
adds the client's IP address in X-Forwarded-For for logging purposes, and
which immediately removes it due to the Connection rule. Connection applies
to the headers received by the proxy, not to headers sent. But as the text
says "forwarded", this is ambiguous. The example with Transfer-Encoding is
similar and nastier since it can lead to content smuggling, though I'm not
aware of a proxy which applies Connection without fixing Content-Length
and/or Transfer-encoding.

Regards,
Willy

---

From: Dmitry Kurochkin <[email-masked]>
Date: Sat, 03 Dec 2011 17:47:38 +0400

Hi Mark.

> My reading is that they must. Roy?

(There is a long discussion in this thread on this.)

> I don't think that's required here, because the values are duplicates (as discussed).

I think this depends on the result of the discussion on the first
question.  If a proxy is required to correctly interpret duplicate CL
values, then there is no framing problem and connection should not be
closed.  But if a proxy can interpret duplicate CL as invalid and reject
the message, then it can not correctly determine the message length and
MUST close the connection.

> There is no upstream connection at this point; in this context, the proxy hasn't started to forward the request yet.

The above question is related to the sentence (quoted below) about
handling an invalid upstream response by a proxy.  So there are both
upstream and downstream connections.

> I think replacing "then" with "also" in the text is sufficient.

It still leaves the question on which connection (upstream or
downstream) MUST be closed.

Regards,
  Dmitry

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 5 Dec 2011 13:13:17 +1100

> MUST close the connection.

You're requiring us to make a false choice. The current text allows the proxy to choose whether to reject the message or to replace the value, but does not require the connection to be dropped. I.e., it's an error, but not a framing error.

Are you arguing that duplicate C-L is a framing error? If so, we can have that discussion, but let's separate it from the other concerns here.

> upstream and downstream connections.

[...]

>>> a 502 (Bad Gateway) status code response as its downstream response.

Ah.

That seems quite straightforward; your proposed text seems fine.

Thanks,

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 7 Dec 2011 19:49:14 +1100

On 03/12/2011, at 5:53 PM, Willy Tarreau wrote:

> it. By forwarding a wrong one, they will violate the spec. 

This is a good point. I think we can address this by changing this in the sections on conformance (one per draft, IIRC):

>    This document also uses ABNF to define valid protocol elements
>    (Section 1.2).  In addition to the prose requirements placed upon
>    them, Senders MUST NOT generate protocol elements that are invalid.

to something like:

>    This document also uses ABNF to define valid protocol elements
>    (Section 1.2).  In addition to the prose requirements placed upon
>    them, Senders MUST NOT generate protocol elements that are invalid, 
>    unless the element is out of their control (such as a header generated by
>    an upstream sender), in which case they MAY attempt to correct
>    the syntax before sending.

Thoughts? We'd still have the option of making specific exceptions where we require generation to be conformant (e.g., when there are security implications).

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 7 Dec 2011 10:47:43 +0100

Hi Mark,

> Thoughts? We'd still have the option of making specific exceptions where we require generation to be conformant (e.g., when there are security implications).

That's perfect in my opinion! I really like this approach, as it will imply
that proxies will have much less excuses for not respecting the rules. It
will make it easier to insist on what is important.

Thanks!
Willy

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 07 Dec 2011 23:41:02 +1300

> Thoughts? We'd still have the option of making specific exceptions where we require generation to be conformant (e.g., when there are security implications).

The multi-part sentence seems a bit bulky. Maybe a separate sentence 
specifically aimed at middleware? "generate" is pretty clearly aimed at 
agents adding things to the headers.

Would it make sense to specify that proxies MUST (SHOULD? "ought to"?) 
fix headers which they pay attention to and identify as invalid?
That leaves a huge loophole that they can ignore a header entirely for 
both usage and relay purposes without being in violation. But if they 
are smart enough to make use of it and validate they become part of the 
interoperability correction mechanism for HTTP.

IMHO something stronger than MAY would be useful to encourage in-transit 
corrections toward validity. The process of erasing invalid headers 
pushes the valid formats more into view of those who dont read the specs.

AYJ

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 07 Dec 2011 10:02:46 -0700

On 12/07/2011 01:49 AM, Mark Nottingham wrote:
>> Your

> This is a good point. I think we can address this by changing this in
> the sections on conformance (one per draft, IIRC):

>>    them, Senders MUST NOT generate protocol elements that are invalid.

The above wording is an improvement over RFC 2616 but it raises the same
"generate vs. forward" doubts. Many proxy developers claim that their
proxies do not generate end-to-end headers and, hence, ignore the above
MUST NOT. The specs do not make it clear whether they are right. We must
polish this.

> Thoughts?

I like the intent, but the text is still problematic in several areas:

* "generate" is probably undefined and may mean different things to
different developers

* "out of control" is probably undefined and will mean different things
to different developers

* "correct the syntax" permission implies that other corrections are not
allowed, which contradicts best practices in some cases.

I am afraid we have to explicitly acknowledge that forwarding of
protocol elements exists and that forwarding comes with its own small
set of rules. Here is a draft:

    We use the term "generate" to describe an agent action of creating
    a protocol element. We use the term "forward" to describe the
    intermediary action of sending a protocol element received from
    another agent. By definition, a forwarded element is sent exactly
    as received except for permitted whitespace changes. Any other sent
    element is considered generated by the agent sending it.

    This document uses ABNF and prose requirements to define valid
    protocol elements (Section 1.2). An agent MUST NOT generate invalid
    protocol elements. Unless explicitly required to do otherwise for a
    specific element, an intermediary MAY forward invalid protocol
    elements. If an intermediary receives an invalid protocol element,
    it MAY generate and send an equivalent valid protocol element,
    taking the responsibility for all associated risks.

The above needs more work, but I think it resolves the major problem of
trying to have the same set of rules for element generation and
forwarding, which are fundamentally different actions. What do you think?

Thank you,

Alex.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 7 Dec 2011 09:30:05 -0800

> Thoughts? We'd still have the option of making specific exceptions where we require generation to be conformant (e.g., when there are security implications).

A proxy is responsible for complying with all requirements on senders,
clients, and proxies.  That is how the entire protocol is written.
While I understand that some folks may not want to do that, the answer
is that their software is not compliant with HTTP.  There is no need for
further exceptions.

There are many "proxies" that are not HTTP proxies.

....Roy

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 07 Dec 2011 11:52:21 -0700

> clients, and proxies.  That is how the entire protocol is written.

Does the above imply that all compliant proxies must _validate_ all
forwarded headers defined by RFC 2616, to make sure those headers do not
violate any of the 600+ MUSTs?

If this is how the protocol has to be interpreted, we must clarify that
in HTTPbis because (without an explicit confirmation) many folks would
continue to use a less demanding interpretation. We should then also
explain what a proxy should do if a to-be-forwarded header field fails
validation but is not needed for correct proxy operation (from UA and
origin server points of view)?

Please consider the following specific example. A proxy receives an
otherwise valid message with a Date header that violates the following MUST:

  The [Date] field value MUST be sent in rfc1123-date format.

When forwarding the message, the proxy has a few choices:

  0) Send the Date header field as it was received.
  1) Do not send any Date header field.
  2) Create and send a new Date header.
  3) Reject the entire received message.

What should a compliant proxy do?

And Date is just one example. There are many other complex end-to-end
headers that a given proxy does not need to validate to function
correctly (from UA and origin server points of view) and that are
difficult or even impossible to "fix" without creating more problems.

Thank you,

Alex.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 7 Dec 2011 12:46:39 -0800

> violate any of the 600+ MUSTs?

In some respects, yes, such as for framing and folding.  In cases where
the proxy is not supposed to muck with the header field, the requirements
should be more tightly scoped.

> origin server points of view)?

Why do we need to clarify that in general?  We should just fix the bugs.

>  The [Date] field value MUST be sent in rfc1123-date format.

That's a bug.  For one thing, it is a poorly phrased requirement
because it doesn't target the responsible party (in this case, the
program that generates the field-value).  For another, there is
(or was) some other requirement somewhere that intermediaries must
not modify so-called end-to-end metadata used for caching or that
might appear in digest hashes.

> difficult or even impossible to "fix" without creating more problems.

I don't agree -- we have been fixing this type of bug all throughout
this process.

....Roy

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 07 Dec 2011 15:48:20 -0700

> should be more tightly scoped.

Glad we agree that forwarded header validation and, hence, complying
with the corresponding sender MUSTs is necessary in only _some_ cases,
such as when dealing with framing, folding, and similar "core" protocol
operations.

I think I know understand that you prefer to make every sending-related
set of rules tight enough to avoid any misunderstanding on what a proxy
should do. I agree that it is possible, but I think having a set of
_default_ rules would [still] be very helpful, especially because many
HTTP-based specifications are not that tight and will not be revised as
a part of this effort.

In summary "a proxy is a sender" is the right principle, but we must be
careful when specifying sending rules because not all senders are
proxies. The proxy should not be made responsible for everything it
_forwards_, just for the "core" stuff such as framing headers. There are
several ways to polish the specs accordingly.

> Why do we need to clarify that in general?  We should just fix the bugs.

I think that a few general forwarding rules are better than many
specific and often duplicated ones, but I agree that the problem can be
fixed by adding explicit forwarding rules to each protocol element
and/or rephrasing some of the existing element-specific sending rules to
not apply to proxies.

Either way, defining "[generating and] sending" differently from
"[receiving and] forwarding" would probably be necessary for the
new/fixed rules to make sense.

> program that generates the field-value).

Exactly. If we just rephrase the above MUST as

    The agents MUST send Date value in rfc1123-date format

then we have suddenly required all proxies to validate forwarded Date
headers (because a proxy is an agent) which seems unnecessary and costly
in general.

If we rephrase the above MUST as

    The agents MUST generate the Date value in rfc1123-date format

then proxies are excused from validation, but the requirement itself
looks weird/overreaching: Why would HTTP care about something that is
generated unless it is actually sent?

A possible solution is to keep formulating the requirements in terms of
sending, as usual, but allow proxies to ignore sending requirements when
forwarding certain headers. There are two ways to define those "certain
headers":

1) All headers except those explicitly required to be validated (e.g,
Content-Length).

2) No headers except those explicitly allowed to be forwarded without
validation (e.g., Date).

>  For another, there is

There is actually no such explicit MUST in RFC 2616 (another bug), but
we are not talking about modification of end-to-end headers by proxies
here. We are talking about validation of received end-to-end headers by
proxies for the purpose of obeying the "proxy is a sender and the sender
MUST not send invalid stuff" principle.

> I don't agree -- we have been fixing this type of bug all throughout

Sorry, I am not sure what you are disagreeing with. If you are saying
that a proxy can fix invalid headers before forwarding them, then I do
not see how a proxy can reliably fix invalid Date, Server, Warning,
Vary, etc. headers (in general).

Thank you,

Alex.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 8 Dec 2011 10:13:34 +1100

On 08/12/2011, at 4:30 AM, Roy T. Fielding wrote:
[…]
> There are many "proxies" that are not HTTP proxies.

So, how is a proxy that receives an invalid Date header supposed to handle it? Re-generate the Date? Does *any* implementation do this?

Likewise for, say, a Cache-Control header where an extension directive doesn't meet the generic ABNF; what should a proxy do? Does any existing implementation actually do it?

If no (or very little) software is compliant with HTTP, that begs the question of whether compliance with HTTP is defined well.

--
Mark Nottingham
http://www.mnot.net/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 7 Dec 2011 16:30:06 -0800

> If no (or very little) software is compliant with HTTP, that begs the question of whether compliance with HTTP is defined well.

Most people who have implemented a "proxy" have, in fact, implemented
a broken tunnel, TCP forwarder, or gateway.  Relaxing the HTTP requirements
on proxies does not help them whatsoever.  The requirements are there for
actual proxy implementations that wish to be interoperable with the
recipients of the messages they send.  If a specific requirement is not
necessary for interoperability, then it shouldn't be in the spec as a
requirement!

One of the major problems with 2616 is that a huge amount of prose of 2068
was elevated from description to requirement very late in the review cycle.
If you find a broken requirement, then it should be fixed.  In contrast,
giving proxies a free pass does not help them -- it just fails to define
the protocol.  Let's not forget that almost all problems associated
with extending HTTP and using pipelines in practice are due to broken
intermediary implementations not adhering to one or more of the
interoperability requirements of HTTP.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 8 Dec 2011 11:34:04 +1100

On 08/12/2011, at 11:30 AM, Roy T. Fielding wrote:
> on proxies does not help them whatsoever.

Oh, absolutely. I don't have much interest in making these devices conformant to HTTP by changing the spec.

> The requirements are there for

Right. See discussion above; this is what we're trying to clarify.

> was elevated from description to requirement very late in the review cycle.

Ack; the evidence of this is everywhere, and it's quite annoying.

> interoperability requirements of HTTP.

Right. So, back to the questions above -- what is a HTTP proxy supposed to do in these situations? Should this specific requirement be relaxed?

--
Mark Nottingham
http://www.mnot.net/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 7 Dec 2011 16:43:04 -0800

> Right. So, back to the questions above -- what is a HTTP proxy supposed to do in these situations? Should this specific requirement be relaxed?

Yes, that's what I said earlier -- it is a bug to say that the field-value
MUST be sent in a specific form when senders are not expected to rewrite
the field-value, and might actually be forbidden from doing so by other
requirements.  What it is supposed to say is that newly generated Date
field-values MUST be of the standard form.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 8 Dec 2011 12:03:02 +1100

On 07/12/2011, at 9:41 PM, Amos Jeffries wrote:

> Would it make sense to specify that proxies MUST (SHOULD? "ought to"?) fix headers which they pay attention to and identify as invalid?
> That leaves a huge loophole that they can ignore a header entirely for both usage and relay purposes without being in violation. But if they are smart enough to make use of it and validate they become part of the interoperability correction mechanism for HTTP.
> 
> IMHO something stronger than MAY would be useful to encourage in-transit corrections toward validity. The process of erasing invalid headers pushes the valid formats more into view of those who dont read the specs.

I don't know if I like this. One implementation's "correction" is another's loss of information...

--
Mark Nottingham
http://www.mnot.net/

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 08 Dec 2011 08:25:25 -0700

> field-values MUST be of the standard form.

I think it is likely that there is consensus that proxies may forward
invalid protocol elements that they did not generate and that are not
related to "framing" and similar "core" issues.

Where we may not agree is how to codify that in the protocol. We have
two basic options:

1. Avoid any global exceptions for proxies. Make sure every sending
requirement is carefully formulated so that proxies are excluded if they
ought to be excluded. For example, a "Date-generators MUST generate Date
in standard form" rule would exclude proxies unless they add Date (when
such addition is required).

2. Provide a global exception for proxies. Make sure every "core"
sending requirement is carefully formulated to preserve "framing" and
such. For example, a "Agents MUST send Date in standard form" rule works
fine because the "global proxy exception" will provide a necessary
exception for proxies.

As far as compliant HTTPbis implementations are concerned, both
approaches will result in exactly the same outcome. Niether approach
grants proxies more freedoms!

IMO, the key difference between the two approaches actually lies outside
RFC 2616 and HTTPbis. We are not going to rewrite all existing
HTTP-based specs and we are unlikely to be very good at controlling all
new ones. If not explicitly codified, the understanding of this issue
will die and the problems will resurface. That is why I think it is
better to provide an explicit global exception for proxies.

Please note that both approaches will require introduction of new
normative vocabulary. AFAICT, we currently do not have any normative
language that clearly distinguishes "generation" from "forwarding" in
the context of "sending". That has to be fixed regardless of the
approach. Otherwise, proxy developers will continue to misinterpret
"send" and "sender" rules.

I have proposed specific wording for the "global exception for proxies"
approach. If there is consensus that the first approach is the way to
go, I would be happy to try to reshape that proposal to just introduce
the necessary generation/forwarding vocabulary.

Thank you,

Alex.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 8 Dec 2011 16:41:19 +0100

Hi Alex,

On Thu, Dec 08, 2011 at 08:25:25AM -0700, Alex Rousskov wrote:
(...)
> the necessary generation/forwarding vocabulary.

I think that if we carefully define what a "sender" is and what "send"
means, we can avoid the global exception without rewriting every MUST
rule, because after all the base of the problem lies there.

Regards,
Willy

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 08 Dec 2011 09:39:40 -0700

> rule, because after all the base of the problem lies there.

A definition of "send" and "sender" that avoids rewriting most sending
rules while allowing proxies to forward some invalid headers is
essentially the same as the global exception.

Alex.