Subject: Re: Question about H2 and chunked encoding

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 25 Sep 2017 13:52:10 -0700

> On Aug 6, 2017, at 8:26 AM, Mark Nottingham <[email-masked]> wrote:
> 
>> On 4 Aug 2017, at 8:15 pm, Cory Benfield <[email-masked]> wrote:
>> 
>>> On 4 Aug 2017, at 08:43, Stefan Eissing <[email-masked]> wrote:
>>> 
>>> The live, so far, in my test cases for Apache httpd. I'd expect them to get used when the web starts worrying about message integrity more, once the https everywhere thing is done. ;-)
>> 
>> Adding to the list, HTTP/1.1 trailers are barely supported in the Python ecosystem: none of the major libraries or tools can emit them, and many fail to parse them. The better implementations parse-but-ignore. The Python HTTP/2 stack supports emitting and parsing trailers.
> 
> This thread makes me wonder if we can't improve things slightly in HTTPtre. E.g., we could say:
> 
>> A header field MUST NOT appear in trailers unless its definition allows it. Receiving implementations MUST ignore header fields appearing in trailers when their definitions do not allow it.
> 
> ... with an appropriate column in the header field registry to record this. I don't think this would break any existing implementations, because trailers are so seldom used right now, and I think there's a strong argument that this is the case anyway; the consuming application has to expect something in trailers to act upon it.
> 
> <https://github.com/httpwg/http11bis/issues/16#issuecomment-320513541>

We spent an enormous amount of time separating HTTP message processing from
required understanding of its semantics.  What you describe above is not an
improvement.  It isn't even implementable, in general.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 26 Sep 2017 14:01:24 +1000

> On 26 Sep 2017, at 6:52 am, Roy T. Fielding <[email-masked]> wrote:
> 

This has nothing to do with message processing.

--
Mark Nottingham   https://www.mnot.net/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 26 Sep 2017 16:26:39 -0700

> On Sep 25, 2017, at 9:01 PM, Mark Nottingham <[email-masked]> wrote:
> 
> 

The existing requirements are in
http://httpwg.org/specs/rfc7230.html#rfc.section.4.1.2

  A sender MUST NOT generate a trailer that contains a field necessary for
  message framing (e.g., Transfer-Encoding and Content-Length), routing
  (e.g., Host), request modifiers (e.g., controls and conditionals in
  Section 5 of [RFC7231]), authentication (e.g., see [RFC7235] and [RFC6265]),
  response control data (e.g., see Section 7.1 of [RFC7231]), or determining
  how to process the payload (e.g., Content-Encoding, Content-Type,
  Content-Range, and Trailer).

  When a chunked message containing a non-empty trailer is received, the
  recipient MAY process the fields (aside from those forbidden above)
  as if they were appended to the message's header section.
  A recipient MUST ignore (or consider as an error) any fields that are
  forbidden to be sent in a trailer, since processing them as if they
  were present in the header section might bypass external security filters.

You suggested adding two normative requirements to message processing,
both based on "their definitions".  The first isn't targeted but translates
into MUST NOT send (instead of the existing MUST NOT generate).  The second
is a recipient MUST ignore based on *other* definitions.  Both impact
message processing by intermediaries.

If we want to add a table to the registry that indicates what header
fields are forbidden to be in trailers, we can do that without impacting
the existing text in 7230 (which is carefully worded to avoid being a
message processing requirement).

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 27 Sep 2017 05:05:02 +0200

Hi Roy,

> message processing requirement).

In fact I think the root cause of this permanent difficulty is that we
all know that some header fields are used to influence message parsing
and framing while others are related to content definition, which used
to be called entity-header fields in 2616. But by considering all of
them as header fields without a strict distinction, we're seeing some
exceptions like above appear. I tend to think that the wording in 7230
using "e.g." in the list is exactly what makes it hard for implementations
to establish a strict list, and that it might be useful to draw such a
list of the fields affecting message parsing and framing, those affecting
message routing/acceptance (host,cookie/set-cookie,auth,cache-control,...)
and maybe other classes, and establish a matrix of which ones are allowed
where. For example for H2 we have a strict list of the fields that must be
discarded during H1 to H2 translation, that makes it easier to implement.

Just my 2 cents,
Willy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 27 Sep 2017 12:29:57 -0700

> On Sep 26, 2017, at 8:05 PM, Willy Tarreau <[email-masked]> wrote:
> 
> Hi Roy,
> 
> exceptions like above appear.

Well, no, the root cause is that there is no syntactic distinction between
various types of header fields that could be easily seen by all recipients.
That's how we make a protocol self-descriptive.

The problem is that we have no choice in "considering all of them as header
fields without a strict distinction".  That's HTTP/1.  It isn't going to
change by adding requirements to a spec that simply aren't true for HTTP/1.
Even if we required a prefix on names, we'd still have to deal with the
folks who don't read specifications and either fail to use the prefix or
copy the wrong prefix from somewhere else.

> I tend to think that the wording in 7230
> where.

But, there is no strict list.  That's the point of the e.g.'s.  We can't add
requirements that have no basis in the protocol practice.  I think it is a
fine idea to make distinctions clearer in the registry, but that doesn't
change the normative requirements (how to interoperate).  A strict list is
not going to be interoperable after deployment.

> For example for H2 we have a strict list of the fields that must be

*sigh*

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 29 Sep 2017 00:13:18 +0200

> That's how we make a protocol self-descriptive.

I totally agree that it would have been much easier that way but that's not
what we have *for now*.

> copy the wrong prefix from somewhere else.

In fact HTTP/1 probably isn't going to evolve much on the framing side, at
least by lack of incentive from those who used to push it as far as possible
for performance reasons. It seems to me it would be reasonable to draw this
list for H1 and find other mechanisms for H2+. For example, in H2, the use
of ":status" etc could seem strange at first but actually is a good solution
to split those critical header fields and the other ones. It's just one
solution, but I think we should not declare that it's too late just because
the protocols are already in the wild. What's deployed works with what it
does now, we can try to find ways to improve how we'll proceed in the future.

> not going to be interoperable after deployment.

I don't necessarily agree here, it depends if we have a version to help
make the distinction. For instance, while HTTP/1.0 vs 1.1 was a pain to
deal with regarding persistent connections due to 1.1 mostly being emitted
to try to normalize what people were already doing, it still managed to
work. I agree however that fixing too hard limits is not future proof
and that at least one dimension must be allowed to change when the other
ones are fixed.

> > discarded during H1 to H2 translation, that makes it easier to implement.

I mean it's easier to implement than having to implement "e.g.". This
implementation is tied to this version and is not allowed to evolve for
that version. That's the limitation.

Cheers,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 29 Sep 2017 17:06:02 +1000

On 27 Sep 2017, at 9:26 am, Roy T. Fielding <[email-masked]> wrote:
>>> We spent an enormous amount of time separating HTTP message processing from required understanding of its semantics.  What you describe above is not an improvement.  It isn't even implementable, in general.
>> 

[...]

> message processing requirement).

The intent was to affect the processing of the header field's semantics, not generic message handling. As noted in <https://github.com/httpwg/http11bis/issues/16#issuecomment-320587821>, it's not really possible to do it as part of message processing.

The problem we're trying to address is that right now, any header (barring those that explicitly prohibit it) can be sent as a trailer, and potentially upgraded to a header by an intermediary on the path or the receiving endpoint. If the header isn't designed for this, it can be surprising in an unpleasant way, which is one of the reasons browser security people get antsy about trailers.

If we invert permissions so that the default is that nothing can go in trailers unless it's explicitly allowed by definition, I think it's much closer to the reality of deployment now; that unless a header explicitly nominates itself for use in trailers, it's not going to work there.

Note that I'm not suggesting that we require generic message processors to enforce this (although they're free to try, of course, e.g., in Web Application Firewalls). What I am suggesting is that when someone generates a trailer, they do so only when the header definition allows it, and when someone consumes a trailer, likewise the definition had better allow it. 

We also should remove the language saying that trailers are "identical" and "processed as if they were appended to the message's header section", when clearly they're different.

While we're at it, we should probably at least discourage implementations from moving trailers into the headers, or presenting trailers as headers. I know that historically this may have happened, but I don't *think* it's the case with much modern software.

Cheers,