Subject: Re: #536: clarify extensibility for :pseudo header fields

From: "Martin Nilsson" <[email-masked]>
Date: Tue, 01 Jul 2014 09:59:17 +0200

On Tue, 01 Jul 2014 06:33:54 +0200, Matthew Kerwin <[email-masked]>  
wrote:

> ​Ideologically I prefer setting up a registry, since I expect (hope?)  
> that extensions will end up informing HTTP2bis and/or >bootstrapping  
> HTTP3.
>
>> However, I can't think of a use case for a :header that can't equally  
>> be served by an extension frame.  Consider this strawman, >which is the  
>> best proposal I can come up with for safe but extensible :headers:
>
>> ]  An endpoint must raise a PROTOCOL_ERROR if it receives an unknown
> ]  :header. Extensions may define new :headers on the conditions that
> ]  1) support has to be advertised before the header can be sent, and
> ]  2) the receiver has to translate/remove it before forwarding, if the
> ]  downstream peer hasn't advertised support.
>
>> (editorial issues and tone notwithstanding)

I like this. It makes it much easier to tie request specific meta data to  
the correct request than separate extension frames.

>
>> Unless the new :header has widespread support off the bat, it's going  
>> to end up being used hop-by-hop, which -- because of the >resulting  
>> churn in HPACK contexts -- makes it less valuable than an extension  
>> frame.
>
>> A less restrictive proposal (e.g. only forcing the translation/removal  
>> at a 1.1 gateway) provides end-to-end extension >:headers in a pure  
>> HTTP2 environment, but what's the utility of a header that has to be  
>> stripped if it passes a 1.1 leg on its >journey?

Extension frames don't survive 1.1 gateways either, so I don't see this as  
a good argument.

/Martin Nilsson

>
>> So I suspect option 2 is the better choice.​​
>
>
> -- Matthew Kerwin
>  http://matthew.kerwin.net.au/

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

---

From: Martin Thomson <[email-masked]>
Date: Tue, 1 Jul 2014 10:49:43 -0700

On 30 June 2014 23:47, Julian Reschke <[email-masked]> wrote:
>> 2) Make the list of acceptable :headers static with respect to the ALPN
>> token
>
> ...which would still allow revising HTTP/2, adding new pseudo header fields.

That.  An update can establish a registry if it likes.

I'm split between demanding RST_STREAM+PROTOCOL_ERROR or ignoring
unknown values.  Slight preference for the former though.  I think
that's consistent with Matthew's proposal.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 2 Jul 2014 12:11:38 +1000

> that's consistent with Matthew's proposal.

+1 for hard fail.

--
Mark Nottingham   https://www.mnot.net/

---

From: David Krauss <[email-masked]>
Date: Wed, 2 Jul 2014 10:21:31 +0800

>> that's consistent with Matthew's proposal.

That gives applications a tough choice.

Perhaps the corresponding non-pseudo header should always be a fallback, implemented at the API level, if the pseudo header is absent?

I’m not clear on what appeal pseudo headers currently have at all, except to a punctuation fetish.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 2 Jul 2014 12:24:35 +1000

> I’m not clear on what appeal pseudo headers currently have at all, except to a punctuation fetish.

Pseudo headers shouldn't be available to the application, full stop; they're just a mechanism to get what was special-case syntax in HTTP/1 into general header syntax in /2. 

They contain a colon specifically because it's illegal in field names, so that they're clearly different. Making them extensible is IMO inviting them to be exposed by APIs as headers or similar, and that's a land of rich bugs and security problems.

---

From: David Krauss <[email-masked]>
Date: Wed, 2 Jul 2014 10:48:37 +0800

> Pseudo headers shouldn't be available to the application, full stop; they're just a mechanism to get what was special-case syntax in HTTP/1 into general header syntax in /2. 

The application is required (or at least, asked nicely) to fill them. And they can be freely read at the other end. The restriction is on defining new ones.

ALPN based specs will need to decide whether to sacrifice compatibility with HTTP/1.1 proxies. Where the sacrifice is made, we’ll still see applications choosing to forgo pseudo headers and omit the colon, effectively defining a new non-pseudo. That’s the tough choice.

> They contain a colon specifically because it's illegal in field names, so that they're clearly different. Making them extensible is IMO inviting them to be exposed by APIs as headers or similar, an…

Different how? A field name with a colon is also a field name. It’s not an HTTP/1.1 field name, but I must be missing something here.

Nothing in the current spec suggests they shouldn’t be accessible as headers or similar. Without some indication to the contrary, APIs will expose them simply by default, and the damage will be done.

I’m not suggesting extensibility. My initial reply to this thread was the strictest so far, that pseudo headers should only be added where deemed necessary by HTTP itself. This only really makes sense with additional, new restrictions such as that they should be encoded first in a header set representation. But without restrictions the colon just seems like a new color of bike shed paint.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 1 Jul 2014 19:50:10 -0700

#2/ hard fail looks good to me here.
imho keys beginning with ':' are reserved for future versions of the
protocol, and should not be treated the same as others.
-=R

On Tue, Jul 1, 2014 at 7:24 PM, Mark Nottingham <[email-masked]> wrote:

> > Perhaps the corresponding non-pseudo header should always be a fallback,
> implemented at the API level, if the pseudo header is absent?
> >
> Pseudo headers shouldn't be available to the application, full stop;
> they're just a mechanism to get what was special-case syntax in HTTP/1 into
> general header syntax in /2.
>
> They contain a colon specifically because it's illegal in field names, so
> that they're clearly different. Making them extensible is IMO inviting them
> to be exposed by APIs as headers or similar, and that's a land of rich bugs
> and security problems.
>
>
> --

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 2 Jul 2014 14:37:42 +1000

> The application is required (or at least, asked nicely) to fill them. And they can be freely read at the other end. The restriction is on defining new ones.

Yes; read my statement as "Pseudo headers *not defined by the ALPN token* shouldn't..."

> ALPN based specs will need to decide whether to sacrifice compatibility with HTTP/1.1 proxies. Where the sacrifice is made, we’ll still see applications choosing to forgo pseudo headers and omit the colon, effectively defining a new non-pseudo. That’s the tough choice

It's a lot more than that; if they're trying to map into HTTP semantics, they'll have to figure out how to represent the headers in existing APIs. 

> Different how? A field name with a colon is also a field name. It’s not an HTTP/1.1 field name, but I must be missing something here.

The range of HTTP header field names is not magically changed by HTTP/2; it's still as defined in RFC7231 <http://tools.ietf.org/html/rfc7230#section-3.2>.

> Nothing in the current spec suggests they shouldn’t be accessible as headers or similar. Without some indication to the contrary, APIs will expose them simply by default, and the damage will be don…

<http://http2.github.io/http2-spec/#HttpHeaders>:

"""
Header field names that start with a colon are only valid in the HTTP/2 context. These are not HTTP header fields. Implementations MUST NOT generate header fields that start with a colon, and they MUST ignore and discard any header field that starts with a colon. In particular, header fields with names starting with a colon MUST NOT be exposed as HTTP header fields.
"""

---

From: David Krauss <[email-masked]>
Date: Wed, 2 Jul 2014 13:46:27 +0800

> It's a lot more than that; if they're trying to map into HTTP semantics, they'll have to figure out how to represent the headers in existing APIs. 

Simply strip the colon. I suspect that even when the ALPN spec mentions nothing of the sort, it will be a practical workaround anyway. It’s obvious enough to warrant a specific prohibition, if you want folks not to do this.

Graceful degradation dictates that the subset of HTTP/2 usage surviving translation through HTTP/1.1 should be maximized.

I think we’re basically viewing the situation through different glasses: you’re supposing people will use the protocol as intended, and I’m supposing that people will use whatever workarounds maximize compatibility. Call it devil’s advocate.

> “""

OK, thanks, I stand corrected. This is the rule that we are currently adjusting, and my suggestion was only the status quo.

Still, even such a prohibition against exposing as header fields doesn’t prevent “something similar.” Are general HTTP/2 APIs really going to force users to wait for support for each ALPN token, or will they allow for early adoption and customization by exposing the pseudos?

This territory sounds like it will be difficult to legislate, because competitive pressures will push the other way.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 2 Jul 2014 15:53:23 +1000

> Graceful degradation dictates that the subset of HTTP/2 usage surviving translation through HTTP/1.1 should be maximized.

Um, no. Collapsing the two namespaces into one is a security nightmare along the lines of <https://www.owasp.org/index.php/HTTP_Request_Smuggling>.

---

From: David Krauss <[email-masked]>
Date: Wed, 2 Jul 2014 14:04:34 +0800

> Um, no. Collapsing the two namespaces into one is a security nightmare along the lines of <https://www.owasp.org/index.php/HTTP_Request_Smuggling>.

Then, say so in the spec that ALPN tokens shouldn’t alias pseudos with headers and ALPN-basis APIs shouldn’t expose headers as pseudos (the converse of the existing restriction).

Just to point out, you don’t have that problem in the first place if you don’t open the second namespace.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 2 Jul 2014 16:07:51 +1000

> Just to point out, you don’t have that problem in the first place if you don’t open the second namespace.

Now I'm confused. We're currently talking about whether extra :headers are a hard error when http/2 is in use. What's an ALPN-basis API?

---

From: David Krauss <[email-masked]>
Date: Wed, 2 Jul 2014 14:10:03 +0800

> Now I'm confused. We're currently talking about whether extra :headers are a hard error when http/2 is in use. What's an ALPN-basis API?

Something used inside a networking framework to expose the full expressive power of HTTP/2 to a modular implementation of WebSockets or whatever.

The API which serves the layer that knows which :headers are valid or not.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 02 Jul 2014 06:30:11 +0000

In message <[email-masked]>, David Krauss writes:
>
>On 2014=9607=9602, at 2:07 PM, Mark Nottingham <[email-masked]> wrote:
>
>> Now I'm confused. We're currently talking about whether extra :headers =
>are a hard error when http/2 is in use. What's an ALPN-basis API?

I always assumed that the :headers only were HTTP/2s internal way of
representing the first line of HTTP/1 messages on the wire, not a way
to add an entirely new namespace to HTTP headers/metadata ?

If some extensions need namespaces in HTTP headers, we should add
namespaces which will (or can be made to) work in both HTTP/1,
HTTP/2 and any future HTTP versions, not latch onto an implementation
detail in one particular version of HTTP.

Traditionally namespaces delimiters have been ':' but that will certainly
break too many HTTP/1 implementations to be feasible, but there are plenty
of ASCII characters available that could be used: '=', '/' and so on.

In other words:  Please limit :headers to the ones in 8.1.3.1 and 8.1.3.2
and make attempts to smuggle any other :header through a hard protocol error.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 2 Jul 2014 16:31:50 +1000

Personally - 

> to add an entirely new namespace to HTTP headers/metadata ?

+1

> detail in one particular version of HTTP.

+1 -- although I think they're just headers... it could be that we add structure to header field-names (indeed, I saw that suggested privately very recently), but we're not yet there.

> and make attempts to smuggle any other :header through a hard protocol error.

+1

---

From: Martin Thomson <[email-masked]>
Date: Wed, 2 Jul 2014 09:37:09 -0700

>> and make attempts to smuggle any other :header through a hard protocol error.

I think that I've seen enough support for this approach, so I'll
prepare a change that says exactly that.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 2 Jul 2014 09:53:16 -0700

> prepare a change that says exactly that.

https://github.com/http2/http2-spec/pull/546

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 15 Jul 2014 14:54:24 +1000

That looks good, Martin. I haven't heard any pushback, so marking editor-ready.