Subject: Re: draft-ietf-httpbis-jfv: what's next

From: Matt Menke <[email-masked]>
Date: Thu, 13 Oct 2016 20:53:43 +0000

I think the draft looks good, but have a couple comments:

The token rule in RFC7230 already includes asterisks, so I don't think identifier or token_or_asterix is needed.

Would it make sense to codify behavior if a part of a h1_common_structure value fails to parse, at least if it uses the proposed "><" format)?  I suspect what browsers do is inconsistent here, and having some official rule (ignore the entire element vs ignore the entire line vs ignore the broken parameter) seems like it would be worth having?  I'd go with throw away the entire header line, if it uses the new format and that happens, since that's easiest to standardize on.

I think it's unfortunate that the HTTP/1 serialization can't distinguish between identifiers, numbers, and timestamps.  It means that per-specific-header logic will have to be responsible for that extra round of parsing for HTTP/1 headers.  If H2/QUIC goes with a fancier binary encoding scheme that doesn't have that issue, that means that those formats would not have to do the extra found of parsing, which seems a bit weird, but I don't see a good way around that asymmetry.

On single/multiple headers:  The draft has a comment about this, but it doesn't really comment on whether the following is legal:

Foo: >bar<
Foo: >baz<

Or:

Foo: >bar<, >baz<

According to Chrome's current logic, both would be valid and are equivalent (Assuming >< is treated like quotes, with extra logic for nested quotes, and "Foo" isn't one of a small set of non-coalescing headers).  Is different behavior intended in these two cases?

---

From: Kari Hurtta <[email-masked]>
Date: Sat, 15 Oct 2016 10:07:03 +0300 (EEST)

> The token rule in RFC7230 already includes asterisks, so I don't think identifier or token_or_asterix 

Yes. 

3.2.6.  Field Value Components
https://tools.ietf.org/html/rfc7230#section-3.2.6

|     token          = 1*tchar
|
|     tchar          = "!" / "#" / "$" / "%" / "&" / "'" / "*"
|                    / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
|                    / DIGIT / ALPHA
|                    ; any VCHAR, except delimiters

> whether the following is legal:

3.  HTTP/1 serialization of HTTP header Common Structure
https://tools.ietf.org/html/draft-kamp-httpbis-structure-00#section-3

seems NOT accept later (this is problem)

|       h1_common-structure-header =
|               ( field-name ":" OWS ">" h1_common_structure "<" )
|                       # Self-identifying HTTP headers
|               ( field-name ":" OWS h1_common_structure ) /
|                       # legacy HTTP headers on white-list, see {{iana}}

But 

Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing
https://tools.ietf.org/html/rfc7230

says

3.2.2.  Field Order
https://tools.ietf.org/html/rfc7230#section-3.2.2

|   A recipient MAY combine multiple header fields with the same field
|   name into one "field-name: field-value" pair, without changing the
|   semantics of the message, by appending each subsequent field value to
|   the combined field value in order, separated by a comma. 

/ Kari Hurtta

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 15 Oct 2016 09:14:38 +0000

--------
In message <[email-masked]>, Kari Hurtta wri
>> The token rule in RFC7230 already includes asterisks, so I don't think identifier or token_or_asterix 

The "_or_asterix" comes from the intial analysis where I used a much
tighter specification for the name of the dictionaries in the list.

But I forgot to clean this up entirely in the normative ABNF, specifically
it should be:

       identifier = token  [ "/" token ] 

>> whether the following is legal:

This is, but my draft does not define if the two values should be
appended as one list (most sane headers) or treated as separate
instances (Cookie header).

I'm personally against repeat headers, because you cannot act on
a header until you've seen/decompressed the full set of headers.

>> Foo: >bar<, >baz<

This is not inside Common Structure, and I don't think we should
make space for it.

>|   the combined field value in order, separated by a comma. 

Yes, this is what I don't like, and I think we should limit the application
of it by all possible means.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 15 Oct 2016 09:41:07 +0000

--------
In message <CAEK7mvoXqyX3cADJytjU+C158EULgPLbzAb5kiUN=[email-masked]>
, Matt Menke writes:
>The token rule in RFC7230 already includes asterisks, so I don't think =
>identifier or token_or_asterix is needed.

Yes, I just fixed that.

>Would it make sense to codify behavior if a part of a 
>h1_common_structure value fails to parse, at least if it uses the 
>proposed "><" format)?  I suspect what browsers do is inconsistent here, 
>and having some official rule (ignore the entire element vs ignore the 
>entire line vs ignore the broken parameter) seems like it would be worth 
>having?  I'd go with throw away the entire header line, if it uses the 
>new format and that happens, since that's easiest to standardize on.

So this is a bit of a sticky wicket.

Today that is a per-header decision, for instance Accept-Encoding
can safely ignore anything it doesn't understand/parse, whereas
Content-Encoding has to be parsed perfect.

It is also a soft spot which has been used in a number of creative
attacks on deeper layer in HTTP/1 sandwiches.

Looking forward, if we want to be able to use CS to build H3
compression, we cannot allow CS headers with format errors.

I'm uncomfortable with a rule which says "just ignore", so I would
propose that failure to parse a the CS level should cause a 4xx
error, just like an ascii BEL in a HTTP1 header would.

But please note that this is only at the CS level, how valid CS
which is semantically invalid (ie: "Content-Length: ABCD") should
be handled is outside the scope of this ID.  I'm not even sure
we can give a meaningful "default" rule.

>I think it's unfortunate that the HTTP/1 serialization can't distinguish 
>between identifiers, numbers, and timestamps.

Yes, but we don't really get to decide where we start.

My hope is that we can build a machine-readable specification language
for HTTP headers from which the "semantic parsing" code can be generated,
but that is clearly in the "future work" column.

>It means that 
>per-specific-header logic will have to be responsible for that extra 
>round of parsing for HTTP/1 headers.

Not necessarily.  Parsing CS in HTTP/1 serialization is very trivial
and it is not obvious to me that it always would or should be a
separate step.  With a specification language as mentioned above,
you probably would generate combined CS+semantic parser code.

The big advantage of CS is that we don't need to know the semantics.

If your implementation receives "My-Private-Header: >[...]<" it can
take it apart and present it as a native datastructure, and the
application logic can apply the privatly known semantics to that.

---

From: Martin Thomson <[email-masked]>
Date: Sat, 15 Oct 2016 20:58:29 +1100

> compression, we cannot allow CS headers with format errors.

I tend to agree with this, though there are levels of format errors.
For instance, if you use the >< notation and the < is absent, that's a
flat parse error (I would argue that the < is redundant actually, save
an octet).  But as you say, use of an unknown Content-Encoding is a
semantic-level failure that the parser can't be expected to handle.
Besides, we have to allow for extension in values, and that's hard.

But what I think that Matt is looking for is a grammar that supports
an in-band signal about type so that syntax checking can be done by
the parser (and not by the semantics layer).  That - to me - seems
like a pretty reasonable request.  That means that you get a level of
generic validation.  Based on what you have, you could reduce this to
string/number, but I think that you could be a tiny bit more
adventurous (and include date and binary for starters).

I agree that going any non-trivial distance down this path is a
crapshoot.  The need for rules - or at least any consistency - rapidly
evaporates.  But there is a modicum of value in being able to
distinguish type, if only to allow for the use of better types in that
datastructure you talk about.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 15 Oct 2016 15:49:04 +0000

--------
In message <[email-masked]>
, Martin Thomson writes:
>an octet).

It is redundant, but it might still be a good idea.

Truncation of headers happens a lot more than it should in the wild,
so apart from the recursive role of the '<' I do like that it also
tells you that you are not missing half the header.

>like a pretty reasonable request.

Yes, I agree, but it runs into the very inclusive definition of
RFC7230::token.

We need three markers: '(h1_)number', 'h1_timestamp' and 'h1_blob',
which are all valid 'identifier' (= RFC7230::token) today.

We have three options:

1. Keep using RFC7230::token for 'identifier'

    Then only '<', '>', '{', '}', '[' and ']' are still available.

    We have already given '<', '>' meaning and even though we could
    disambiguate them, I'd really like to avoid overloading.

    So one proposal could be:

	h1_number = '}' number

	h1_timestamp = '{' number

	h1_blob = '[' base64

    Leaving us with only ']' if we forgot something.

    (It rattles my OCD to use "precious" balanced glyphs this way...)

    (For reasons of transmission efficiency I am intentionally not proposing
    formats such as "{#" number "}")

2. Restrict 'identifier'

    If we use a restricted RFC7230::token for 'identifier', we can shake
    some special characters free for type marking duty.

    Maybe:

	identifier = ALPHA *(ALPHA / DIGIT / '_' / '-' / '*')

	h1_number = '#' number

	h1_timestamp = '$' number

	h1_blob = "'" base64 "'"

3. Let the semantic layer sort it out.

    As the draft does today.

    This has best H1/H2/HPACK transmission efficiency.

    This also enforces only the minimum necessary restriction on
    HTTP-header inventors.

    For instance: h1_blob is a valid identifier and thus a valid
    name of a dictionary.

    On the contra side, exploiting such "loopholes" is almost
    guaranteed to hurt H3 compression for that header later on.

I picked 3 based on 'minimum intrusiveness', but I can live with
all three.

Given that a H3 compression likely will serialize the type, the two
first alternatives are probably the most forward-compatible.

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 15 Oct 2016 23:35:19 +0200

> tells you that you are not missing half the header.

And it also helps when headers are duplicated. Below it's becoming
obvious what type of duplication happened, and where :

     Foo1: >blah<, >bar<
     Foo2: >blah, bar<

> 1. Keep using RFC7230::token for 'identifier'
(...)
> 2. Restrict 'identifier'
(...)
> 3. Let the semantic layer sort it out.
(...)

> I picked 3 based on 'minimum intrusiveness', but I can live with

And what about #4 consisting in indicating the encoding in the header
field *name* instead, since we're suggesting to use this for new fields ?
It could be possible to prepend something like "-1", "-2", etc... in front
of the field name to indicate its expected type and how it's supposed to
be decoded, and reserve such fields' name syntax only for these typed
header fields.

Just my two cents,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 15 Oct 2016 22:26:04 +0000

--------

>obvious what type of duplication happened, and where :

True, but that doesn't make me any more well disposed towards
header split/concatenation.

>field *name* instead

That's quite a hack, isn't it ?

I'd prefer not to complicate things that way if I can...

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 16 Oct 2016 07:26:41 +0200

> That's quite a hack, isn't it ?

Not that much after all when you think about it, because our parsers
will have to consider the field name to know how to decode the value.
Content-length will remain integer for example. Thus new fields could
be classified by their name using a prefix for easier sorting/testing.

> I'd prefer not to complicate things that way if I can...

Same for me, don't worry, which is why I'm only suggesting :-)

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 16 Oct 2016 07:08:24 +0000

--------
>will have to consider the field name to know how to decode the value.

At the semantic level, yes.

But not at the serdes level.

And if we pull off the "data dictionary" idea, we can machine generate
parsers which do that for specific cases.

But I would hate to make it part of the definition.

If nothing else, because it takes the 19 existing headers out of
the CS cover.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 17 Oct 2016 20:50:38 +0000

--------
In message <[email-masked]>
, Matt Menke writes:

>I don't think it needs to be that complicated.  If there's just a prefix
>before all types (Say '#' for numbers, '&' for times, and '!' for
>identifiers), we don't have to exclude the prefix characters from
>identifiers, do we?

True, if we prefix all values we can avoid that problem.

However, that would kill one of the major attraction of CS:  That
19 RFC723x headers already conform to it.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 17 Oct 2016 20:53:37 +0000

--------
In message <CAEK7mvomoxSqTs5APVj=[email-masked]>
, Matt Menke writes:

>I don't think we want a 4xx error - with HTTP status error codes, browsers
>display the page contents of the error page, set cookies if needed, etc.  I
>think we'd want a hard failure here, more along what we do if we can't
>establish a network connection.

It's not clear to me what you consider a "hard failure" here ?

In my world this would be 400 like semantics, which I would consider
a "hard failure" ?

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 17 Oct 2016 22:58:11 +0200

> 19 RFC723x headers already conform to it.

This is actually a very good point that we constantly need to keep in mind
and which is the best justification for not using prefixes on field names.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 17 Oct 2016 21:01:15 +0000

--------
In message <[email-masked]>
, Matt Menke writes:

>If we treat it more like a network error, we'll show a
>browser-generated error page, we don't set cookies, and ignore the response
>body entirely.

Just to make sure I understand, if you received a HTTP/1 response
of "ABCD GOAWAY\r\n\r\n" would that be a "network error" in your
terminology ?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 17 Oct 2016 21:14:39 +0000

--------
In message <[email-masked]>
, Matt Menke writes:

>No, that would be treated as an HTTP/0.9 response.

Wow... that's a lot more tolerant than I would have expected.

Do you really still see HTTP/0.9 reponses in the wild ?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 17 Oct 2016 21:28:13 +0000

--------
In message <CAEK7mvq=Z8-E4hm3S-aPpUnsYjAauA-9BbgFkq=[email-masked]>
, Matt Menke writes:

>If we had buy-in from other browser vendors to all drop
>support at once, I think it would be easier to finally remove it, though
>that would potentially leave a lot of end users with unconfigurable
>hardware.

How about really annoying pop-ups for a starter ?

After all, that has been very successful at keeping self-signed
certificates from making NSAs job harder ? :-)

---

From: Mike Bishop <[email-masked]>
Date: Mon, 17 Oct 2016 21:38:28 +0000

Sadly, yes.

-----Original Message-----
From: Poul-Henning Kamp [mailto:[email-masked]] 
Sent: Monday, October 17, 2016 2:15 PM
To: Matt Menke <[email-masked]>
Cc: HTTP working group mailing list <[email-masked]>
Subject: Re: draft-ietf-httpbis-jfv: what's next

--------

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 17 Oct 2016 23:43:55 +0200 (CEST)

> Do you really still see HTTP/0.9 reponses in the wild ?

Yes. Of course subject to debate around what "in the wild" means...

-- 

  / daniel.haxx.se

---

From: Frederik Braun <[email-masked]>
Date: Wed, 19 Oct 2016 10:57:10 +0200

> Yes. Of course subject to debate around what "in the wild" means...

There was a recent security bug with regards to HTTP/0.9 compat [1] that
suggested a coordinated fix among multiple browsers. Browser vendors
decided that they'd start off killing HTTP/0.9 compat on non-standard
ports only, because overall usage was still too high.
Firefox has HTTP 0.9 usage data[2] and so do other browsers. Please
chime in.

Hope that helps,
Frederik

[1] See entry in Bugzilla with links to Webkit and Chrome bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1262128
[2] I'm new to our telemetry dashboard, so maybe there could be a more
useful graph than this one
 https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-09-22&keys=__none__!__none__!__none__&max_channel_version=release%252F49&measure=HTTP_09_INFO&min_channel_version=null&product=Firefox!Fennec&sanitize=1&sort_keys=submissions&start_date=2016-09-16&table=0&trim=1&use_submission_date=0

---

From: Matt Menke <[email-masked]>
Date: Wed, 19 Oct 2016 10:30:30 -0400

With Chrome, about 0.003% of main frame responses are detected as HTTP/0.9,
and 0.005% of subresources are HTTP/0.9.  Since HTTP/0.9 includes no
explicit identifying headers, some of these could be broken servers
responding with nonsense data.

We actually did try to go it alone in remove HTTP/0.9 support, but soon
backed off after trying it in our pre-release channels, because of the
aforementioned incident with the configuration page of a line of home
routers.

On Wed, Oct 19, 2016 at 4:57 AM, Frederik Braun <[email-masked]> wrote:

>  https://telemetry.mozilla.org/new-pipeline/dist.html#!
> cumulative=0&end_date=2016-09-22&keys=__none__!__none__!__
> none__&max_channel_version=release%252F49&measure=HTTP_
> 09_INFO&min_channel_version=null&product=Firefox!Fennec&
> sanitize=1&sort_keys=submissions&start_date=2016-09-16&table=0&trim=1&use_
> submission_date=0
>
>
>

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 19 Oct 2016 16:45:43 +0200

Hi Matt,

> aforementioned incident with the configuration page of a line of home

That's very useful data. On haproxy, we never ever had a single report of
a broken application caused by the fact that we block 0.9 responses. So
your observation of home routers using it tends to confirm our observation
in that it is probably only found in cheap consumer devices (that we never
see) and never on the hosting side (where haproxy is normally installed).

It could be useful to know if Squid sees such responses, because in general
it will not be used to access local network devices but will still face the
internet and a wide spectrum of self-hosted servers that we don't see.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 19 Oct 2016 15:30:34 +0000

--------
In message <[email-masked]>, Willy Tarreau writes:

>a broken application caused by the fact that we block 0.9 responses.

Ditto Varnish (as far as I know...)

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 19 Oct 2016 10:29:56 -0600

> internet and a wide spectrum of self-hosted servers that we don't see.

The Squid Project does not collect customer statistics but there is
plenty of evidence that Squid HTTP/0 code is still in use. Naturally,
most of that evidence comes as bugs related to misinterpreting malformed
HTTP/1 traffic as HTTP/0 but, IIRC, intercepting customers still see
genuine HTTP/0 _requests_ from various scripts and devices. Some of
those requests may expect an HTTP/0 response...

Alex.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 19 Oct 2016 18:40:22 +0200

On Wed, Oct 19, 2016 at 10:29:56AM -0600, Alex Rousskov wrote:
> HTTP/1 traffic as HTTP/0

That's exactly the way I would expect them to report it.

> but, IIRC, intercepting customers still see

Indeed, very good point.

Thanks Alex,
Willy