Subject: JFV and Common Structure specifications

From: Mark Nottingham <[email-masked]>
Date: Mon, 21 Nov 2016 13:38:48 +1100

In Seoul, we discussed both of these specs:
  https://tools.ietf.org/html/draft-ietf-httpbis-jfv
  https://tools.ietf.org/html/draft-kamp-httpbis-structure

Draft minutes:
  https://github.com/httpwg/wg-materials/blob/gh-pages/ietf97/minutes.md#draft-ietf-httpbis-jfv-and-draft-kamp-httpbis-structure

The feeling in the room was that we should abandon the JFV draft and adopt the structure draft in its place, with the understanding that it better reflected our current thinking in this area.

If you have concerns about this, please bring them up on list ASAP; otherwise we'll proceed.

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Martin Thomson <[email-masked]>
Date: Mon, 21 Nov 2016 14:16:59 +1100

> The feeling in the room was that we should abandon the JFV draft and adopt the structure draft in its place, with the understanding that it better reflected our current thinking in this area.

This seems like a good plan.  PHK's work is closer to what we do today
and therefore less disruptive.

We discussed some of the more ambitious features of the common
structure document, such as integer dates and the angle bracket; is
the intent to pursue these separately?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 21 Nov 2016 09:23:01 +0000

--------
In message <CABkgnnV8=[email-masked]>, Martin Thomson writes:
>> The feeling in the room was that we should abandon the JFV draft
>and adopt the structure draft in its place, with the understanding
>that it better reflected our current thinking in this area.
>

My personal opinion: Yes, and no, in that order.

The integer date is only a "gedankenexperiment" in this draft,
included simply to indicate one potential future benefit of CS.

To actually deploy integer Date in CS format in H1, will require a
separate draft which goes into how detection/negotiation works.

A HPACKbis or H3 protocol could/should use a binary CS serialization
and could decide to convert H1 Date headers to CS integers during
transmission, but that would also be in a separate draft.

I think using the angle brackets to say "this header is common
structure" for privately defined headers should be part of this
draft, so the future HPACKbis/H3 can semantically compress them
without needing a white-list.

There is a good argument for also decorating future standardized
headers with ><, so that the IANA registry white-list does not
need to be monitored in (near) real-time.

I also think we should keep the angle brackets/recursion "trick",
so that complex data structures can be built for privately defined
headers.  Recursion is what makes it possible to convert 1:1 from
JSON to CS and back.

But we should probably caution or even SHALL NOT against using
recursion in standardized headers.

Poul-Henning

PS: I'm personally not terribly happy about the name "Common
Structure", but I found "Http-Header Data Model" even worse.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 22 Nov 2016 10:59:35 +1100

> without needing a white-list.

I'm not yet convinced here.  If that usage came with some other
guarantees (around whitespace use, for example, then maybe the
benefits would outweigh the costs.  I'm definitely opposed to
recursion on the basis that complexity could kill this.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 22 Nov 2016 07:49:51 +0000

--------
In message <[email-masked]>, Martin Thomson writes:
>benefits would outweigh the costs.

I think I have a comment in there somewhere, about whitespace
being acceptable on input (to cater for the existing RFC723x
headers) but not generated on output.

>  I'm definitely opposed to

If you don't like recursion, then don't use recursion ?

I totally agree that recursion should not be used in
in standards headers, but they are used in private headers
and if want any chance there, we have to support it.

Also: We should deliver tools, not policies.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 22 Nov 2016 19:05:46 +1100

> On 22 Nov. 2016, at 6:49 pm, Poul-Henning Kamp <[email-masked]> wrote:
> 

[ personal response ]

This phrase is starting to lose meaning for me, as it seems like it could be used as a critique of most any decision about a specification that's out of favour. 

E.g., given that headers themselves are constrained (effectively, a list of tuples with semantics around combination, naming, etc.), should we remove the "policy" from them and make it a bucket of bytes? Or is that too focused on a policy of byte-based computing?

The insistence of a linear interpretation of time implied by discussions around dates is likewise a policy discussion. Let's not even get into numbers...

Adding features like recursion has a real cost in complexity, cognitive load, bugs and security. All of these things add friction to a standard, and can make or break it. That's not to say that we shouldn't do it, but dismissing these concerns with a catchphrase isn't going to be sufficient (at least for me).

Cheers,

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 22 Nov 2016 09:51:35 +0000

--------
In message <[email-masked]>, Mark Nottingham writes:

>Adding features like recursion has a real cost in complexity, cognitive 
>load, bugs and security. All of these things add friction to a standard, 
>and can make or break it. That's not to say that we shouldn't do it, but 
>dismissing these concerns with a catchphrase isn't going to be 
>sufficient (at least for me).

I would hope not!

But I still think it is a very important distinction to make
when we are talking about a general purpose data structure.

The inspiration for my CS analysis was your own complaint about
needing a dozen bespoke parsers.  I am just trying to replace
one more parser than you were thinking about, when you said that.

Some people want to pass a MYPRIV: header with a dictionary of
dictionaries of lists from one side of their web-app to the other,
and that is their choice.  Our opinion and policies does not,
should not and will not count.

Today we see people resort to base64(JSON) or even base64(gzip(JSON))
to scratch that itch, both of which rob us of the chance to
improve serialization efficiency.

If we make CS, *as a data structure*, support general depth, they
will have the choice of using that instead, and if we do our job
right, HPACKng/H3 will do a better job moving their data.

If we are really lucky, somebody will write a JS and PHP package,
give it a fancy name and people will start using that instead.

But giving CS that generality does not mean we have to use it
ourselves.  As I have said repeatedly, I am totally down with a
severe restriction, or preferably outright ban, on recursion in
the HTTP headers we put into IETF documents.

But I see absolutely no advantages for us or anybody else, in trying
to impose that policy on everybody else, by delivering CS as a less
capable tool[1].

> E.g., given that headers themselves are constrained (effectively,
>a list of tuples with semantics around combination, naming, etc.),
>should we remove the "policy" from them and make it a bucket of
>bytes? Or is that too focused on a policy of byte-based computing?

I sort of tried to push for something like that during the rush to
H2, but back then there seemed to be no appetite for it.

I would far prefer if a HTTP message consisted of four onion layers
instead of just header/body like today:

1. Routing information (Host/authority, URL up to '?' and session-id)
	All load-balancers will need to look at.

2. Transport headers (I-M-S, Vary, Range, Content-Encoding, Transport-Encoding)
	Endpoints and proxies also needs these

3. End-to-end metadata (Content-Type, Cookies, MYPRIV: from above)
	Application only

4. End-to-end data (the message body)
	Applications only

(In that model:  No CS-recursion in layer 2, free for all in 3.)

>Adding features like recursion has a real cost in complexity,
>cognitive load, bugs and security.

Absolutely true.  But if by being more general and usable you can
prevent having to have two standards to pay attention to, the
net benefit is very big.

Poul-Henning

[1] In my very extensive study of computing history, I have _never_
found an instance where that strategy worked for anybody, and
computing today is litered with examples of the opposite: C++ vs
C, PHP vs Perl, AMD64 architecture vs Itanic, GPL/GCC vs CLANG,
and so on.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 23 Nov 2016 11:56:53 +1100

> to impose that policy on everybody else, by delivering CS as a less
[...]
> C, PHP vs Perl, AMD64 architecture vs Itanic, GPL/GCC vs CLANG,

JSON.  The principle you are looking for is "just barely good enough
to work".  And that applies to the web in spades if you want great
object lessons.

Let's not talk about policy and instead concentrate on cost-benefit.
Mark has summarized the costs quite well, I think.  I would go further
and point out that a recursive syntax with quoting cannot have
elements that are skipped by a parser, which I find results in simply
externalizing those costs.

The onus on you is to justify these costs in terms of tangible
benefits.  Concrete use cases might help.

(BTW, I like your onion layers concept, though I'm sure that Roy would
chime in and point out that this is a feature of Waka.)

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 23 Nov 2016 11:26:14 +1000

> chime in and point out that this is a feature of Waka.)
​(And that Content-Encoding has been bumped down to transport, instead of
end-to-end metadata.)

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 23 Nov 2016 07:28:06 +0000

--------

>externalizing those costs.

Have you ever implemented SOM, EOM and EOT according to spec in any
of the programs you have written to deal with ASCII text ?

No ?

So can we agree that the cost of those three ASCII characters have
been very close to zero for you ?

Just because a standard contains an extension point, does not mean
you have to implement that extension - unless you actively decides
to use it.

And I keep repeating:  There should be no recursion in IETF defined
HTTP headers, but we should leave the extension there for end-to-end
usage.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Nov 2016 18:46:58 +0100

> If you have concerns about this, please bring them up on list ASAP; otherwise we'll proceed.

I'm +1 on anything that helps us to improve the current situation.

That said, we need to keep in mind that at least one of the arguments 
brought up against JSON applies to this proposal as well -- if we 
consider the underlying data model to be a dictionary, we'll have to 
figure out what to do when that parameter occurs multiple times.

Things are relatively simple when the parser controls this (it can do 
one of take first/take last/abort), but it'll get harder with streaming 
APIs.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 23 Nov 2016 18:39:04 +0000

--------

>figure out what to do when that parameter occurs multiple times.

Yes, we do, but only for each individual header which uses CS.

In difference from JSON, there is nothing in CS which cares either way.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Nov 2016 19:42:31 +0100

> In difference from JSON, there is nothing in CS which cares either way.

How does this work with the hope of using common parsers? Do they need 
to be configurable with respect to that?

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 23 Nov 2016 20:58:35 +0000

--------

>to be configurable with respect to that?

I see it more as a matter of API design, but yes, "blind" parsing
is impossible, because our header definitions are ambiguous and/or
incomplete [1].

Let me try to tackle the general version of this issue, instead
of just the specific one you refer to.

First, I want to make it absolutely clear that when we talk about
"parsers" in this context, we are really talking about deserializers
for a particular serialization of CS.

Right now we only have the HTTP/1 serialization, but we should keep
HPACKng/H3 firmly in the picture, because they are a large part of
the reason to even consider CS in the first place.

We normally see parsers/deserializers as something which produces
an exact and unique output stream, but they dont have to be, and
in the case of CS, because it is synthesized from existing and loose
definitions, they cannot be (see [1])

If we take a hypothetical HPACKng/H3 serialization it is almost
certain that it could/would communicate type information, because
that is where the money is for improved compression:

	number={2345}, ascii_string={foo}, identifier={bla}

The HTTP1/1 serialization allows us to derive some type distinctions
for instance string vs. identifier, but other types we cannot
distinguish, in particular number vs. identifer.

In practice there are only two ways the H1 deserializer/parser can
find out if {2345} is a number or an identifier:

A)  It can have a schema for the header under consideration,
    which says what it is.

B)  The "schema" instead lives in code which calls the parser,
    using either the gimme_number() or the gimme_identifer()
    entry points.

Given the state of art in "schemas", the you would almost certainly
implement A) as a layer of header-specific subroutines calling B).

I don't think this ambiguity is a problem however, because the
conversions between the ambiguous types are trivial.

If you have an identifier and find out you need a number,
you have briefly postponed the unavoidable call to strtoul(3),
and the conversion is safe and unique.

In the future we will have more exiting corner cases: H3 deserializer
hands over number={0.3} but you want identifier.

When we define that H3 serilization, and the H1->H3 inter-op rules,
we will have to ensure that a HTTP1.1 identifier {000.003} doesnt
arrive as {0.3} in the other end.  This is no big deal, worst case
is numbers which are not in canonical format, being transmitted as
identifiers with lower compression efficiency.

And so, to get back to the answer to your question:

I don't know of any HTTP headers today which allow duplicate indicies
in their dictionaries.

But I also don't know anything which prevents them from existing
outside my bubble or from being defined by this WG in the future.

As for the question of deep structure/recursion I would prefer to
have CS be general, and let the individual HTTP headers impose the
limitations on what they allow, and I strongly support that "official"
HTTP headers are banned from using both deep structure and duplicate
dictionary indicies.

However, if the sense of the WG is that "Postel Was Wrong", and
that we should tighten bolts and nuts because he was, CS should
loose both the recursion and unicode_string, both of which are
forward-looking speculative extensions of the vocabulary over the
set of rfc723x headers from which CS is synthesized.

But if that is the case, we should also deprecate RFC5987 instead
of bis'ing it ?

Poul-Henning

[1]  Let me show you one rabbit-hole I passed through along the way:

We probably all tend to read and understand ABNF top down, but look
at RFC723x bottom up, and you find:

	tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
	    "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA

	token = 1*tchar

	type = token

	subtype = token

	media-type = type "/" subtype *( OWS ";" OWS parameter )

	Content-Type = media-type

Which means that as far as RFC723x is concerned, these are both OK:

	Content-Type: application/666

	Content-Type: -3.14159/...---...

It would be tempting to assume that if we see the '/' then the two
tokens surrounding it are not numbers, but there is no foundation
to build that assumption on.

RFC2046 enumerates seven "initial" types, but doesn't say anything
about the shape of the namespace.

The IANA registry has no guidance about it either and "Ask the
IANA Expert Reviewers" is not a valid parser algorithm.

Nothing anywhere says that the next media type being defined cannot
have the subtype defined as "a number", with the implied or explicit
understanding that 'foo/3' sorts before 'foo/10'.

[2]