Subject: Re: identifier in draft-ietf-httpbis-header-structure-01.txt

From: Alex Rousskov <[email-masked]>
Date: Mon, 1 May 2017 09:44:59 -0600

On 04/30/2017 12:50 PM, Poul-Henning Kamp wrote:
>>> Header Structure is not a syntactical specification, 

>> You have fooled several people into thinking that the draft specifies
>> syntax rules (among other things). 
>> [...]
>> IMHO, you have to decide where to stop:

> Exactly!

We may be talking past each other: You are discussing whether defining a
sequence of named dictionaries (in addition to the sequence building
blocks) is a good idea. I am discussing ambiguous grammar used to define
those building blocks. Both problems/questions are important and
answering one does not answer the other.

> The only syntax ("parser level") rule is "The data model of Common
> Structure is an ordered sequence of named dictionaries."

Perhaps our "parser" and "syntax" definitions differ, but the draft has
many other syntax ("parser level") rules IMO. The rule you cite as the
only one, is a parser-level rule only if it is based on syntax rules
that allow the parser (including the lexer/tokenizer, if any) to extract
the building blocks like "identifiers", "numbers", and "integers". I was
talking about needless ambiguity in that extraction.

> Maybe it should stop instead at the "lexical" level, and only define
> a family of "tokens", and leave it to the individual header-specifying
> documents to define their precise order?

A. If the goal is defining as many existing fields using as few rules as
possible, then forcing all fields into h1-common-structure is probably
the right approach. I do not think this should be the goal though (more
on that below), and having One Rule For All is irrelevant when it comes
to writing efficient HTTP parsers: If I want efficiency, I am not going
to define Date NG as a list of ambiguous elements!

B. If the goal is limiting needless token inventions while allowing
efficient parser designs, then h1-common-structure (as defined now) is
irrelevant and we should focus on a small set of unambiguous elements
(i.e., defining and optimizing syntactical vocabulary).

IMHO, goal A is invalid as stated. We should not pursue it! Instead, we
should turn it inside out and use h1-common-structure not as a rigid
structure for all future fields, but as an "understanding tool" of past
ones. Yes, we can describe many old fields (and many future ones!), as
h1-common-structure, and that insight _is_ important, but it is not that
useful for building efficient production parsers and defining new fields
(that can be efficiently parsed and correctly interpreted).

After we define a small set of unambiguous elements (goal B) for
building field grammars and define popular fields using our elements, we
can still limit needless structure inventions by adopting something like
the following (unpolished!) rules for any future field grammar:

* MUST be unambiguous
* MUST use flat comma-separated lists for any repetition
* SHOULD be derived from the grammar elements defined in the draft
* SHOULD avoid repetition
* SHOULD reuse one of the existing field grammars

As an HTTP agent developer, I do not really care much if a new field I
need to interpret uses one of the N supported field grammars or adds an
N+1st one: It is easy for me to add efficient support for another
(draft-compliant!) field grammar to my software because all the building
blocks are there, and I naturally have to add code to store that field
parsed/interpreted value anyway.

In summary: Keep something like h1-common-structure as an insight but
remove it from the syntax-level rules. The syntax level rules in the
draft should not attempt to unify everything under One Rule but provide
excellent building blocks for handling past and future fields. N good
rules is much better than one unusable rule! For future fields, focus on
unambiguous/efficient/correct parsing/interpretation, using new
constructs (like the inverted ">...<" bracketing you invented) as needed.

Cheers,

Alex.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 01 May 2017 16:17:50 +0000

--------
In message <[email-masked]>, Alex Rousskov writes:

>We may be talking past each other

I think we are.

>A. If the goal is [...]

To be brutally honest, it has never been entirely clear what the goal
of this document is :-)

It came into existence after attempts to fit almost-but-not-quite-JSON
into HTTP headers, because I had a trainride from Stockholm to
Copenhagen and Mark had given me the idea to find out what kind of
structure our existing headers actually fit into.

Because of the JSON heritage, I approached it as "can we synthesize
a format which will allow us to move the same data as JSON could have
moved, and the draft as currently written (still) reflects this, despite
active lack of enthusiasm for this aspect in the WG.

As a beneficial side-effect, a HPACK2 which could semantically compress
headers more efficiently than HPACK might be possible.

> [...] and we should focus on a small set of unambiguous elements [...]

That is simply not possible for existing headers.

Current header definitions make 3.14159265 a valid "token":

   tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
    "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA

   token = 1*tchar

So there is no way to tell what you have when you encounter that,
unless you know which header you are in and where in that header
you are.

What we can do is make it easier to get from the definition of
the header to code which can parse it, or at least check its
validity.

And those (machine-readable!) definitions could still be used to
produce a semantic compressor (HPACK2 or whatever).

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 1 May 2017 11:03:15 -0600

> To be brutally honest, it has never been entirely clear what the goal

I applaud your honesty, but since this is a WG draft now, the lack of a
clear goal is no longer your personal fault :-). I hope we can fix that
flaw before it becomes fatal.

> That is simply not possible for existing headers.

Can we define all existing headers using a small set of unambiguous
elements? Probably not, but that should not be our goal. We only need to
cover common usage of common fields plus (nearly) all future ones.

> Current header definitions make 3.14159265 a valid "token":

Yes, but _we_ do not have to define 3.14159265 as both "number" and
"identifier". AFAICT, we can define it as "number" only, excluding
tokens that start with [-]DIGIT from our "identifier" definition.
Ambiguity problem solved! This works for both parsers and generators.

> unless you know which header you are in and where in that header

There is a way to tell for all the important use cases AFAICT.

Yes, if I am building a generic parser that wants to understand senders
intent in _all_ legacy cases, then my parse tree cannot use some of the
draft elements "as is", but since understanding senders intent in all
cases is an unsolvable problem in legacy HTTP, we should not focus on
that esoteric use case too much!

A generic parser that might confuse a number with identifier is possible
to build for legacy HTTP while perfect parsers are possible for new
>fields<, and that is good enough. Same for generators.

HTH,

Alex.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 01 May 2017 17:41:58 +0000

--------

>cover common usage of common fields plus (nearly) all future ones.

We can come very close, but you will always need to know what
header you are parsing to know how to parse it.

This is the "bespoke parsers" the draft talks about.

It would be a giant leap forward, if we could respecify the
existing headers in a format which would allow those parsers
to be machine generated directly from the spec.  (See strawman
in my first reply to you)

>"identifier".

For future headers:  Absolutely.

For existing headers: No, we don't get to narrow their definition
that way.

>that esoteric use case too much!

I don't think it is unsolveable, after all, the current headers have
ABNF definitions and they may be flawed, but they are not random.

And if we want decent compression from a semantic(ally informed)
HPACK2/QUIC/whatever serializerers, they, ipso facto, has to start
with a generic parser for the most common headers to be worth
the effort.

>>fields<, and that is good enough. Same for generators.

Well, that really depends on what the goal is, doesn't it ?  :-)

And yes, I'm open for input on that

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 1 May 2017 14:49:42 -0600

On 05/01/2017 11:41 AM, Poul-Henning Kamp wrote:
> Alex Rousskov writes:

> This is the "bespoke parsers" the draft talks about.

It is not an all-or-nothing choice. For example, if the draft grammar is
good enough, I can write an efficient generic key=value parser that
handles arbitrary key=value pairs and then reuse that general parser in
20 bespoke field parsers "as is", without letting it know which field is
being parsed. Will that approach cover all legacy headers? No. However,
it may cover the vast majority of the headers my agent needs.

On the other hand, if the draft grammar is not good enough, then I will
not be able to create and reuse an efficient generic key=value parser
(without redefining the grammar so that it becomes usable for that
purpose) and may end up with either a slow "everything is a token"
generic parser or end up with 20 bespoke field parsers, each parsing its
own key=value flavor. Neither is terrible, but we should be able to do
better.

Also, if the grammar says that everything that looks like a number is a
number and nothing else is, then I _can_ correctly (and efficiently!)
parse a number field without knowing that it is a number field. Will I
occasionally parse an extension field as a number while the sender was
sending what is an identifier to her? Yes! Should the draft care about
such rare cases that hurt nobody? No, it should not (as long as those
cases are indeed rare and hurt nobody).

> to be machine generated directly from the spec.

I still think it is possible to define a grammar like that. Side note: I
do not think we should limit ourselves to grammars that some existing
parser generator tool(s) can grok without any human involvement. That
100% automation goal is likely to limit a human-friendly grammar too
much, making it less useful in practice.

> For existing headers: No, we don't get to narrow their definition

Why not?! For example, I do not understand why you insist that the draft
cannot write

  "q=" number

when defining an Accept field _and_ exclude "number" from "identifier".
I do not see why doing something like that will break or prohibit
something _important_. Please give an example of such breakage.

> ABNF definitions and they may be flawed, but they are not random.

It is unsolvable because an extension header does not have a (known to a
generic parser) ABNF.

Sure, I can find the right ABNF if I want to support that extension in
my agent and add a custom parser for it (perhaps even generating one
from that ABNF!), but a generic parser can say nothing about senders
intent in this case. The extension field is just an opaque blob (or a
list of blobs, but even that is a risky step!) to such a parser. The
legacy grammar does not allow a generic parser to discover the extension
field structure. We can use a more strict grammar elements to provide
building blocks for the interesting/important cases.

Alex.