Subject: If not JSON, what then ?

From: Poul-Henning Kamp <[email-masked]>
Date: Mon, 01 Aug 2016 07:43:34 +0000

Based on discussions in email and at the workshop in Stockholm,
JSON doesn't seem like a good fit for HTTP headers.

A number of inputs came up in Stockholm which informs the process,
Marks earlier attempt to classify header syntax into groups and the
desire the for a efficient binary encoding in HTTP[3-6] (or HPACK++)

My personal intuition was that we should find a binary serialization
(like CORS), and base64 it into HTTP1-2:  Ie: design for the future
and shoe-horn into the present.  But no obvious binary serialization
seems to exist, CORS was panned by a number of people in the WS as
too complicated, and gag-reflexes were triggered by ASN.1.

Inspired by Marks HTTP-header classification, I spent the train-trip
back home to Denmark pondering the opposite attack:  Is there a
common data structure which (many) existing headers would fit into,
which could serve our needs going forward?

This document chronicles my deliberations, and the strawman I came
up with:  Not only does it seem possible, it has some very interesting
possibilities down the road.

Disclaimer:  ABNF may not be perfect.

Structure of headers
====================

I surveyed current headers, and a very large fraction of them
fit into this data structure:

	header: ordered sequence of named dictionaries

The "ordered" constraint arises in two ways:  We have explicitly
ordered headers like {Content|Transfer}-Encoding and we have headers
which have order by their q=%f parameters.

If we unserialize this model from RFC723x definitions, then ',' is
the list separator and ';' the dictionary indicator and separator:

     Accept: audio/*; q=0.2, audio/basic

The "ordered ... named" combination does not map directly to most
contemporary object models (JSON, python, ...) where dictionary
order is undefined, so a definition list is required to represent
this in JSON:

	[
	    [ "audio/*", { "q": 0.2 }],
	    [ "audio/basic", {}]
	]

It looks tempting to find a way to make the toplevel JSON a dictionary
too, but given the use of wildcards in many of the keys ("text/*"),
and the q=%f ordering, that would not be helpful.

Next we want to give people the ability to have deeper structure,
and we can either do that recursively (ie: nested ordered seq of
dict) or restrict the deeper levels to only dict.

That is probably a matter of taste more than anything, but the
recursive design will probably appeal aesthetically to more than
just me, and as we shall see shortly, it comes with certain economies.

So let us use '<...>' to mark the recursion, since <> are shorter than
[] and {} in HPACK/huffman.

Here is a two level example:

	foobar: foo;p1=1;p2=abc;p3=<x1,x2,x3;y1=1;y2=2>;p4, bar

Parsed into JSON that would be:

	[
	    [
		"foo",
		{
		    "p1": 1,
		    "p4": {},
		    "p3": [
			[
			    "x1",
			    {}
			],
			[
			    "x2",
			    {}
			],
			[
			    "x3",
			    {
				"y2": 2
				"y1": 1,
			    }
			]
		    ],
		    "p2": "abc"
	        }
	    ],
	    [
		"bar",
		{}
	    ]
	]

(NB shuffled dictionary elements to show that JSON dicts are unordered)

And now comes the recursion economy:

First we wrap the entire *new* header in <...>:

	foobar: <foo;p1=1;p2=abc;p3=<x1,x2,x3;y1=1;y2=2>;p4, bar>

This way, the first character of the header tells us that this header
has "common structure".

That explicit "common structure" signal means privately defined
headers can use "common structure" as well, and middleware and
frameworks will automatically Do The Right Thing with them.

Next, we add a field to the IANA HTTP header registry (one can do
that I hope ?) classifying their "angle-bracket status":

 A) not angle-brackets -- incompatible structure use topical parser
	Range

 B) implicit angle-brackets -- Has common structure but is not <> enclosed
	Accept
	Content-Encoding
	Transfer-Encoding

 C) explicit angle-brackets -- Has common structure and <> encloosed
	all new headers go here

 D) unknown status.
	As it says on the tin.

Using this as whitelist, and given suitable schemas, a good number
of existing headers can go into the common parser.

And then for the final trick:   We can now define new variants of
existing headers which "sidegrade" them into the common parser:

	Date: < 1469734833 >

This obviously needs a signal/negotiation so we know the other side
can grok them (HTTP2: SETTINGS, HTTP1: TE?)

Next:

Data Types
==========

I think we need these fundamental data types, and subtypes:

1)   Unicode strings

2)	ascii-string (maybe)

3)	binary blob

4)   Token

5)   Qualified-token

6)   Number

7)      integer

8)   Timestamp

In addition to these subtypes, schemas can constrain types
further, for instance integer ranges, string lengths etc.
more on this below.

I will talk about each type in turn, but it goes without saying
that we need to fit them all into RFC723x, in a way that is not
going to break anything important and HPACK should not hate
them either.

In HTTP3+, they should be serialized intelligently, but that
should be trivial and I will not cover that here.

1) Unicode string
-----------------

The first question is do we mean "unrestricted unicode" or do
we want to try to sanitize it.

An example of sanitation is RFC7230's "quoted-string" which bans
control characters except forward horizontal white-space (=TAB).

Another is I-JSON (RFC7493)'s:

   MUST NOT include code points that identify Surrogates or
   Noncharacters as defined by UNICODE.

As far as I can tell, that means that you have to keep a full UNICODE
table handy at all times, and update it whenever additions are made
to unicode.  Not cool IMO.

Imposing a RFC7230 like restriction on unicode gets totally
roccoco:  What does "forward horizontal white-space" mean on
a line which used both left-to-right and right-to-left alphabets ?
What does it mean in alphabets which write vertically ?

Let us absolve the parser from such intimate unicode scholarship
and simply say that the data type "unicode string" is what it says,
and use the schemas to sanitize its individual use.

Encoding unicode strings in HTTP1+2 requires new syntax and
for any number of reasons, I would like to minimize that
and {re-|ab-}use quoted-strings.

RFC7230 does not specify what %80-%FF means in quoted-string, but
hints that it might be ISO8859.

Now we want it to become UTF-8.

My proposal at the workshop, to make the first three characters
inside the quotes a UTF-8 BOM is quite pessimal in HPACK's huffman
encoding:  It takes 68 bits.

Encoding the BOM as '\ufeff' helps but still takes an unreasonable
48 bits in HPACK/huffman.

In both H1 and H2 defining a new "\U" escape seems better.

Since we want to carry unrestricted unicode, we also need escapes
to put the <%20 codepoints back in.  I suggest "\u%%%%" like JSON.

(We should not restict which codepoints may/should use \u%%%% until
we have studied if \u%%%% may HPACK/huffman better than "raw" UTF-8
in asian codepages.)

The heuristic for parsing a quoted-string then becomes:

	1) If the quoted-string first two characters are "\U"
		-> UTF-8

	2)  If the quoted-string contains "\u%%%%" escape anywhere
		-> UTF-8

	3)  If the quoted-string contains only %09-%7E
		-> UTF-8 (actually: ASCII)

	4)  If the quoted-string contains any %7F-%8F
		-> UTF-8

	5)  If header definition explicitly says ISO-8859
		-> ISO8859

	6)  else
		-> UTF-8

2) Ascii strings
----------------

I'm not sure if we need these or if they are even a good idea.

The "pro" argument is if we insist they are also english text
so we have something the entire world stands a chance to understand.

The "contra" arguement is that some people will be upset about that.

If we want them, they're quoted-strings from RFC723x without %7F-%FF.

It is probably better the schema them from unicode strings.

3) Binary blobs
---------------

Fitting binary blobs from crypto into RFC7230 should squeeze into
quoted-string as well, since we cannot put any kinds of markers or
escapes on tokens without breaking things.

Proposal:

	Quoted-string with "\#" as first two chars indicates base64
	encoded binary blob.

I chose "\#" because "#" is not in the base64 set, so if some
nonconforming implementation eliminates the "unnecessary escape"
it will be clearly visible (and likely recoverable) rather than
munge up the content of the base64.

Base64 is chosen because it is the densest well known encoding which
works well with HPACK/huffman:  The b64 characters on average emit
6.46 bits.

I have no idea how these blobs would look when parsed into JSON,
probably as base64 ?  But in languages which can, they should
probably become native byte-strings.

4) Token
--------

As we know it from RFC7230:

   tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
    "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA
   token = 1*tchar

5) Qualified Token
------------------

   qualified_token = token 0*1("/" token)

All keys in all dictionaries are of this type.  (In JSON/python...
the keys are strings)

Schemas can restrict this further.

6 Numbers
---------

These are signed decimal numbers which may have a fraction

In HTTP1+2 we want them always on "%f" format and we want them to
fit in IEEE754 64 bit floating point, which lead to the following
definition:

	0*1"-" DIGIT 0*nDIGIT 0*1("." 0*mDIGIT )	n+m < 15

(15 digits fit in IEEE754 64 binary floating point.)

These numbers can (also) be used for millisecond resolution absolute
UNIX-epoch relative timestamps for all forseeable future.

7) Integers
-----------

	0*1"-" 1*15 DIGIT

Same restriction as above to fit into IEEE 754.

Range can & should be restricted by schemas as necessary.

8 Timestamps
------------

I propose we do these as subtype of Numbers, as UNIX-epoch relative
time.  That is somewhat human-hostile and is leap-second-challenged.

If you know from the schema that a timestamp is coming, the parser
can easily tell the difference between a RFC7231 IMF-fixdate or a
Number-Date.

Without guidance from a schema it becomes inefficient to determine
if it is an IMF-fixdate, since the week day part looks like a token,
but it is not impossible.

Schemas
=======

There needs a "ABNF"-parallel to specify what is mandatory and
allowed for these headers in "common structure".

Ideally this should be in machine-readable format, so that
validation tools and parser-code can be produced without
(too much) human intervation.  I'm tempted to say we should
make the schemas JSON, but then we need to write JSON schemas
for our schemas :-/

Since schemas basically restict what you are allowed to
express, we need to examine and think about what restrictions
we want to be able to impose, before we design the schema.

This is the least thought about part of this document, since
the train is now in Lund:

Unicode strings:
----------------

* Limit by (UTF-8) encoded length.
	Ie: a resource restriction, not a typographical restriction.

* Limit by codepoints
	Example: Allow only "0-9" and "a-f"
	The specification of code-points should be list of codepoint
	ranges.  (Ascii strings could be defined this way)

* Limit by allowed strings
	ie: Allow only "North", "South", "East" and "West"

Tokens
------

* Limit by codepoints
	Example: Allow only "A-Z"

* Limit by length
	Example: Max 7 characters

* Limit by pattern
	Example: "A-Z" "a-z" "-" "0-9" "0-9"
	(use ABNF to specify ?)

* Limit by well known set
	Example: Token must be ISO3166-1 country code
	Example: Token must be in IANA FooBar registry

Qualified Tokens
----------------

* Limit each of the two component tokens as above.
	
Binary Blob
-----------

* Limit by length in bytes
	Example: 128 bytes
	Example: 16-64 or 80 bytes

Number
------

* Limit resolution
	Example: exactly 3 decimal digits

* Limit range
	Example: [2.716 ... 3.1415]

Integer
-------

* Limit range
	Example [0 ... 65535]

Timestamp
---------

(I cant thing of usable restrictions here)

Aaand... I'm in Copenhagen...

Let me know if any of this looks usable...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 01 Aug 2016 10:33:22 +0200

Hi Poul,

(I'm only lurking in this WG because the HTTP protocols are useful for
certain IoT applications; however I do understand the design focus here
is more on browser-web/big-web applications.)

Poul-Henning Kamp wrote:
> (like CORS), 

I'm assuming here you mean CBOR?

> and base64 it into HTTP1-2:  Ie: design for the future
> too complicated, 

(If you are talking about CBOR:)
Well, it is more complicated than doing nothing.

Once a bespoke design of a data model and serialization is completed,
that is likely to be as complicated as CBOR (or even more).

The real problem is then that we have added another data model and
serialization of that data model to the overall complexity that needs to
be managed by a system that connects to the web.
(That may not make a difference for a browser, but it does for IoT and
other machine-to-machine applications.)

The specific data model that you have designed looks fine to me; it
should be representable in CBOR without problem.  What remains is the
cognitive dissonance of having done a base64(url) transformation, but as
you say that is not a real technical problem with HPACK.
(I am aware of the debugging advantages of text encoding, but these are
values that go into HTTP/2 headers and are obscured by HPACK, anyway.)

> and gag-reflexes were triggered by ASN.1.

I do sympathize here :-)

Grüße, Carsten

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 1 Aug 2016 10:57:43 +0200

Hi Poul-Henning,

I've read this a bit quickly but had a thought regarding this :

> 2) Ascii strings

That made me think that most of the header fields I'm seeing do not use
non-ascii characters at all, I'd even say non-printable-ascii. Most of
them contain :
  - host names (Host)
  - uris (Referer, Location)
  - user-agent strings (UA)
  - tokens (Connection, Accept, ...)
  - numbers

Thus in fact I'm wondering if it's really worth focusing the efforts on
non-ascii strings instead. I think we should be able to support them, but
not try to save too many resources on them if this comes at the expense of
the rest of the encoding, or with extra implementation complexity and/or
risks of vulnerabilities (eg: when an unterminated utf-8 sequence precedes
an LF character, I've seen some parsers consume that LF as part of the
character thus not seeing it as a line delimiter).

Just a few thoughts anyway. I'll take more time reading everything in
details, but for now I'm seeing good stuff here overall.

Cheers,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 01 Aug 2016 09:55:50 +0000

--------

>I'm assuming here you mean CBOR?

Sorry yes, not enough tea yet...

>that is likely to be as complicated as CBOR (or even more).

Not if we follow the datamodel I proposed.

>be managed by a system that connects to the web.

Well, the problem is that if we do not add a common datamodel,
each and every new header brings its own.

And by basing the datamodel on the existing HTTP(1) header syntax,
we would not need to base64 encode a binary format in HTTP1

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 01 Aug 2016 09:57:25 +0000

--------
In message <[email-masked]>, Willy Tarreau writes:

>non-ascii strings instead.

My take is that the data-model and serialization should be general
and unconstrained, and the constraints be applied in a/the schema
for each individual header.

---

From: Nicolas Mailhot <[email-masked]>
Date: Mon, 01 Aug 2016 10:41:41 +0000

IMHO it would be way simpler to specify that the dicts used in http are ordered rather than invent another representation

Anyway, please do not use < or > web people have enough tag soup problems in html (that Will be used with http)

If you're ready to invent binary représentations it's way simple to specify utf8 as encoding than fall again on multiple encoding trap which instead of helping anyone means everyone is incompatible with everyone else in subtle way

Finaly , is hostile to everyone that writes numbers unlike the USA
-- 
Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 1 Aug 2016 12:44:05 +0200

> for each individual header.

But we're talking about protocol efficiency as well, which passes via
taking into account what we have. We could for example consider the
notion of "extended strings" which are only used for header fields
which are not relevant to the protocol itself (eg: not used in
accept/range/connection/...) and which would allow unicode to be
safely transmitted. It might be used for user-agent if needed.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 01 Aug 2016 10:50:51 +0000

--------
In message <[email-masked]>, Nicolas Mailhot 
writes:

>IMHO it would be way simpler to specify that the dicts used in
>http are ordered rather than invent another representation

We sort of already did that, only we never formally declared
that they were dicts or what the datamodel actually looked like.

My document was an attempt to do that.

No matter what we decide, we cannot change how JSON defined their
dicts, and consequently whatever we do needs to be mapped into JSON,
python, $lang's data models somehow.

>Anyway, please do not use < or > web people have enough tag
>soup problems in html (that Will be used with http)

They are part of the serialization, like ',' and ';' and they would
not be visible in any context near HTML.

>If you're ready to invent binary representations it's way
> imple to specify utf8 as encoding than fall again on multiple
> encoding trap which instead of helping anyone means everyone is
> incompatible with everyone else in subtle way

Please elaborate, I have no idea what your are talking about here.

>Finaly , is hostile to everyone that writes numbers unlike the USA

We already use ',' as the field delimiter in HTTP headers, and we
should *never* have to take I18N/NLS into account to *parse* a
HTTP header.

I18N/NLS may be necessary to *interpret* the HTTP header, but it
should not be necessary to *parse* the HTTP header.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 01 Aug 2016 11:01:21 +0000

--------

>safely transmitted. It might be used for user-agent if needed.

Unicode can already be safely transmitted as UTF-8, problem is that
people don't know if it is UTF-8 or ISO8859.

The "\U" prefix/escape would solve that efficiently.

---

From: Cory Benfield <[email-masked]>
Date: Mon, 1 Aug 2016 13:30:03 +0100

> On 1 Aug 2016, at 11:50, Poul-Henning Kamp <[email-masked]> wrote:
> 

JSON, sure, but don’t let Python hold you back. All supported versions of Python have an OrderedDict in their standard library. And any Python tool dealing with HTTP has inevitably had to invent something like a CaseInsensitiveOrderedMultiDict in order to deal with HTTP headers, so any tool that’s likely to deal with this kind of thing is already swimming in dictionary representations that we can use for ordering fields in header values.

So just to clarify: the lack of ordering in a JSON object is a reasonable problem with using JSON, but that doesn’t mean we can’t use ordered representations in other serialisation formats. Programming languages have all the abstractions required to do this, and it’s just not that hard to write an Ordered Mapping in $LANG that wraps $LANG’s built-in Mapping type. (Hell, some Python interpreters have *all* dicts ordered, such that they define OrderedDict by simply writing “OrderedDict = dict”).

Cory

---

From: James M Snell <[email-masked]>
Date: Mon, 1 Aug 2016 07:57:26 -0700

phk,

I'm very happy to see the discussion of efficient binary encoding of
HTTP headers coming back around. This is an area that I had explored
fairly extensively early in the process of designing HTTP/2 with the
"Binary-optimized Header Encoding" I-D's (see
https://tools.ietf.org/html/draft-snell-httpbis-bohe-13). While HPACK
won out with regards to being the header compression scheme used for
HTTP/2, there is still quite a bit in the BOHE drafts that could be
useful here.

- James

---

From: Alcides Viamontes E <[email-masked]>
Date: Mon, 1 Aug 2016 16:10:30 +0200

Hi!

TL;DR: I also think that trying to fit HTTP headers in anything else other
than their current representation is a bad idea. But creating a semi-formal
compilation of rules and behaviours for core HTTP headers would be worth
it.

Long rant:

Recently we revamped how ShimmerCat handles HTTP headers and we ended up
creating a separate library with a "Headers Document Object Model". The
bare minimum set of different headers we needed to understand and
manipulate to offer basic functionality is 20, and for each of them we
needed to take the following into account:

       * Representation round-trip: How header ASCII values are parsed to
"things that the program can easily manipulate" (see next), and the other
way around, how to convert to ASCII values. This is slightly different for
HTTP/1.1 and HTTP/2, because of connection specific headers, the "Cookie: "
header, and the rather non-trivial dance with "Host: " and ":authority:".

        * What in-memory representation makes sense for the program: "Date:
" should be a date, "Cookie: "  is  a dictionary, "Set-Cookie: " is a set
indexed by cookie name,  path and perhaps other attributes (exercising the
RFC with  Wordpress teaches you one or two things), "Forwarded: " is
actually a list, "Link: " headers from the point of view of a server doing
HTTP/2 Push are all different beasts each getting their own thing, and so
on. This of course is very program specific and probably not generally
interesting, but it is easier to talk about data structures instead of
ASCII text when defining:

       * How header values combine: there shouldn't be more than one "Date:
" in a given response, even if both a proxy server and an application may
try to stamp a "Date: ". However, a server may "add" cookies to an
application response, and the "Forwarded" header needs to be composed in a
sequence. Similar decisions are needed with CORS headers, Link headers,
Cache-Control, Etag and so on.

        * Headers are extensible, so one needs default policies for header
values where there are no RFC dispositions.

I would find daunting the task of fitting all the idiosyncrasies and
behaviours of HTTP headers in a common bytes representation (serialisation)
without some kind of updated compendium of what they do and how they
behave. For example, it would be nice to have a doc similar to RFC 4229,
with formalised candidate data structures and algorithms for how
intermediaries in different roles should handle the headers. Furthermore,
some HTTP headers are more important/common  than others (is there anybody
using the "From:" header?), or they are relevant to different roles, so
maybe we need to group headers in some sensible way (so that we can say,
e.g. "my CMS emits core http content headers" or "my server is
caching-compliant because it interprets correctly http core caching
headers" or "my server/application implements correctly the security
measures implied by the core security HTTP headers", whatever any of these
can be).

On Mon, Aug 1, 2016 at 2:30 PM, Cory Benfield <[email-masked]> wrote:

> JSON, sure, but don’t let Python hold you back. All supported versions of
> Python have an OrderedDict in their standard library. And any Python tool
> dealing with HTTP has inevitably had to invent something like a
> CaseInsensitiveOrderedMultiDict in order to deal with HTTP headers, so any
> tool that’s likely to deal with this kind of thing is already swimming in
> dictionary representations that we can use for ordering fields in header
> values.
>
> So just to clarify: the lack of ordering in a JSON object is a reasonable
> problem with using JSON, but that doesn’t mean we can’t use ordered
> representations in other serialisation formats. Programming languages have
> all the abstractions required to do this, and it’s just not that hard to
> write an Ordered Mapping in $LANG that wraps $LANG’s built-in Mapping type.
> (Hell, some Python interpreters have *all* dicts ordered, such that they
> define OrderedDict by simply writing “OrderedDict = dict”).
>
> Cory
>

./Alcides

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 2 Aug 2016 13:33:39 +0200

Hey PHK,

Sorry for the delay, been in (and still remain in) transit hell (hi from FRA!).

Overall I like this.

A few thoughts come to mind:

1) Using the first character of the field-value as a signal that the encoding is in use is interesting. I was thinking of indicating it with a suffix on the header field name (e.g., Date-J). Either is viable, but I don't think it's a good idea to reuse existing header field names and rely on that signal to differentiate the value type; that seems like it would cause a lot of interop problems to me. Defining a new header field (whether it's Date-J or Date2 or whatever) seems much safer to me.

2) Regardless of #1, using < as your indicator character is going to collide with the existing syntax of the Link header.

3) I really, really wonder whether we need recursion beyond one level; e.g., I can see a list of dicts, or a dict of dicts, but beyond that seems like a lot of complexity to support. Fields like Accept that have complex structure turn out not to be implemented (qvalues are commonly ignored); having an ordered list would work much better (and defining new header fields as per #1 means we have an opportunity to do this!).

4) I agree with the sentiment that non-ascii strings in header field values are comparatively rare (since most headers are not intended for display), so while we should accommodate them, they shouldn't be the default.

5) I like the idea of 'implicit angle brackets' to retrofit some existing headers. Depending on the parse algorithm we define, we could potentially fit a fair number of existing headers into this, although deriving the specific data types of things like parameter arguments is going to be difficult (or maybe impossible). Needs some investigation before we know whether this would be viable.

Cheers,

> On 1 Aug 2016, at 9:43 AM, Poul-Henning Kamp <[email-masked]> wrote:
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 2 Aug 2016 13:53:55 +0200

Hi Mark,

On Tue, Aug 02, 2016 at 01:33:39PM +0200, Mark Nottingham wrote:
> 1) Using the first character of the field-value as a signal that the encoding
> is in use is interesting. I was thinking of indicating it with a suffix on
> the header field name (e.g., Date-J). Either is viable, but I don't think
> it's a good idea to reuse existing header field names and rely on that signal
> to differentiate the value type; that seems like it would cause a lot of
> interop problems to me. Defining a new header field (whether it's Date-J or
> Date2 or whatever) seems much safer to me.

I had the same feeling initially but I retracted. I fear that having two
header fields will result in inconsistencies between the two (possibly
intentional when that may be used to benefit an attacker). We'd rather
avoid reproducing the Proxy-Connection vs Connection mess we've been seeing
for a decade, where both were sent "just in case".

However if we enumerate certain header fields that would deserve being
encoded differently and find a way to group them, we may think about
sending a composite, compact header field for transport/routing, another
one for the entity where available information are grouped when relevant.
Then maybe it could be decided that when one agent consumes such a field,
before passing the message it must delete occurences of the other ones,
and/or rebuild them from the composite one, in order to avoid inconsistency
issues.

We have more or less this regarding Transfer-Encoding which voids
Content-Length, and the Host header field which must always match the
authority part of the URI if present.

These are just thoughts, maybe they are stupid.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 02 Aug 2016 12:01:50 +0000

--------
In message <[email-masked]>, Mark Nottingham wri
tes:

>1) Using the first character of the field-value as a signal that the 
>encoding is in use is interesting. I was thinking of indicating it with 
>a suffix on the header field name (e.g., Date-J).

Yeah, that could work too, but I suspect it would be more cumbersome
to implement, and it creates a new class of mistakes which need to
be detected  - "Both Date and Date-J ??"
 
>2) Regardless of #1, using < as your indicator character is going to 
>collide with the existing syntax of the Link header.

If Link is "<> blacklisted" in the IANA registry, that wouldn't be a
problem, and all currently defined headers will need to be checked
against some kind of white/black list, if we want them to use the
new "common structure".

I picked <> because they were a cheap balanced pair in HPACK/huffman
and I only found Link that might cause a false positive.

Strictly speaking, it doesn't have to be a balanced pair, it could
even be control-characters but HPACK/huffman punish those hard.

I didn't dare pick () even though it had even shorter HPACK/huffman.

Thinking about it now, I can't recall any headers starting with a '('
so () might be better than <> and thus avoid the special case of Link.

>3) I really, really wonder whether we need recursion beyond one level; 

As do I.

However, if it is recursion, the implementation cost is very low,
and I would prefer to "deliver tools, not policy" and let people
recurse until they hurt if they want.

I particular do not want to impose complexity limits on private
headers, based on the simplicity of public headers, because my
experience is that private headers are more complex.

I would prefer a simple, general model, restricted by machine
readable schemas, rather than a complex model with built in
limitations.

>4) I agree with the sentiment that non-ascii strings in header field 
>values are comparatively rare (since most headers are not intended for 
>display), so while we should accommodate them, they shouldn't be the 
>default.

That was the idea behind: \U  Make people explicitly tag UTF8

>5) I like the idea of 'implicit angle brackets' to retrofit some 
>existing headers. Depending on the parse algorithm we define, we could 
>potentially fit a fair number of existing headers into this, although 
>deriving the specific data types of things like parameter arguments is 
>going to be difficult (or maybe impossible). Needs some investigation 
>before we know whether this would be viable.

Schemas!  Have I mentioned already how smart I think schemas usable
to build code with would be ?  :-)

PS: I had expected you to ask if was trying to sabotage your Key header :-)

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 02 Aug 2016 12:05:37 +0000

--------

>These are just thoughts, maybe they are stupid.

Nope, that's actually a very interesting idea.

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 2 Aug 2016 14:23:50 +0200

Please never, ever introduce "Date" and "Date-J" or something like that.

> Am 02.08.2016 um 14:05 schrieb Poul-Henning Kamp <[email-masked]>:
> 
> --------

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 02 Aug 2016 12:38:27 +0000

--------
In message <[email-masked]>, Stefan Eissing

>Please never, ever introduce "Date" and "Date-J" or something like that.

That was exactly _not_ what Willy suggested :-)

---

From: [email-masked]
Date: Tue, 2 Aug 2016 14:40:30 +0200 (CEST)

----- Mail original -----
De: "Poul-Henning Kamp"

>Please elaborate, I have no idea what your are talking about here.

I thouroughly mislike all the encoding dance and escaping one time it's UTF-8 another not.

It's too complex, people won't read it, just do their usual mess and assume it will work (just like they post garbage nowadays and add ISO-8859-1 boilerplate without checking anything. Encoding hints do not help, instead of wrong encoding failure you add combos of bad encoding hint + whatever failures)

Please use a simpler rule like
 1. Everything not explicitely encoded is UTF-8 with no escaping allowed (or escaping restricted to a shortlist, not generic escaping that people will apply to all codepoints several times over just in case, no BOM, no pseudo-BOM, if they write nonsensical UTF-8 it's *their* problem not the transport problem, as long as they respect basic UTF-8 rules)
 2. an UTF-8 header that does not validate in whatever version of the next hop unicode engine aborts the frame (if you don't want to learn UTF-8 restrict yourself to basic latin block in your headers, it's valid UTF-8 and no more complex than ASCII)
 3. anything that does not fit there must use binary with explicit binary flag

That is simple enough people will remember it, they will curse you but apply it, instead of fell-good rules that everyone follows approximatively with an explosion of special cases to handle all the approximation results

Regards,

-- 
Nicolas Mailhot

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 2 Aug 2016 14:41:19 +0200

> On 2 Aug 2016, at 1:53 PM, Willy Tarreau <[email-masked]> wrote:
> 
> Hi Mark,
> 

I know, I don't like it either. I'm just concerned that if we keep the name the same, it's much more likely it's going to not be properly converted, and that could enable attacks too.

Stepping back, I think we're talking about a set of rules something like this;

A. For a newly defined header field that explicitly uses the new format, send it in the new format
B. For existing header fields, if their expression in the new format is defined:
  1. If you have evidence that your peer can accept the new header format, send them in the new format
  2. Otherwise, send them in the original format.
C. All other fields are always sent in the original, HTTP/1 format.

I.e., having both versions of a single header's semantics the wire at the same time is an error.

This means that the format of those headers is effectively a hop-by-hop attribute; you might have a situation where a non-format-aware node forces the hops surrounding it back to the original format (for headers with two different ways to express those semantics).

This gives me pause. Converting from new to old and back to new is very likely to tickle a lot of bugs and cause a lot of interop problems. So, we could say that conversion only happens as a downgrade; i.e., if the next hop doesn't support the encoding, you can downgrade, but you never upgrade it again to the new encoding.

Presumably, the last "hop" might be inside the origin server, when it converts those header fields into the old format for backwards compatibility with existing applications that aren't aware of the new format.

Applications that *are* aware of the new format will still need to handle the original format, because there will be clients / hops generating it for the foreseeable future. 

This kind of seems like a mess to me, and leads me to think that the only time we should attempt this is during a major protocol revision (i.e., h3), and even then, with great trepidation.

If we get that far, deciding how to signal which headers are encoded seems more manageable :)

> These are just thoughts, maybe they are stupid.

Not stupid at all, but I am concerned about adding too much "magic"; if implementations are doing too much on your behalf, issues will arise (see above).

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 2 Aug 2016 14:58:55 +0200

Unfortunately, implementation of the Connection header was really, really bad. :-/

> On 2 Aug 2016, at 2:55 PM, Willy Tarreau <[email-masked]> wrote:
> 
> On Tue, Aug 02, 2016 at 02:41:19PM +0200, Mark Nottingham wrote:
>> Not stupid at all, but I am concerned about adding too much "magic"; if
>> implementations are doing too much on your behalf, issues will arise (see
>> above).
> 
> You probably know that I hate magic as well, that's why I prefer to rely
> on what we have. For example, passing "connection:" with the new headers
> to optimize their eviction along non-compatible paths is doable. It's not
> 100% safe but doable. Ensuring that compatible actors replace the old
> version is doable as well because it would be a "MUST" in the spec and we
> know these actors don't exist yet. So all in all we can possibly do useful
> things. I just don't want to have a tens of headers being advertised in
> Connection nor having to add many extra headers for the sake of saving
> space and parsing time, because we know that it will add extra work that
> may sometimes offset the savings. Hence the idea to compact what can be
> compacted if we went down that route.
> 
> Cheers,
> Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 02 Aug 2016 13:06:52 +0000

--------
In message <[email-masked]>, Mark Nottingham wri

>C. All other fields are always sent in the original, HTTP/1 format.

Yes, something like that.

However, this is digging deep into the extreme far end of the ideas
I presented, and it seems rather too speculative to dig much further.

I am far more interested in hearing what people think about the
important part of the write-up:  Deriving and generalizing a
common header structure from existing HTTP1 header syntaxes ?

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 2 Aug 2016 15:08:37 +0200

> On 2 Aug 2016, at 3:06 PM, Poul-Henning Kamp <[email-masked]> wrote:
> 
> --------

I think that's the way to go, and that using it for *newly defined* headers is a good enough use case.

If we can pause discussion of porting existing headers to the new format for the time being, it may help.

Cheers,

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 02 Aug 2016 13:10:29 +0000

--------
In message <[email-masked]>, Mark Nottingham wri
tes:

>So it's really "implicit angle brackets plus a reference to a 
>retrofitted schema". OK. 

yes.

>Get on another train and start working on that schema language. :)

I should have seen that one coming :-)

>That's one of the reasons I complained about arbitrary recursion.
>
>However, whatever happens here, I think we have to accept that Key will 
>not be able to address all header fields; it's always going to be a 
>subset. If a particular header field wants to leverage Key, it'll need 
>to be specified within its capabilities (provided it gets traction, of 
>course).

The interesting angle is that if we have such a common structure, the
Key header could use it to describe exactly what you want to pull out
of a header, not as a substring, but as correctly parsed data elements.

---

From: [email-masked]
Date: Tue, 2 Aug 2016 15:09:07 +0200 (CEST)

----- Mail original -----
De: "Poul-Henning Kamp" <[email-masked]>

>I think you are barking up another tree than I am.

>I'm trying to find a way to lay down a common structure so people
>dont *have* to break things to get them to work as they do now.

I appreciate the intent (really) but I feel the result of tip-toeing around unicode rules just generates a lot of complexity, makes the general case complex and special cases simple, when it should be the reverse (general case → just UTF-8 with no additional rules, special cases → just use binary)

There are 7 (!) text encoding cases in your proposal

-- 
Nicolas Mailhot

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 2 Aug 2016 15:11:03 +0200

> is a good enough use case.

Same here.

> the time being, it may help.

Very likely but we all know it's very tempting at the same time. They
also serve as good examples of what we may need in the future.

Willy

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 2 Aug 2016 14:40:01 +0200

Just stamping it as a thought-crime in advance...

> Am 02.08.2016 um 14:38 schrieb Poul-Henning Kamp <[email-masked]>:
> 
> --------

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 2 Aug 2016 14:48:34 +0200

> On 2 Aug 2016, at 2:01 PM, Poul-Henning Kamp <[email-masked]> wrote:
> 
> --------

See separate discussion. 

> so () might be better than <> and thus avoid the special case of Link.

That might work.

> readable schemas, rather than a complex model with built in

Nod.

Right now, most implementations limit header field values on length; if they start limiting on recursion as well, it would be good to have a reasonable value specified (or at least suggested).

> That was the idea behind: \U  Make people explicitly tag UTF8

Ok.

> to build code with would be ?  :-)

So it's really "implicit angle brackets plus a reference to a retrofitted schema". OK. 

However, whatever happens here, I think we have to accept that Key will not be able to address all header fields; it's always going to be a subset. If a particular header field wants to leverage Key, it'll need to be specified within its capabilities (provided it gets traction, of course).

Cheers,

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 02 Aug 2016 12:44:15 +0000

--------
In message <[email-masked]>, nicol

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 02 Aug 2016 15:52:05 +0200

I don't know when Poul-Henning will catch a train the next time...

But I was interested in whether CDDL can be used as a specification
language here, because it is always good to look at more use cases.
So let's see (mostly untested at this point because I have no idea
whether anybody wants to use CDDL in this context, but examples should

> allowed for these headers in "common structure".

Indeed, CDDL is essentially ABNF ported to tree grammars.

The top-level data model of the proposed format could be expressed as:

header-value = [* dict-element]
dict-element = [name, value-map]
name = text                          ; possibly restricted
value-map = {* name => value}        ; empty by default
value = text / bytes / number / time / value-map
                                     ; add as needed

> make the schemas JSON, but then we need to write JSON schemas

For -09, we are discussing to add a separate machine-readable (JSON)
encoding to be used by tools, in addition to the human-readable format
to be used by humans.  (No intention to make both the same, that would
be a classical mistake.)

> the train is now in Lund:

OK, let's see what restrictions CDDL offers today (they are called

> Unicode strings:

That would be .size:

dns-label = text .size (1..63)

> 	ranges.  (Ascii strings could be defined this way)

Today this is generally done via regexps.

> 	ie: Allow only "North", "South", "East" and "West"

Those are typically done constructively:

direction = "North" / "South" / "East" / "West"

Of course, regexps can do that, too, if needed.

> 	Example: Allow only "A-Z"

token1 = text .regexp "[A-Z]"

> 	Example: Max 7 characters

CDDL can only count characters (as opposed to bytes) employing regexps
right now.
Another extension may be needed if there indeed is a good use case for
counting characters.

> 	Example: "A-Z" "a-z" "-" "0-9" "0-9"

(Regexps, again)

> 	Example: Token must be in IANA FooBar registry

Interesting.  There currently is no formal interface to IANA (or ISO)
registries; we don't have an informal escape like ABNF has in prose-val.
"Annotations" could be added as needed and they are close.

> Qualified Tokens

blob1 = bytes .size (16..64 / 80)

> 	Example: exactly 3 decimal digits

Would need a new CDDL "annotation", say

q = number .decimals 3

> 	Example: [2.716 ... 3.1415]

etopi = 2.718..3.1415

> 	Example [0 ... 65535]

ex1 = 0..65535
ex2 = uint .size 2

> (I cant thing of usable restrictions here)

ts = uint   ; (or whatever type of timestamp you want;
            ;  `time` as a POSIX time and RFC3339 dates are built in)

Grüße, Carsten

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 2 Aug 2016 14:55:41 +0200

Cheers,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 02 Aug 2016 18:08:06 +0000

--------
In message <[email-masked]>, Carsten Bormann writes:

>language here,

I'm guess it is this draft ?

   https://datatracker.ietf.org/doc/draft-greevenbosch-appsawg-cbor-cddl/

>to be used by humans.  (No intention to make both the same, that would

I don't have an informed opinion about CDDL at this point, but
having two parallel specifications sounds like a clas^H^H^H^Hantiquity
mistake to me.

How do you plan to make sure they both say the same thing ?

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 02 Aug 2016 21:46:43 +0200

> How do you plan to make sure they both say the same thing ?

You generate the JSON one from the original CDDL source when you need it.
(The JSON version is for interchange between tools working on instances
of the specification language, not for humans to work on it.  Of course
it is not hard to write a CDDL parser, but it is even easier to ingest
JSON from an existing CDDL parser, and standardizing this intermediate
format together with the language sounds like a good idea and is simple
enough to do.)

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 2 Aug 2016 22:08:33 +0200

If containers are only allowed to contain simple types, the need for a schema language diminishes quite a bit; headers can be defined pretty easily in prose, perhaps with references to registries where appropriate.

Personally, the discussion below makes me think that's a good thing...

> On 2 Aug 2016, at 9:46 PM, Carsten Bormann <[email-masked]> wrote:
> 
> Poul-Henning Kamp wrote:

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 02 Aug 2016 20:24:54 +0000

--------
In message <[email-masked]>, Mark Nottingham wri
tes:

>If containers are only allowed to contain simple types, the need for a
>schema language diminishes quite a bit; headers can be defined pretty
>easily in prose, perhaps with references to registries where
>appropriate.

It is not significantly harder to specify recursive structures than
flat structures, but of course the work to do so will make many
people want not to.

As for the praise for prose, yes, it is amazing what you can do if
you are Hemmingway, Prachett, Jefferson or Adams, but most of us
are not.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 02 Aug 2016 23:24:43 +0200

> If containers are only allowed to contain simple types, the need for a schema language diminishes quite a bit; headers can be defined pretty easily in prose, perhaps with references to registries w…

The same observation could be made of textual information and ABNF.
In the last 40 years, the industry has converged on using BNF in
specifications of languages as a matter of routine for a number of good
reasons; these become even stronger with a standard notation such as
ABNF which allows throwing some tooling into the specification process
without losing the benefit of a human-readable notation.
Yes, I have run into people who see little value in adding ABNF to
specifications; the benefit may not be universal for all users of a
specification.

I expect the same kind of benefits from using CDDL for JSON-like data
models.  (We had a similar discussion for COSE, which is very simple as
a data model, too*); still the CDDL snippets do help many spec readers,
and the tooling also does help in validating the examples against the spec.)

Grüße, Carsten

*) Well, OK, it *does* have recursion in the recipients in the
encryption spec.

(Pet peeve: "Schema languages" fundamentally come from a database world
and try to solve a different problem, even though they may overlap with
tree grammars; but the usage has become sufficiently confused in the
last 20 years that I won't insist on this difference.  But I won't get
caught calling CDDL a "schema language"...)

---

From: Martin Thomson <[email-masked]>
Date: Wed, 3 Aug 2016 10:23:23 +1000

> so () might be better than <> and thus avoid the special case of Link.

Or ><?

---

From: Kari hurtta <[email-masked]>
Date: Wed, 3 Aug 2016 09:37:30 +0300 (EEST)

| 1) Using the first character of the field-value as a signal that the encoding is in use is interesting. I was 
| thinking of indicating it with a suffix on the header field name (e.g., Date-J). Either is viable, but I 
| don't think it's a good idea to reuse existing header field names and rely on that signal to differentiate 
| the value type; that seems like it would cause a lot of interop problems to me. Defining a new header field 
| (whether it's Date-J or Date2 or whatever) seems much safer to me.
| 
| 2) Regardless of #1, using < as your indicator character is going to collide with the existing syntax of the 
| Link header.

Or perhaps use ':' as indicator? Causes double '::' on HTTP/1

Date::1470205476

Is this viable ?

/ Kari Hurtta

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Aug 2016 08:46:33 +0200

> Or perhaps use ':' as indicator? Causes double '::' on HTTP/1

It could but strictly speaking it will not be "::", it would just be ":"
to start the value, because your field above parses as ":1470205476" for
the value and will be rewritten like this along the path by many
implementations :

    Date: :1470205476

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Aug 2016 06:52:45 +0000

--------
>> so () might be better than <> and thus avoid the special case of Link.

Ohh ... neat thinking...

---

From: Kari hurtta <[email-masked]>
Date: Wed, 3 Aug 2016 12:37:51 +0300 (EEST)

Willy Tarreau <[email-masked]>: (Wed Aug  3 09:46:33 2016)
> the value and will be rewritten like this along the path by many

Yes, that is true.  

Another possibility is use indicator from ascii control
block. For example byte 01. Does not collide with existing use.

That is
Date:1470205476
on HTTP/1 (but probaly not visible; first character
after ':' is byte 01 (ctrl-A)).

( I think that someone suggested that already. )

( byte 01 seems to be valid TEXT on ABNF ? )

/ Kari Hurtta

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Aug 2016 11:45:54 +0200

> after ':' is byte 01 (ctrl-A)).

It doesn't change anything. I have no problem with either proposals,
it's just that what you need to understand is that there is no such
"after ':'". The ":" is not part of the header field, it's the field
name delimitor. So before ':' you have the header field name. After
it you can have any amount of spaces (including zero) which are *not*
part of the value, and the first non-space character starts the value.

Thus, the following are exactly equivalent, though the last one is
deprecated :

    Date:1470205476
    Date: 1470205476
    Date:      1470205476
    Date:
      1470205476

> ( byte 01 seems to be valid TEXT on ABNF ? )

Such bytes are rare and will have a large huffman encoding in H2. Martin's
suggestion of '><' could be more efficient, though I haven't checked.

Regards,
Willy

---

From: Martin Thomson <[email-masked]>
Date: Wed, 3 Aug 2016 14:50:01 +0200

> suggestion of '><' could be more efficient, though I haven't checked.

I tend to think that we should not let hpack drive this.  We should
maybe avoid Huffman encoding and then throw octets into the value
field.  Starting with a > or : or other octet is still probably useful
and might even be necessary.  In HTTP/1.1 we can use base64(url) as a
reasonable space/speed trade-off, again with the same demarc octet.
But we'd be defining a binary encoding.

Binary avoids the nastiness with character encoding (just use UTF-8),
makes numbers and dates much more numbery, and lets us tailor the
other types to our needs.

I think that PHK is perfectly right in recognizing that we don't have
complex needs.  I actually think that this is good.  Limitations are
empowering.

I said this privately to someone at the workshop, but my realization
was that we currently have schema-aware parsing with extremely limited
points of extension.  A revised system that supports that doesn't need
to be very complex.  Even a single level map of string key to
(optional) string value is more extensibility than we can sensibly
defend.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Aug 2016 13:07:20 +0000

--------

>I tend to think that we should not let hpack drive this.

Agreed to that, but if we can easily be huffman-friendly, we should.

>other types to our needs.

What I'm trying to do here and now, is a data model and HTTP/1
serialization which by design overlaps as many existing headers
defined syntax as possible, to minimize the number of parsers
required now and in the future.

By leeching on existing H1 syntax we avoid the b64 hack and we don't
need to touch HPACK before this can go live.

For H3+'s HPACK-ism, we may decide to serialize these structured
headers as "native binary" rather than "text-compression", but until
we know more about H3, that is a premature decision.

The downside of this approach is that the RFC723x's "quoted string"
also carries "utf8-string" and "binary-blob" in this new syntax,
signposted by two new, backwards compatible escape sequences.

>complex needs.  I actually think that this is good.  Limitations are

Indeed.

>(optional) string value is more extensibility than we can sensibly

As with HPACK/huffman, I don't see the need to heurstically restict
the data model, I would rather precisely restrict its use for
individual headers.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 3 Aug 2016 15:09:39 +0200

> On 3 Aug 2016, at 3:07 PM, Poul-Henning Kamp <[email-masked]> wrote:
> 

Keep in mind that we're not going to require new headers to use this thing; if their needs aren't met, they can do something else.

Looking at recently minted headers, I'm still sharing Martin's inclination to keep it simple and flat.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Aug 2016 13:23:58 +0000

--------
In message <[email-masked]>, Mark Nottingham writes:

>Keep in mind that we're not going to require new headers to use this 
>thing; if their needs aren't met, they can do something else.
>

I fully agree, big fan of KISS here.

I just prefer the limitations be located in the per-header schema,
rather than in the data format itself.

Either way, as long as we paint ourselves into a corner which makes
it impossible to have more levels later, I'm fine with only
"releasing" one level initially.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 3 Aug 2016 15:42:01 +0200

> required now and in the future.

I'm skeptical that you will be able to do that without sacrificing
something.  And if the point of the exercise is to define the one true
format (or three or some small number) that is used hereafter, then I
don't see much inherent value in minimizing the distance between the
old thing and the new thing.  I'd rather sacrifice similarity than
lose (for example) decoding efficiency, or UTF-8, or any of the many
things that have been dreamed up.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Aug 2016 14:18:07 +0000

--------
In message <[email-masked]>, Martin Thomson writes:
>something.

There are always tradeoffs.

For instance the HTTP/1 serialization should not be a security risk
if it pops up in a traditional application which does naïve
stringprocessing on HTTP headers.

It would also be nice if HTTP/1 can still be debugged by eye, without
needing b64 decoding, but that is much less important than not
blowing up all PHP or JS programs.

>And if the point of the exercise is to define the one true
>format (or three or some small number) that is used hereafter,

It doesn't have to be perfect, I'll be happy if we can come up with
a common structure which is so usable, that the exceptions become
rare and well considered.

>I don't see much inherent value in minimizing the distance between the
>old thing and the new thing.

Like the HPACK/huffman thing:  I fully agree.  But if the minimal
distance does not hurt us going forward, we would be silly to
increase it needlessly.

And please remember:  I deliberately approached this from the far
opposite side relative to adopting JSON, in order to another data
point on the size of the solution space.

I liked the outcome, it was surprisingly much better than I had
expected, but I would be very surprised if there isn't a better
solution once we ponder it more.

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 3 Aug 2016 19:07:11 +0900

> flat structures, but of course the work to do so will make many

Also, I'd be afraid of the first time there is a real use case that is 
recursive or has more levels than planned for; the separate syntax and 
implementation will be ugly, or people might just go recursive the 
obvious way but implementations will vary on how they take it.

Regards,   Martin.

---

From: Sam Johnston <[email-masked]>
Date: Thu, 18 Aug 2016 14:43:19 +0100

Shame to have missed this discussion as it starts to look like one we had a
few years ago around using headers directly rather than trying to embed
another envelope format in them (ala SOAP in the body):

https://lists.w3.org/Archives/Public/ietf-http-wg/2011OctDec/0155.html

I actually wrote drafts for "Category" and "Attribute" headers at the time;
the latter is yet to see the light of day but the former is here:
https://tools.ietf.org/html/draft-johnston-http-category-header-02

My view is that you should be able to have e.g. a photo with headers
containing attributes like title & summary, categories like landscape, and
links to e.g. author (which Mark has already standardised in RFC5988 Web
Linking).

We got caught up with things like unicode, client library support, etc. at
the time, but I expect some of these things have been resolved in the
interim.

Sam