Subject: JSON headers

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 08 Jul 2016 17:49:09 +0000

Since Marks call for adoption of this draft, I've been thinking a lot about the
general topic.

There is no doubt that the current way we've been doing HTTP headers has turned
into a bit of a boondongle.

Switching to a consistent structured format is certainly a good idea, and as
such formats go, JSON is far from the worst one could pick.

So, yes, in principle I'm all for this.

However, and you knew that would be coming, I have three major issues with this.

Future binary format
--------------------

I really hope we are going to discontinue the text-processing
approach to the HTTP protocol headers required for transport purposes.

Sending numbers, and particular dates as ascii strings is simply
not sensible with speeds north of 10 Gb/sec.

If we decide to standardise on _a_ structured format, we should pick
one that has a high-performance-sensible binary parallel, which
can be used in HTTP/2 and onwards

I know about BSON, BJSON and CBOR, there are probably other as well,
we need to study them carefully so we do not paint ourselves into
a corner here.  The binary versions limitations/encodings in a high
performance setting may affect how we use/constrain usage of the
textual version.

How do we specify this in RFCs
------------------------------

This is the BIG one.

Julians draft does not addrss this at all:  We need a "ABNF" for
specifying the structured syntax of standard headers.

Before we open the floodgates for JSON (or whatever) headers,
we absolutely have to have found and nailed down how they will
be documented/specified in RFCs.

This syntax specification has to be both human- and machine-readable,
so known-to-be-compliant efficient code can be generated directly
from the RFC text.

There's something called "JSON-schema"

	http://json-schema.org/example1.html

It doesn't look apetizing to me, see for instance their
JSON-schema for JSON-schema-JSONs:

	http://json-schema.org/schema

Being able to piggy-back on something like that would be a big
plus over rolling our own.

Simplify semantics
------------------

I realize that Julians draft specifically targets newly defined
headers, and that is a good starting point.

However, we should leverage this to re/bis-define standard headers
in the new format too so they can have their semantics reduced and
simplified with a venegance.

To take Accept-encoding as an example:

It should be constrained to a simple list in order of preference,
with identity being the implicit last element.  Not on the list?
not accepted.

Forget the bloody useless q= values, forget sending it in any order
you happen to like, forget sending multiple headers.

	Accept-Encoding: [ "gzip", "deflate" ]

There, done.

Do we even need both gzip and deflate ?  No, of course we do not.

Other more competent compressions ?  Yes by all means, but the same
compression with/without some trivial header-bytes is just a stupid
waste of everybodys time and code.

Even better:

	Accept-Encoding: [ "gzip" ]

Now lets go one step further:  Most implementations today support
gzip, so the above should be the default if no Accept-Encoding
header is present.  If you do not support gzip, you'll have to
send:

	Accept-Encoding: [ ]

Everybody else can avoid sending Accept-Encoding entirely.

We can repurpose the minor number of HTTP protocol numbers
to indicate the sematic version:

	HTTP/1.0 -> ascii headers we know and have a complex 
		    and unfulfilling relationship with
	HTTP/1.1 -> the bugfix release

	HTTP/2.0 -> HPACK'ed ascii headers
	HTTP/2.1 -> future bugfix release

	HTTP/1.2 -> Ascii JSON headers
	HTTP/2.2 -> Binary JSON headers

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Phil Hunt <[email-masked]>
Date: Fri, 8 Jul 2016 11:44:05 -0700

Not sure if this has been discussed. One of the biggest problems with HTTP request signing has been repeat headers. It presents problem of detecting which headers are intended and which header was signed first.

It would be nice if the JSON encoding handled arrays so that the demand for duplicate headers is removed.  Signing could then be more successful and could even stipulate that the presence of a repeat header in a signed request is a failure condition.

Phil

@independentid
www.independentid.com <http://www.independentid.com/>[email-masked] <mailto:[email-masked]>

> On Jul 8, 2016, at 10:49 AM, Poul-Henning Kamp <[email-masked]> wrote:
>

---

From: Mike Bishop <[email-masked]>
Date: Fri, 8 Jul 2016 21:29:21 +0000

Poul, let me attempt to recast what you're saying:  Structured headers good, binary structured headers better, and let's majorly rev the protocol to make sensible use of them everywhere.  I'm on the same page with you of liking structure, but needing some ABNF++ that can express a valid header format.  I also lament that we lost the binary header support during HPACK development.

Retconning the minor version number as a semantic version number is... intriguing.  The conclusion I reached while writing my draft on the semantic vs. mapping layer was fundamentally that we didn't change the semantic layer in HTTP/2, and consequently don't really have a versioning scheme for it right now.  Thanks for proposing one.  Unfortunately, it fundamentally breaks the promise we just restated in RFC 7230:  "The interpretation of a header field does not change between minor versions of the same major HTTP version, though the default behavior of a recipient in the absence of such a field can change."  I don't really want to go down that painful interoperability path.

You have, both here and in the past, described a number of potential changes for when/if we decide the rev the semantic layer.  That's a really interesting direction, and one I'm hoping we get to explore in the Workshop.  It's also currently outside this WG's charter and well beyond the question of whether we should adopt this draft.

Personally, I'll join in the support for structured header fields for the future.  Whatever it is, it should be easy to specify, easy to validate, and ideally human-readable.  I do support the adoption of a draft to move future headers in that direction.  I'm not convinced JSON is the right format for it, mostly for layering's sake, but I don't have a better candidate and we'll probably need a dedicated parser for whatever we end up with anyway.

-----Original Message-----
From: Poul-Henning Kamp [mailto:[email-masked]] 
Sent: Friday, July 8, 2016 10:49 AM
To: HTTP Working Group <[email-masked]>
Subject: JSON headers

Since Marks call for adoption of this draft, I've been thinking a lot about the general topic.

There is no doubt that the current way we've been doing HTTP headers has turned into a bit of a boondongle.

Switching to a consistent structured format is certainly a good idea, and as such formats go, JSON is far from the worst one could pick.

Future binary format
--------------------

I really hope we are going to discontinue the text-processing approach to the HTTP protocol headers required for transport purposes.

Sending numbers, and particular dates as ascii strings is simply not sensible with speeds north of 10 Gb/sec.

If we decide to standardise on _a_ structured format, we should pick one that has a high-performance-sensible binary parallel, which can be used in HTTP/2 and onwards

I know about BSON, BJSON and CBOR, there are probably other as well, we need to study them carefully so we do not paint ourselves into a corner here.  The binary versions limitations/encodings in a high performance setting may affect how we use/constrain usage of the textual version.

This is the BIG one.

Julians draft does not addrss this at all:  We need a "ABNF" for specifying the structured syntax of standard headers.

Before we open the floodgates for JSON (or whatever) headers, we absolutely have to have found and nailed down how they will be documented/specified in RFCs.

This syntax specification has to be both human- and machine-readable, so known-to-be-compliant efficient code can be generated directly from the RFC text.

It doesn't look apetizing to me, see for instance their JSON-schema for JSON-schema-JSONs:

Being able to piggy-back on something like that would be a big plus over rolling our own.

Simplify semantics
------------------

I realize that Julians draft specifically targets newly defined headers, and that is a good starting point.

However, we should leverage this to re/bis-define standard headers in the new format too so they can have their semantics reduced and simplified with a venegance.

It should be constrained to a simple list in order of preference, with identity being the implicit last element.  Not on the list?
not accepted.

Forget the bloody useless q= values, forget sending it in any order you happen to like, forget sending multiple headers.

There, done.

Other more competent compressions ?  Yes by all means, but the same compression with/without some trivial header-bytes is just a stupid waste of everybodys time and code.

Even better:

Now lets go one step further:  Most implementations today support gzip, so the above should be the default if no Accept-Encoding header is present.  If you do not support gzip, you'll have to
send:

	Accept-Encoding: [ ]

We can repurpose the minor number of HTTP protocol numbers to indicate the sematic version:

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 08 Jul 2016 21:58:26 +0000

--------
In message <[email-masked].
outlook.com>, Mike Bishop writes:

>The conclusion I reached while writing my draft on the semantic vs. mapping
>layer was fundamentally that we didn't change the semantic layer in HTTP/2,
>and consequently don't really have a versioning scheme for it right now.
>Thanks for proposing one.  Unfortunately, it fundamentally breaks the promise
>we just restated in RFC 7230:  "The interpretation of a header field does not
>change between minor versions of the same major HTTP version [...]

I am aware of that.

I am also aware that the major problems people have with HTTP are
not the transport layer, although that causes plenty of grief, but
the semantic layer we do not have.

Some years ago I realized that there is a finite amount of code to
be backwards-compatible with, and that this amount of code by its
nature will become smaller and smaller with passing time.

Looking the other way, there is a potentially infinite amount of
code which has not yet been written, and there will be more and
more of it as time goes by.

The obvious conclusion is that being compatible with the future is
far more important than being compatible with the past.

I am therefore perfectly willing to write in a RFC:

   "That promise in RFC7230?  We changed our mind"

If that can make the future a better place...

---

From: Cory Benfield <[email-masked]>
Date: Sat, 9 Jul 2016 17:22:59 +0100

> On 8 Jul 2016, at 18:49, Poul-Henning Kamp <[email-masked]> wrote:

Before I go any further, let me agree with Mike about this: I am generally 100% on the “structured serialisation format for header values” approach being discussed. I definitely want to see something in this space.

> Everybody else can avoid sending Accept-Encoding entirely.

This is one of those interesting decisions that has a tendency to make life very tricky for implementation authors. Generally speaking it is unwise to have the “do nothing” behaviour be totally wrong: that’s arguably what this would do. Specifically, if you write a HTTP implementation and don’t know anything about the Content-Encoding header, in the current model you shouldn’t be served anything weird. Basically, you’ll get data you recognise and expect. You need to opt-in to the compression. That means that RFC-compliant servers, written to expect that gzip will work in the absence of an Accept-Encoding header saying otherwise, will break naively-written clients that never considered Accept-Encoding.

I’m not sure this is a compelling reason *not* to do this. After all, this is not the only such behaviour that cannot be opted-into (see also chunked transfer encoding). It’s just a data point worth considering.

(I should also note that it’s weird that all Accept-Encoding header lists are implicitly terminated with “identity”, but the absence of such a header translates into “gzip”. That’s a strange inconsistency and I guarantee it’ll trip people up. It’s only an inconsistency though, and your rationale for it is well-reasoned.)

Cory

---

From: Tim Bray <[email-masked]>
Date: Sat, 9 Jul 2016 10:47:34 -0700

> How do we specify this in RFCs

​In my work at AWS I am in the middle of shipping a product with a JSON DSL
and writing a validator for the DSL.  The current repertoire of tools for
specifying and syntax-checking JSON declaratively is ​somewhere between
miserable and pathetic.  JSON Schema is better than nothing but the nature
of its construction makes it basically impossible to produce human-friendly
error messages.  CDDL and JSON Content Rules are interesting attempts but
still not remotely mature.

Also, I have found it more or less impossible to write MustIgnore rules
into JSON Schema saying like “If you spot a field you don't recognize,
that's OK, but if this field does occur it MUST meet these constraints”.
Which I think is important for network payloads.

At work, we actually used JSON Schema to help validate (better than
nothing), but with buckets of procedural code for this and that.  In the
actual specification we're going to publish, screw the schema stuff, it's
all IETF-style English prose: “The Foo object MUST contain a field named
Bar whose value MUST be an array all of whose members MUST be strings that
<description of required string syntax>”.

I’m actually not convinced that the lack of a good way to declaratively
specify a JSON DSL, for example for HTTP headers, is that big of a
problem.  Schemas are overrated.  Clear English prose is under-rated.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 09 Jul 2016 20:45:44 +0000

--------
In message <[email-masked]>, Cory Benfield w
rites:

>This is one of those interesting decisions that has a tendency to make 
>life very tricky for implementation authors. Generally speaking it is 
>unwise to have the 'do nothing' behaviour be totally 
>wrong: that's arguably what this would do. Specifically, if you 
>write a HTTP implementation and don't know anything about the 
>Content-Encoding header, in the current model you shouldn't be 
>served anything weird.

Remember, I'm talking about a new HTTP version here, with a different
and closely shaved semantic layer.

In that version, I would argue that "anything weird" includes
uncompressed data.

If the client sends HTTP/1.1, then the server knows the semantics are
"dot-one" and "anything weird" is compression.

If the client sends HTTP/1.2 *and* the server replies with HTTP/1.2
as well, then they *both* know that "anything weird" is uncompressed.

>(I should also note that it's weird that all Accept-Encoding 
>header lists are implicitly terminated with 'identity', 
>but the absence of such a header translates into 'gzip'. 
>That's a strange inconsistency and I guarantee it'll 
>trip people up. It's only an inconsistency though, and your 
>rationale for it is well-reasoned.)

It was meant as a strawman to show how much improvement could be had,
not as a final proposal :-)

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 09 Jul 2016 21:26:12 +0000

--------
In message <[email-masked]>
, Tim Bray writes:

>I'm actually not convinced that the lack of a good way to declaratively

... and under-available.

That argument would also be more convincing if ABNF was used less in RFCs :-)

But anyway we do it, formally or with (un)clear English, we need
to convince ourselves that there is usable way to specify HTTP
headers comprehensively using JSON.

Best way to do that, is to write up complete specifications for a
couple of nontrivial headers.

And if none of the present formal tools/languages for specifying
JSON are workable, we should seriously consider if we want JSON-lite
instead of the full monty, so that the can be made to work.

---

From: Kevin Marks <[email-masked]>
Date: Sat, 9 Jul 2016 14:51:53 -0700

On Fri, Jul 8, 2016 at 11:44 AM, Phil Hunt <[email-masked]> wrote:
> Not sure if this has been discussed. One of the biggest problems with HTTP
> request signing has been repeat headers. It presents problem of detecting
> which headers are intended and which header was signed first.
>
> It would be nice if the JSON encoding handled arrays so that the demand for
> duplicate headers is removed.  Signing could then be more successful and
> could even stipulate that the presence of a repeat header in a signed
> request is a failure condition.
>

JSON doesn't help with this, as key order in objects (as opposed to
lists) is not required or defined.
Different programming languages behave differently here when
iterating. PHP preserves definition order, python orders by hash of
the key, and Go randomises the  order (to prevent accidental
dependencies).
Parsing JSON into native form and writing it out again makes key order
indeterminate.
As http headers have order dependent behaviour, this is a problem with
replacing the key: value with JSON.

---

From: "Phil Hunt (IDM)" <[email-masked]>
Date: Sat, 9 Jul 2016 15:30:04 -0700

Yes. I was thinking of combining this with the JOSE specs so signing becomes possible. 

Phil

> On Jul 9, 2016, at 2:51 PM, Kevin Marks <[email-masked]> wrote:
>

---

From: Julian Reschke <[email-masked]>
Date: Sun, 10 Jul 2016 09:50:52 +0200

On 2016-07-08 20:44, Phil Hunt wrote:
> Not sure if this has been discussed. One of the biggest problems with
> HTTP request signing has been repeat headers. It presents problem of
> detecting which headers are intended and which header was signed first.
>
> It would be nice if the JSON encoding handled arrays so that the demand
> for duplicate headers is removed.  Signing could then be more successful
> and could even stipulate that the presence of a repeat header in a
> signed request is a failure condition.
> ...

FWIW, duplicate headers aren't really needed, when the header field was 
defined properly (so the only problem is Set-Cookie, see 
<https://greenbytes.de/tech/webdav/rfc7230.html#field.order>).

The current JFV spec uses JSON arrays as data mpdels, and thus maps well 
to the HTTP header field data model:

1. Ordering is significant

2. Values can appear in separate field values, or comma-delimited

3. Multiple values can be recombined using comma as delimiter

etc.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 10 Jul 2016 08:04:59 +0000

--------
In message <[email-masked]>, Julian Reschke
><https://greenbytes.de/tech/webdav/rfc7230.html#field.order>).

I'll second the call for a blanket ban on repeat headers if they
use the new (JSON-)syntax.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 10 Jul 2016 10:34:00 +0200

> use the new (JSON-)syntax.

But that's not what I said.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 10 Jul 2016 09:09:39 +0000

--------
In message <[email-masked]>, Julian Reschke

No, that is what Phil asked for:

> duplicate headers is removed.  Signing could then be more successful and could
> even stipulate that the presence of a repeat header in a signed request is a
> failure condition.

I would go as far to make all repeat header a failure condition under the
new format.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 10 Jul 2016 11:35:49 +0200

> I would go as far to make all repeat header a failure condition under the

I'm not sure how that would be helpful, nor how that would be legal wrt 
the base spec.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 10 Jul 2016 09:45:13 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>I'm not sure how that would be helpful, nor how that would be legal wrt 

In the case where we augment the existing spec with a new way of
encoding the content in (new) headers, we can specify that repeat
(new) headers encoded that way are illegal.

That doesn't change anything in the base spec, it merely imposes a
tighter restriction for a subclass of (new) headers.

In the case where we implement a new semantic layer, that will
by and large be a replacement of the base spec, so we're free
to do what we want, subject to the necessary interop/compat
considerations.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 10 Jul 2016 11:57:13 +0200

> (new) headers encoded that way are illegal.

"A sender MUST NOT generate multiple header fields with the same field 
name in a message unless either the entire field value for that header 
field is defined as a comma-separated list [i.e., #(values)] or the 
header field is a well-known exception (as noted below)." -- 
<https://greenbytes.de/tech/webdav/rfc7230.html#rfc.section.3.2.2.p.2>

But right now the spec *is* written to use the list construct, and I 
believe that's a good thing, as it's IMHO better to consider multiple 
instances as legal, and require the definition of the header field to 
deal with it.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 10 Jul 2016 10:10:26 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>instances as legal, and require the definition of the header field to 

I think it is a bad thing.

It prevents streaming processing of headers, since you never know
when you have the full picture for a particular header, until you've
received them all and seen that there are no more instances.

It means also means that either you have to rewrite the headers, or
all your code needs to do the brute-force collection scan and handle
an array of headers for further processing.  Both of which is wasteful
in terms of CPU and memory.

I see no advantages that come even close to compensating for those
disadvantages, but if I have overlooked something, please enlighten me...

---

From: Yanick Rochon <[email-masked]>
Date: Sun, 10 Jul 2016 08:03:51 -0400

> disadvantages, but if I have overlooked something, please enlighten me...
I agree. Normally, multiple instances are legal in JSON and the latest will
simply override the previous one, but in the case of headers, it is not
good indeed and may produce problem when a repeating header comes after a
few preliminary configurations have been set and will force retrieving the
entire header object instead of the actual implementation (i.e. it would be
a regression, not an improvement). Since streaming headers (feasable in
JSON) is preferrable, duplicated keys in the JSON header should either be
ignored or rejected with an error. It is the responsibility of the sender
to properly format it's header before sending, not the server to untangle a
messy one. In my opinion, repeating headers should be put in an array,
period.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 10 Jul 2016 14:31:09 +0200

On 2016-07-10 14:03, Yanick Rochon wrote:
> On 10 July 2016 at 06:10, Poul-Henning Kamp <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     --------
>     In message <[email-masked]
>     <mailto:[email-masked]>>, Julian
>     Reschke writes
>     :
>
> I agree. Normally, multiple instances are legal in JSON and the latest
> will simply override the previous one, but in the case of headers, it is
> not good indeed and may produce problem when a repeating header comes
> after a few preliminary configurations have been set and will force
> retrieving the entire header object instead of the actual implementation
> (i.e. it would be a regression, not an improvement). Since streaming
> headers (feasable in JSON) is preferrable, duplicated keys in the JSON
> header should either be ignored or rejected with an error. It is the
> responsibility of the sender to properly format it's header before
> sending, not the server to untangle a messy one. In my opinion,
> repeating headers should be put in an array, period.

It seems you are confusing several issues here: multiple header field 
instances in the HTTP message, and duplicate member names in a JSON 
object. These are completely orthogonal issues.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 10 Jul 2016 20:26:21 +0200

> disadvantages, but if I have overlooked something, please enlighten me...

I have a different appreciation on this matter. We've seen over the year
that people were introducing headers for a specific usage and were pretty
sure these ones ought to be unique. But in the end they were not. The best
example of this is X-Forwarded-For. Each reverse-proxy adds it. Most
implementations never used to consider it as a list as nothing was clear
regarding this, and initial designs were simply "setting" the header.
Some naive designs just used to append it after all headers. Some would
replace the CR LF CR LF with "CR LF xff CR LF CR LF". Each of them was
pretty sure to do the right thing regarding a supposedly unique header.

But they were wrong. In fact they were doing the right thing for a list
and not the right thing for a single header. And that's fortunate because
now we all know that seeing chained XFF is something very common when you
start to stack a load balancer and a cache for example.

All this to say we cannot decide in advance how headers will be produced
nor what they will mean. But that's not a problem. The problem is with
how the headers are interpreted. That's what we need to work on. I'd
instead suggest that all new headers are lists, that they will support
multiple occurrences, and that for all those where it's not obvious what
multiple occurrences will mean, we explicitly mention that only the first
one should be used. We can then have a predictible behaviour because the
first header found allows us to take a routing decision on the fly,
unless we know something specific regarding this one. And this way,
appending new values (a la XFF) will have no unexpected side effect on
those not willing to consume them.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 10 Jul 2016 22:20:18 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>object. These are completely orthogonal issues.

Uhm, no?

They only become orthogonal if whoever specifies the JSON header
takes great care to make that happen.

Allowing split headers forces all headers to be defined as JSON
lists, which means we're knee-capping header-designers from the get go.

Take this RFC7231 example:

	Accept: text/plain; q=0.5, text/html, text/x-dvi; q=0.8, text/x-c

If we allow split headers for JSON, this can only be defined as JSON list,
in order that it can also be sent as:

	Accept: <JSON for "text/plain; q=0.5">
	Accept: <JSON for "text/html">
	Accept: <JSON for "text/x-dvi; q=0.8">
	Accept: <JSON for "text/x-c">

But that leaves it to the application writer to spot and detect and
handle this degenerate case from a lazy sender:

	Accept: <JSON for "text/plain; q=0.5">
	Accept: <JSON for "text/html">
	Accept: <JSON for "text/x-dvi; q=0.8">
	Accept: <JSON for "text/x-c">
	Accept: <JSON for "text/plain; q=0.0">

It's even more complicated for a proxy sender which wants to
modify the text/plain priority:  It needs to spot both copies
and change them both (or delete one of them).

(It seems to me that there is a class of smuggling attacks
where the proxy sees q=0.5 and the server q=0.0, which
RFC7231 does not seem address at all?)

If we instead, as I propose, require that JSON headers *never* be
split, then it becomes both possible and rather obviously smarter
to define this header as a JSON object, keyed by the media type:

	Accept: { 					\
		"text/plain": <JSON for "q=0.5">,	\
		"text/html": <JSON for no parameter>,	\
		"text-xdvi": <JSON for "q=0.8">,	\
		"text/x-c": <JSON for no parameter>	\
	}

A sender wishing to modify the priority, just sets the
corresponding JSON object using the native languages
JSON facility:

	req.accept["text/plain"] = <JSON for "q=0">

The receiver can then simply load the JSON as JSON, and the
application does not have to explicitly check for duplicate
media types, but can simply look up "text/plain" in the
JSON object.

So what happens if the sender sends bad JSON anyway ?

	Accept: { 					\
		"text/plain": <JSON for "q=0.5">,	\
		"text/html": <JSON for no parameter>,	\
		"text-xdvi": <JSON for "q=0.8">,	\
		"text/x-c": <JSON for no parameter>	\
		"text/plain": <JSON for "q=0.0">,	\
	}

Well, RFC7159 says:

	4. Objects

	[...]The names within an object SHOULD be unique.

	An object whose names are all unique is interoperable in
	the sense that all software implementations receiving that
	object will agree on the name-value mappings.  When the
	names within an object are not unique, the behavior of
	software that receives such an object is unpredictable.
	Many implementations report the last name/value pair only.
	Other implementations report an error or fail to parse the
	object, and some implementations report all of the name/value
	pairs, including duplicates.

In other words:  Don't do that.

What really happens is this:  We just opened another door to the
exact same smuggling attack as I mentioned above.

But this time we can shut them all with one single line of text:

	"Duplicate keys in JSON objects SHALL cause and be treated
	as connection failure."

So all in all, split JSON headers would be a really bad idea.

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 11 Jul 2016 13:05:44 +0900

Hello Paul-Henning,

> 	req.accept["text/plain"] = <JSON for "q=0">

My understanding is that you are extremely concerned about the speed at 
which headers can be processed. My guess would be that deserializing, 
changing, and reserialising JSON headers takes more time than 
detecting/processing duplicate headers. But I of course might be wrong.

Could you give some more background on why speed-wise, de/serializing is 
okay for you, but duplicate detection isn't?

> 	"Duplicate keys in JSON objects SHALL cause and be treated

How are you going to tell your favorite JSON library to behave that way?

Regards,   Martin.

---

From: Andy Green <[email-masked]>
Date: Mon, 11 Jul 2016 12:37:19 +0800

> detecting/processing duplicate headers. But I of course might be

I'm a bit bemused why the world needs JSON headers instead of the cool
stuff for header coding in http/2, but I can give one point of view
related to duplicate headers and efficiency.

In libwebsockets we use bytewise state machines for everything,
including http/1.x header parsing.  Normally the library tries to stay
out of the way of the application code and provide events and
information to it as it becomes available, without the need for
buffering on the library side.

But in the case of http/1.x headers, we can't give the application any
definitive report on header payload until we got the whole lot, since
headers may be appended to at any point.  Further, it means we have to
keep the whole payload of every recognized header around in case it was
subject to appending later.

Actually it'd be nice, and efficient, if we could assemble one header
payload in the library, pass it up to the application to copy or act
on, and then reuse the buffer, as we go through the incoming, possibly
fragmented, header content.

Parsing the JSON out of it is very cheap and quite compatible with
being integrated into the general header stream, bytewise parser, but
deferring being able to get a definitive result to pass up as it is now
in http/1.x is painful if you are serious about memory efficiency.

In the actual scenario being asked about if you can eliminate always
having the final result for a header pending until all the headers came
(in JSON, by saying each header may only appear once and multiple
results come in an array on that), you can reissue the headers to
forward header-by-header as they are parsed without storing them all,
which is radically more efficient if what you're doing with them allows
it.

-Andy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 11 Jul 2016 05:53:06 +0000

--------

>which headers can be processed.

Speed, reliability *and* security.

HTTP is so infested with weird corner-cases that 10 RFC's were needed
to explain _most_ of them, whatever we do going forward, we should
strive for the simplest realistic solution.

Therefore I want to eliminate as many cornercases as possible before
they ever appear in the wild.

>okay for you, but duplicate detection isn't?

De-serialization of JSON objects already perform duplicate detection,
why should we have to do it again in the application code ?

What happens when people who don't know about this fine detail
fails to deduplication in their application code ?

>How are you going to tell your favorite JSON library to behave that way?

I don't need to.  As long as a relevant fraction of HTTP speakers
do, then attempting to send duplicate keys will be sufficiently
broken that it won't work in practice.  Look at it as "herd immunity".

Also, my JSON parser was 500 linies of C-code in first try, it's not
like it is rocket science.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 11 Jul 2016 05:55:34 +0000

--------

>fragmented, header content.

Exactly.

You also avoid the O(N^2) search operation to see if split headers
happened in the first place.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 08:02:55 +0200

> takes great care to make that happen.

That's what my draft tries to enforce by requiring the data model to be 
a JSON *array*.

> lists, which means we're knee-capping header-designers from the get go.

I would call it: "steer them to a format that is robust with regards how 
header fields can be repeated in messages"

> 	Accept: <JSON for "text/plain; q=0.0">

The "degenerate" case can happen in a single header field as well.

(yes, discussing *simplifiyng* certain header fields is interesting as 
well, but it again is orthogonal to this discussion)

> and change them both (or delete one of them).

Yes.

> RFC7231 does not seem address at all?)

Only if it's broken by not processing all field instances, no?

> media types, but can simply look up "text/plain" in the

Yes, that's a different data model for this header field, that makes 
certain operations simpler.

> 	"Duplicate keys in JSON objects SHALL cause and be treated

But then we can't rely on generic JSON parsers anymore.

> So all in all, split JSON headers would be a really bad idea.

After replying to your mail, I'm even more convinced that it's the right 
thing to do.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 08:13:21 +0200

On 2016-07-11 06:44, Jim Manico wrote:
> JSON-Schema should be able to drive such a rule; if we can't do that
> today - party on, new version of JSON-schema?

How does JSON-Schema affect what a RFC7159-compliant JSON parser does?

(Remember the problem is on the producing side)

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 08:12:12 +0200

> related to duplicate headers and efficiency.

I don't understand the "instead" here. The data model for header fields 
exactly the same in HTTP/1.1 and HTTP/2, so the proposed format applies 
to both.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 08:19:03 +0200

> strive for the simplest realistic solution.

10?

> they ever appear in the wild.

I understand that, and it's an important topic for new versions of HTTP. 
*This* spec however isn't about a new version of HTTP. It tries to 
address a concrete problem (people consistently getting definitions for 
new header fields wrong), and it's supposed to apply to all HTTP 
versions currently in us.

> why should we have to do it again in the application code ?

It doesn't do that.

> broken that it won't work in practice.  Look at it as "herd immunity".

But then, recipients are already free to do that on the layer above 
(think broken Content-Length information), so if applying pressure to 
broken senders is the goal, this is not really needed.

> like it is rocket science.

Well, people having to write a custom parser is exactly what I want to 
avoid, because most get it wrong.

Best regards, Julian

---

From: Andy Green <[email-masked]>
Date: Mon, 11 Jul 2016 14:37:37 +0800

On Mon, 2016-07-11 at 08:12 +0200, Julian Reschke wrote:
> > > My understanding is that you are extremely concerned about the
> > > speed
> > > at

I mean now it exists and is formalized, and implementations exist,
shouldn't people be being encouraged to directly use HPACK?

Agreed, JSON would in retrospect be cooler than HTTP/1.x headers.  But
I'm surprised there is any energy to go and make more ways to send HTTP
headers.

Mightn't it be better to look at tweaking HPACK to have a "strict flag"
or somesuch to give some of the qualities discussed here, like only one
instance of each header?  Otherwise HPACK is pretty nice.

Apologies if I miss the point but JSON headers won't directly fly on
HPACK AFAICS, unless your idea is making an uber-header and its value
is the whole JSON payload.  But that's needlessly incompatible when
it's dealing with the same traditional headers and values HPACK already
supports actually.

-Andy

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 11 Jul 2016 16:54:31 +1000

Catching up with a pile of post-holiday e-mail...

> On 9 Jul 2016, at 3:49 AM, Poul-Henning Kamp <[email-masked]> wrote:
> 

Good. To be clear, in the lead-up to adoption, we said that this draft would be a starting point, and the approach *could* change dramatically, based upon discussion (and hopefully, running code and experimentation).

> However, and you knew that would be coming, I have three major issues with this.

Personally, I have always assumed that one of the major goals of such a convention would be enabling such an encoding in the future (probably in HTTP/n, since it's a backwards-incompatible change, although we might be able to do something in /2).

> How do we specify this in RFCs

I agree with the identified need (VERY much; again, one of the big goals for me here is to minimise the need for "advice" like we documented in 7231 8.3.1), but I'm not so sure about JSON Schema (as others have noted). No doubt we'll get into this.

> Simplify semantics

Yet again personally, I was thinking we might want to introduce a set of alternative JSON headers to the most common ones, and allow some form of negotiation for their use. Perhaps j-accept-encoding, j-user-agent and so on.

Introducing minor version numbers to HTTP has proven to cause a lot of confusion; that's the main reason we got rid of minor version numbers in HTTP with /2. 

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 08:56:25 +0200

> shouldn't people be being encouraged to directly use HPACK?

Doesn't parse.

If you use HTTP/2, you always use HPACK. If you use HTTP/1.1, you never 
use HPACK. There is nothing to choose here.

> I'm surprised there is any energy to go and make more ways to send HTTP

Actually, it's an attempt to have *less* ways, in that it gives people 
defining new header fields a framework to use, instead of having to 
invent new micro-syntax every time.

> it's dealing with the same traditional headers and values HPACK already

They "fly" exactly the same way as in HTTP/1.1; could you elaborate on 
where you see a difference?

Best regards, Julian

---

From: Andy Green <[email-masked]>
Date: Mon, 11 Jul 2016 15:15:03 +0800

On Mon, 2016-07-11 at 08:56 +0200, Julian Reschke wrote:
> > > > I'm a bit bemused why the world needs JSON headers instead of
> > > > the
> > > > cool

In there thread there's talk of this becoming HTTP/1.2...

> invent new micro-syntax every time.

Maybe following this thread I get the wrong end of the stick... if it's
just the additional option of JSON-formatted values where there is
structure in the header value, that makes sense all around, and works
fine with HPACK.

In the thread it seemed people discuss actual JSON formatted headers as
a whole, ie, the whole header block becomes JSON.

I dunno there is any point to that, and it won't fly with HPACK.  But
maybe I just misread what was suggested.

> where you see a difference?

JSON formatted values would be fine... in fact websocket extensions
option negotiation could have done with that...

-Andy

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 09:26:07 +0200

On 2016-07-11 09:10, Jim Manico wrote:
> < How does JSON-Schema affect what a RFC7159-compliant JSON parser does?
> We're talking higher level standards that consume JSON in some way, right? So the producer would run the final JSON through a standard JSON-Schema before shipping off the JSON to ensure compliance (ensuring the producer work is accurate), and the consumer would use the same Schema to validate before parsing (so we can catch bad JSON way before a fragile parser gets involved).
>
> This is nothing special, many frameworks that do JSON-Object mapping do this already...

Just like senders validate what they send today :-)

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 11 Jul 2016 09:29:10 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>header fields can be repeated in messages"

You still have not given any reasoning why repeating them is good idea
in the first place.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 11:59:48 +0200

> You still have not given any reasoning why repeating them is good idea

It may not have been a good idea in the first place, but that's what we 
currently have in HTTP, both 1.* and 2. I'm arguing that it's better to 
live with it, and to make things as consistent as possible with respect 
to this.

Best regards, Julian

---

From: Cory Benfield <[email-masked]>
Date: Mon, 11 Jul 2016 11:19:25 +0100

> On 9 Jul 2016, at 21:45, Poul-Henning Kamp <[email-masked]> wrote:
> 

Fair enough. =) If we’re making supporting Content-Encoding: gzip mandatory, then that makes sense. However, in that world I’d want to say that we shouldn’t be implicitly appending identity, we should be implicitly appending gzip. If we’re saying that uncompressed data is weird, then you should have to ask for it explicitly, IMO.

> not as a final proposal :-)

Sure: that’s why I put it as a parenthetical note rather than stick on it. I just wanted to flag it now so that it’s dealt with at some point, because as I said earlier, I like the general shape of this.

Cory

---

From: Cory Benfield <[email-masked]>
Date: Mon, 11 Jul 2016 11:25:22 +0100

> On 11 Jul 2016, at 08:15, Andy Green <[email-masked]> wrote:
> 

I don’t think anyone has suggested that. They have suggested making header field *values* entirely JSON, but no-one has suggested making the entire header block a JSON object as far as I have seen.

Cory

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 11 Jul 2016 19:32:25 +0900

> happened in the first place.

There are many well-known data structures that avoid O(n^2).

[I personally might try something like gperf (perfect hashing, see 
https://savannah.gnu.org/projects/gperf/) on a wide collection of 
well-known headers, which would make search for these headers O(1), and 
overall O(n) for n headers. Linear search (which leads to O(n^2) for a 
series of headers) could still be used on less well known stuff.]

Regards,   Martin.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 11 Jul 2016 10:54:52 +0000

--------

>There are many well-known data structures that avoid O(n^2).

Yes, and they're probably all overkill in the typical N<20 case of
HTTP headers...

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 11 Jul 2016 10:57:15 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>live with it, and to make things as consistent as possible with respect 

I couldn't possibly disagree more.

We should eliminate and sunset as many of the warts of HTTP as we
reasonably can, in order to make it an easier and more efficient
protocol to understand and implement.

In particular, I find it very unintuitive that we would ever constrain
the JSON formats utility, in order to retain a misfeature like
repeated headers.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 13:13:26 +0200

> protocol to understand and implement.

Maybe. But this is a complex change, and it will be hard to deploy.

The intent of this draft was to describe something that works today with 
the semantics we have.

Both are interesting things to do; I just believe that requiring to do 
both things at the same time is likely to kill the effort, or delay it 
significantly.

> the JSON formats utility, in order to retain a misfeature like

How exactly is it "constrained"?

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 11 Jul 2016 13:20:38 +0200

> How exactly is it "constrained"?

The top level is an array (even though you nicely hide that by calling
it a list).

Grüße, Carsten

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 11 Jul 2016 11:20:33 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>How exactly is it "constrained"?

As I described previously:  It forces all headers to use a list-form,
even when a key-value form is what they need.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 13:43:29 +0200

> even when a key-value form is what they need.

You can have a *list* of objects, and these objects can contain 
key-value form.

See example in 
<https://greenbytes.de/tech/webdav/draft-ietf-httpbis-jfv-01.html#example.content-disposition>.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 11 Jul 2016 13:59:57 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>You can have a *list* of objects, and these objects can contain 

Yes, but you can't exploit the uniqueness of the object keys,
as my example showed.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 16:14:13 +0200

> Yes, but you can't exploit the uniqueness of the object keys,

a) you still could (put the object into an array...), but

b) relying on the uniqueness is a problem anyway (due to how JSON works).

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 11 Jul 2016 17:44:30 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>b) relying on the uniqueness is a problem anyway (due to how JSON works).

Sorry, but I have no idea what you are saying there...

---

From: Yanick Rochon <[email-masked]>
Date: Mon, 11 Jul 2016 14:02:09 -0400

> Sorry, but I have no idea what you are saying there...
I believe he is saying that standardized JSON parsers do not care for
duplication in JSON properties, and simply overwrite any previous values,
which is as defined in the specs and an expected behaviour, which does not
resolve the issue at all.

I personally would favor a strict enforcement, where a duplicated key
should generate errors, but this is agains the specs...

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 20:05:32 +0200

> Sorry, but I have no idea what you are saying there...

You can't rely on how a JSON parser reports or that it rejects duplicate 
fields.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 11 Jul 2016 18:27:52 +0000

--------
In message <[email-masked]>
, Yanick Rochon writes:
>--001a113ec7849c5bac05375ff43b

>resolve the issue at all.

As long as no implementation prefers the first key/value pair, that
does resolve the "smuggling" issue.

>should generate errors, but this is agains the specs...

I'm still confused, which specs say you can have duplicate keys ?

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 11 Jul 2016 21:01:07 +0200

> You can't rely on how a JSON parser reports or that it rejects duplicate

Guys, I still don't understand why it would not work if everything is
considered to be a list. A list can be ordered, right ?

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Jul 2016 21:11:09 +0200

> considered to be a list. A list can be ordered, right ?

Inside a JSON *object*, the wire format allows multiple instances of the 
same key (member name), and the recipient behavior for these cases is 
undefined.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 11 Jul 2016 21:25:15 +0200

> same key (member name), and the recipient behavior for these cases is

OK thanks for explaining. But then what situation could lead to this
confusing object to be emitted ? I thought that it was only related to
posting multiple headers in which case I don't see the issue if we
consider that all these headers are lists, and are then concatenated
by the recipient.

Thanks,
Willy

---

From: Yanick Rochon <[email-masked]>
Date: Mon, 11 Jul 2016 15:47:32 -0400

> consider that all these headers are lists, and are then concatenated
If, for example, a sender sends a request such as

{ ..., "Accept":
"application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
..., "Accept": "text/plain;q=0.9", ... }

through a clumsy string concatenation, what is the suggested behavior?

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 12 Jul 2016 00:35:54 +0200

> same key (member name), and the recipient behavior for these cases is

That is a rather liberal use of "allows".  It definitely not "allowed",
but it is sure *possible* to send malformed JSON.

4.  Objects

   An object structure is represented as a pair of curly brackets
   surrounding zero or more name/value pairs (or members).  A name is a
   string.  A single colon comes after each name, separating the name
   from the value.  A single comma separates a value from a following
   name.  The names within an object SHOULD be unique.

      object = begin-object [ member *( value-separator member ) ]
               end-object

      member = string name-separator value

   An object whose names are all unique is interoperable in the sense
   that all software implementations receiving that object will agree on
   the name-value mappings.  When the names within an object are not
   unique, the behavior of software that receives such an object is
   unpredictable.  Many implementations report the last name/value pair
   only.  Other implementations report an error or fail to parse the
   object, and some implementations report all of the name/value pairs,
   including duplicates.

Grüße, Carsten

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 12 Jul 2016 05:22:37 +0200

> through a clumsy string concatenation, what is the suggested behavior?

For me it means it sent two values for the Accept header defined as a list,
the first one being "appliation/xml...", and the second one "text/plain...".

I'm having an issue with the representation above though (and maybe I missed
certain things recently), because my understanding of JSON headers till now
was that one would send :

   { "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5" }

as the value of the first Accept header field, then

   { "text/plain;q=0.9" }

as the value of the second Accept header field.

And it's the recipient which would then decide to make a list of it because
we'd specifiy that each JSON-encoded header is a clearly defined list, so on
first header received it would do something like this (I'm not 100% sure about
the syntax as I'm not good at JSON) :

   { "Accept": {
        "application/xml", "application/xhtml+xml", "text/html;q=0.9",
        "text/plain;q=0.8", "image/png", "*/*;q=0.5" } }

Then when receiving the second header, it would simply append its value to
the list :

   { "Accept": {
        "application/xml", "application/xhtml+xml", "text/html;q=0.9",
        "text/plain;q=0.8", "image/png", "*/*;q=0.5", "text/plain;q=0.9" } }

Since it seams easy to me I'm having an issue understanding the deeper
problem and am pretty sure I missed something :-/

Regards,
Willy

---

From: Yanick Rochon <[email-masked]>
Date: Mon, 11 Jul 2016 23:58:03 -0400

On 11 July 2016 at 23:22, Willy Tarreau <[email-masked]> wrote:

> ... *snip*

> problem and am pretty sure I missed something :-/

Well, first arrays are enclosed in square brackets, nor curly ones :)

Second, I am not talking about the format per se, but about possible
headers being sent as described (i.e. with duplicated keys). Of course,
JavaScript objects cannot have duplicate keys, however JSON representations
as strings are not bound to this limitation, and may very well be sent as
such. (Also, even if it is expected that every values must be an array,
JSON allows a few more possible values to be set, like strings, etc.) The
point is that, { "Accept": "...", "Accept": [ "..." ] } is a valid JSON
object, yet is (or should be) invalid as header value. What then?

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 12 Jul 2016 06:10:40 +0200

> Well, first arrays are enclosed in square brackets, nor curly ones :)

OK thanks :-)

> object, yet is (or should be) invalid as header value. What then?

But precisely my point is to declare that all JSON headers are lists (or
arrays apparently in JSON). Thus you just send array *elements* and the
recipient appends them to the array. So you don't have the object above,
you just have { "Accept": [ "..." ] }. This removes the possibility of
duplicate keys and keeps the ability to have multiple values for a header.
And it also allows to stop at the first value when you expect to use only
one value for a header that you don't intend to check for uniqueness.

Willy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Jul 2016 06:45:33 +0200

> but it is sure *possible* to send malformed JSON.

It is allowed by the structure of the *wire format*.

The *specification* has a "SHOULD have unique names", but then, that's 
only a SHOULD (exactly because we know we can't rely on it, otherwise we 
wouldn't have the prose about what recipients can do with it).

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 12 Jul 2016 08:41:52 +0200

> It is allowed by the structure of the *wire format*.

The syntax indeed cannot prevent it.
It's still not *allowed* in JSON.

> wouldn't have the prose about what recipients can do with it).

It is a SHOULD because people were chickening out because of a possible
political conflict with ECMA 404.  Note well that no reason is given to
ever violate that SHOULD.

Now, for performance reasons, there is no requirement on a receiver to
check for this constraint.  Protocol design 101 tells us that a lack of
checking will cause implementations to emit invalid JSON just because
they can (the "soup" effect).  Hence the description in RFC 7159 what
goes wrong when you do that.  (However, the security considerations fail
to mention the check-vs-use vulnerabilities that inevitably come from
the variety in implementation strategies; the last paragraph of Section
8 of RFC 7049 does apply.)

This discussion may be a bit off-topic for the HTTP WG, but I think it
is important to understand JSON when using it in HTTP.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Jul 2016 08:51:07 +0200

> ever violate that SHOULD.

But we do know why it's violated in practice:

1) Streaming might make it hard to check for senders,

2) People abusing it to add comments to JSON (by choosing a member name 
for comments, and repeating it).

...and probably for other reasons I'm not aware of.

> 8 of RFC 7049 does apply.)

Maybe something for JSONbis?

> is important to understand JSON when using it in HTTP.

Absolutely; and the conclusion might well be that we won't use JSON on 
the wire.

Best regards, Julian

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 12 Jul 2016 16:05:17 +0900

> Yes, and they're probably all overkill in the typical N<20 case of

Then just don't use them for the typical case. You can still make them 
kick in if you detect a large number of headers.

Regards,    Martin.

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 12 Jul 2016 19:10:57 +1200

> Absolutely; and the conclusion might well be that we won't use JSON on

Personally I hope we don't.

I was fine with it as a way to write ABNF-like descriptions in future
RFCs to make everyones custom headers have a more generic syntax that
our parsers could handle easier.

But using a textual representation on the wire for future improvements
is something we should be looking at avoiding, not encouraging.

HTTP/2 HPACK offers some new possibilities by adding integer encoding
for header field-value that the recipient is not required to write in
textual format before processing. Lets not throw that advantage away.

Amos

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Jul 2016 09:41:25 +0200

> textual format before processing. Lets not throw that advantage away.

HPACK, as defined, does not allow it.

A future version might (requiring to bump up the protocol version).

One goal of a common syntax for field values of course is to later on 
take advantage of it, just the way you said.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 12 Jul 2016 09:55:19 +0200

> is something we should be looking at avoiding, not encouraging.

Going to a JSON-like data model would also prepare using other
representations (such as binary ones) of that data model.

Grüße, Carsten

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 12 Jul 2016 08:22:03 +0000

--------
In message <[email-masked]>, Julian Reschke writes
:

Tl;DR:  I would prefer CBOR and base{64|85}(CBOR) to JSON

>for comments, and repeating it).

That is one of the places where I went "Doh?" when I first saw JSON,
who the heck defines a data-interchange format with no support for
comments ?!

The answer, of course, is that JSON is not a data-interchange format,
it is a subset of the JavaScript source code syntax, and they didn't
include comments in that subset.

JSON being "source compatible", has been correctly critized for
being a wide open invitation for unsafe data-ingestion practices,
with plenty of real world instances to validate the argument.

We can take it as an absolutely certainty, that if we choose JSON
serialization, too many people will just "eval" it into Javascript,
opening them wide for malicius point-edits[1] and hostile actors.

This is of course not our first priority for deciding, but it counts
clearly against JSON.

>Absolutely; and the conclusion might well be that we won't use JSON on 

Carstens CBOR presentation from IETF94 is very good background here,
and as a compentent data-interchange format CBOR (RFC7049) will get
my vote over JSON any time.

The problem with CBOR is that there does not seem to be a "CAOR"
we can use with H1[2], but maybe that is a good thing.

Eyeballing it, the size of base85(CBOR(obj)) or even base64(CBOR(obj))
seems to acceptably close to JSON.

An implementation would use the same CBOR serdes code for both H1
and H2, with a trivial gloss of baseXX serdes on top for H1.

I like that, in particular I like that it makes H1<-->H2 interop
trivial, streamable, reliable and debuggable.

Security-wise it is also preferable, because it forces everybody
to use a proper serdes code[3], and it permantently closes the door
to unsafe hacks such as 's/q=[0-9.]*/q=0/'

CBOR gets my vote.

Poul-Henning

[1] My security friends would point to general principle, that one
    should avoid serializations which can be "point-edited" to change
    semantic content.  (Text-book example:   Computer printed cheques
    filling the amount field with non-space: "****1000$*".).

    Such "point-edited" HTTP headers are in widespread use in H1,
    for instance "nnoCection:" and similar hacks, but they are also
    used on the other side of the colon by smut-filters and "privacy
    controls", typically mangling Location:, (Set-)Cookie: and in some
    cases the URL.

    H2 and HPACK makes that shortcut a lot longer, but I very much
    suspect that these devices may retain their "application logic"
    and just deserialize, mangle and reserialize the headers.

[2] CBOR does have a "Diagnostic Notation" which renders it to text, but
    it is not meant to be a data-interchange format, and as a debugging
    tool includes information that is surplus to (our) requirements.

[3] At least until CBOR becomes valid syntax in INTERCAL.

---

From: Yanick Rochon <[email-masked]>
Date: Tue, 12 Jul 2016 09:15:38 -0400

> one value for a header that you don't intend to check for uniqueness.
As Carsten mentioned, the problem is not the possibilities as you describe,
as I think JSON would elliminate a few problems that arises with the actual
implementation, but the issue is that people will not honor it. Meaning
that, JSON allows non-array values, so you cannot expect people to strictly
observe a rule (HTTP header having array values) and not another (JSON
values may be of other types). My question is more in the nature of "if a
sender sends duplicated keys in the header, and/or some of them are not
array values, what then?" As others have stated, the JSON specs is
undefined on this (i.e. overwriting previous ones, ignoring subsequent
duplicated keys, or rejecting the object with an error), and most
implementations end up overwriting as a convention. However this would lead
into problems where hijackers could simply insert their headers before the
closing curly brace, thus overriding any previous, original header.

You can't use an undefined behavior, make it a convention and expect people
to honor it; it has to be defined.

---

From: [email-masked]
Date: Tue, 12 Jul 2016 15:31:45 +0200 (CEST)

> confusing object to be emitted ?

I suppose that this can happen in pipelined architecture where each processing step thinks it owns a header and does not check if another step didn't already set it before

IE when you mashup loads of third-party javascript and hope the resulting soup will work

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 12 Jul 2016 15:30:12 +0200

On Tue, Jul 12, 2016 at 09:15:38AM -0400, Yanick Rochon wrote:
> My question is more in the nature of "if a
> array values, what then?"

Ah OK that's much clearer now, though now I have no idea how to deal with
this :-/

Thanks for your explanation!
Willy

---

From: Martin Thomson <[email-masked]>
Date: Wed, 13 Jul 2016 09:53:02 +1000

FWIW, my dislike of CBOR is strong, but I've been warming to this
notion. But I would suggest that we NOT base{64,85} anything.  HTTP/2
carries octets very nicely, the only catch being 0x00, which we can
finagle around.

On 12 July 2016 at 18:22, Poul-Henning Kamp <[email-masked]> wrote:
> CBOR [...] includes information that is surplus to (our) requirements

Here you have identified the core challenge, and the biggest weakness of CBOR.

Apologies for the mangled quoting, but I think that it's a very
important point to make.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Jul 2016 02:57:11 +0200

On 13 July 2016 at 01:58:25, Martin Thomson ([email-masked]) wrote:
the only catch being 0x00, which we can 
finagle around.

Consistent overhead byte stuffing (COBS)?  (Just brainstorming.  Don’t know how important consistent overhead is here.)

Grüße, Carsten

---

From: Martin Thomson <[email-masked]>
Date: Wed, 13 Jul 2016 12:42:15 +1000

On 13 July 2016 at 10:57, Carsten Bormann <[email-masked]> wrote:
> Consistent overhead byte stuffing (COBS)?  (Just brainstorming.  Don’t know
> how important consistent overhead is here.)

I wouldn't bother with COBS, I would just negotiate or signal the use
of a new encoding, and say that that comes with no attached meaning
for 0x0.  I think that also fits neatly with PHK's desire to remove
multiple instances of the same header field.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 13 Jul 2016 07:57:44 +0000

--------
In message <CABkgnnWtXiac_E1X-ztYJMNQVzrVyPZBCvWHtmX=ksYvgU=[email-masked]>

>carries octets very nicely, the only catch being 0x00, which we can

We would use native CBOR in HTTP/2.

base{64|85} is only to shoe-horn CBOR into HTTP/1

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 14 Jul 2016 11:50:24 -0700

> On Jul 9, 2016, at 10:47 AM, Tim Bray <[email-masked]> wrote:
> 
> On Fri, Jul 8, 2016 at 10:49 AM, Poul-Henning Kamp <[email-masked]> wrote:
> ​In my work at AWS I am in the middle of shipping a product with a JSON DSL and writing a validator for the DSL.  The current repertoire of tools for specifying and syntax-checking JSON declaratively is ​somewhere between miserable and pathetic.  JSON Schema is better than nothing but the nature of its construction makes it basically impossible to produce human-friendly error messages.  CDDL and JSON Content Rules are interesting attempts but still not remotely mature.  
> 
> Also, I have found it more or less impossible to write MustIgnore rules into JSON Schema saying like “If you spot a field you don't recognize, that's OK, but if this field does occur it MUST meet these constraints”.  Which I think is important for network payloads.

I used Orderly <https://github.com/lloyd/orderly> for TPE, as in

  https://tools.ietf.org/html/draft-ietf-http-v11-spec-00#section-3.11

which is at least concise enough to not mind additional prose for if-this-then-must rules.

> At work, we actually used JSON Schema to help validate (better than nothing), but with buckets of procedural code for this and that.  In the actual specification we're going to publish, screw the schema stuff, it's all IETF-style English prose: “The Foo object MUST contain a field named Bar whose value MUST be an array all of whose members MUST be strings that <description of required string syntax>”.  
> 
> I’m actually not convinced that the lack of a good way to declaratively specify a JSON DSL, for example for HTTP headers, is that big of a problem.  Schemas are overrated.  Clear English prose is under-rated.

Use of JSON for anything other than javascript declarations is overrated.  But that's life.
I'd prefer to resurrect logic bags with the addition of UTF-8 and Unicode character escapes. ;-)

    https://tools.ietf.org/html/draft-ietf-http-v11-spec-00#section-3.11

....Roy

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 14 Jul 2016 12:29:39 -0700

> On Jul 10, 2016, at 3:10 AM, Poul-Henning Kamp <[email-masked]> wrote:
> 
> --------

Which is the nature of header fields in general, since we don't know if
a spoofing attempt is being made until we read all of the headers,
including the ones not defined as a list.  In any case, we can certainly
process them as a stream as long as we remember what has already been
processed so that an appropriate action can be taken for duplicates.

> in terms of CPU and memory.

Umm, no; we can just use a bit to indicate that, or a pointer for speed.

> disadvantages, but if I have overlooked something, please enlighten me...

The reason duplicate header fields exist is because email wanted them for
stuff like Received being appended to header blocks, so the RFC822 syntax
enables them and recipients need to deal with that regardless of semantics.
The semantics we defined for HTTP/1 allow for new field values to be
processed and forwarded by a recipient without knowing its definition.
This is used in practice today by high-speed message forwarding with
zero-copy constraints to append certain fields at the end of each
header block.

It is theoretically possible to do the same thing within a special encoding
of a header field value, but then every recipient MUST implement that special
encoding in that special way prior to any deployed instances of that protocol,
which cannot happen for HTTP/1.x.  No, changing the minor version is not sufficient,
since the fields are required to be processable regardless of field definition
by existing intermediaries, which means duplicates will mess up the field in
unexpected ways unless the value encoding assumes a list syntax will happen
whether you like it or not.

In any case, this discussion has nothing to do with Julian's proposal, which
is to define a common field syntax for HTTP/1.x field values that for some
insane reason want to use JSON for field processing.  Maybe that's just to add
UTF-8 characters, or for embedding field name-value pairs within other fields
(as was the case for logic bags, ages ago, or Key today), but it deserves to
be discussed in terms of what the draft defines and not hijacked into a
general discussion of HTTP/3.

....Roy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 14 Jul 2016 20:47:19 +0000

--------
In message <[email-masked]>, "Roy T. Fielding" w

>processed so that an appropriate action can be taken for duplicates.

There is a difference between "possible" and "allowed".

>The reason duplicate header fields exist is because email wanted them [...]

And that was probably a worthwhile concern back then.

Today email protocols should have no influence on the _future_ of HTTP.

>processed and forwarded by a recipient without knowing its definition.

That will still hold true with my proposal, with the footnote that
we do not expect intermediaries to add (copies) of headers they do
not understand the semantics of.

>zero-copy constraints to append certain fields at the end of each

None of which soundes like an obvious candidate for new structured
format headers to me ?

>insane reason want to use JSON for field processing.

I think you must have misunderstood my proposal if you think so.

My proposal is that once we're done, the RFC should contain these
three parts.

	A) Here is how to put structured data into new HTTP headers
	   (Be it JSON or something saner).

		1) HTTP/1

		2) HTTP/2

	B) A header using this format can only appear at most once
	   in any HTTP message.  Split or repeat headers *using
	   this format* are not allowed.

	C) Here is how you write up a specification for such a header
	   in an RFC.  (ABNF/Schema/something)

That does not change the semantics of any currently defined
headers, nor of any headers not using the new format - JSON or
otherwise.