Subject: structured headers "why not JSON" FAQ

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Jun 2018 09:09:57 +0200

Hmmm. It's good to have a FAQ, but it shouldn't be misleading...

> A.1. Why Not JSON?
> 
> Earlier proposals for structured headers were based upon JSON [RFC8259], but failed to get traction for a variety of reasons, including:
> 
>     Interoperability issues (e.g., regarding large numbers and objects with duplicate members)

<https://greenbytes.de/tech/webdav/draft-reschke-http-jfv-08.html#interop>

>     Difficulties specifying field values using ABNF [RFC5234]

Please elaborate. What are the difficulties, and how is this different 
in structured headers?

>     Need to encode JSON to appear in HTTP headers, thereby making it not-JSON

Not true.

>     Reluctance to embed a JSON parser in some HTTP implementations (e.g., servers and intermediaries)

Well, there was also reluctance to add yet another parser, when JSON is 
already available. Just saying.

>     Concerns about JSON’s ability to nest to arbitrary depths, and the resulting memory commitment that might involve

If this is really a problem, a simple fix would be to restrict the size 
in characters, just as in structured headers. Exactly the same problem.

>     Feelings that JSON doesn’t “look right” in HTTP headers

True, it's not pretty.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 13 Jun 2018 18:01:27 +1000

> On 13 Jun 2018, at 5:09 pm, Julian Reschke <[email-masked]> wrote:
> 

That was a first attempt in an editors' draft from memory; improvements welcome.

> <https://greenbytes.de/tech/webdav/draft-reschke-http-jfv-08.html#interop>

My recollection is that people felt that merely cautioning people not to use UTF-8 or big numbers was not sufficient to avoid interoperability problems.

> Please elaborate. What are the difficulties, and how is this different in structured headers?

How can you use ABNF to define the allowable structures in JFV? See the editors' draft of SH for an example there (a work in progress).

>>    Need to encode JSON to appear in HTTP headers, thereby making it not-JSON

I'll change this to "Need to process JSON to assure that characters are escaped in a manner that's appropriate for HTTP headers."

> Well, there was also reluctance to add yet another parser, when JSON is already available. Just saying.

Ack.

> If this is really a problem, a simple fix would be to restrict the size in characters, just as in structured headers. Exactly the same problem.

Again, specifying this as JSON is going to make developers believe that it *is* JSON, and therefore just as flexible as JSON; any limitations you put into the spec are likely to be ignored by at least some implementers, since they'll be able and very inclined to just hook up their JSON parser/serialiser to their HTTP implementation. 

>>    Feelings that JSON doesn’t “look right” in HTTP headers

It's good to find common ground.

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Jun 2018 10:33:43 +0200

On 2018-06-13 10:01, Mark Nottingham wrote:
> ...

Precisely which part?

> I'll change this to "Need to process JSON to assure that characters are escaped in a manner that's appropriate for HTTP headers."

Either that, or use a serializer where the encoding (and whitespace 
handling) can be configured.

<https://greenbytes.de/tech/webdav/draft-reschke-http-jfv-08.html#rfc.section.3>

> ...

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 13 Jun 2018 18:37:37 +1000

After some thought and offline discussion, I've taken another stab at explaining the underlying reason here -- that using JSON means that people will tend to point JSON implementations at it and violate the constraints that make it "fit" into HTTP headers.

Review appreciated.

https://httpwg.org/http-extensions/draft-ietf-httpbis-header-structure.html#why-not-json

> On 13 Jun 2018, at 6:01 pm, Mark Nottingham <[email-masked]> wrote:
> 
> 
>

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 13 Jun 2018 18:38:06 +1000

> On 13 Jun 2018, at 6:33 pm, Julian Reschke <[email-masked]> wrote:
> 

https://httpwg.org/http-extensions/draft-ietf-httpbis-header-structure.html#specify

> Either that, or use a serializer where the encoding (and whitespace handling) can be configured.
> 

See more recent revision.

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Jun 2018 10:53:28 +0200

> https://httpwg.org/http-extensions/draft-ietf-httpbis-header-structure.html#specify

Ah. There are two things here.

1) Overall header field syntax: this is not needed for JFV, because it's 
always the same, see ABNF at the end of 
<https://greenbytes.de/tech/webdav/draft-reschke-http-jfv-08.html#data-model-and-format>.

2) Constraints on specific field values - that could be done the same 
way as in structured headers.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Jun 2018 11:01:14 +0200

Here we go:

> A.1. Why Not JSON?
> 
> Earlier proposals for structured headers were based upon JSON [RFC8259]. However, constraining its use to make it suitable for HTTP header fields requires senders and recipients to implement specific additional handling.

The constraints are on the senders. *If* they follow them, recipients 
using off-the-shelf JSON parsers will be fine. (And yes, the *if* is the 
potential issue).

> Because of JSON’s broad adoption and implementation, it is difficult to impose such additional constraints across all implementations; some deployments would fail to enforce them, thereby harming interoperability.

...but interop would only be harmed for non-conformant field values.

> For example, JSON has specification issues around large numbers and objects with duplicate members. Although advice for avoiding these issues is available (e.g., [RFC7493]), it cannot be relied upon.

(see above)

> Likewise, JSON strings are by default Unicode strings, which have a number of potential interoperability issues (e.g., in comparison). Although implementers can be advised to avoid non-ASCII content where unnecessary, this is difficult to enforce.

Not sure what interop problems you are referring to. SH currently only 
has ASCII, forcing non-ASCII content (when needed) into binary values. 
That doesn't seem to be an advantage for those who actually need non-ASCII.

> Another example is JSON’s ability to nest content to arbitrary depths. Since the resulting memory commitment might be unsuitable (e.g., in embedded and other limited server deployments), it’s necessary to limit it in some fashion; however, existing JSON implementations have no such limits, and even if a limit is specified, it’s likely that some header field definition will find a need to violate it.

a) A limit could be enforced *before* feeding the field values into the 
JSON parser.

b) "and even if a limit is specified, it’s likely that some header field 
definition will find a need to violate it" - how is that different in SH?

> Since a major goal for Structured Headers is to improve interoperability and simplify implementation, these concerns led to a format that requires a dedicated parser and serialiser.
> 
> Finally, there were widely shared feelings that JSON doesn’t “look right” in HTTP headers.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 13 Jun 2018 20:01:37 +1000

> On 13 Jun 2018, at 7:01 pm, Julian Reschke <[email-masked]> wrote:
> 
>> A.1. Why Not JSON?
> The constraints are on the senders. *If* they follow them, recipients using off-the-shelf JSON parsers will be fine. (And yes, the *if* is the potential issue).

We need constraints on recipients too; otherwise their laxness will encourage some senders to become non-conformant. Good interop requires behaviour on both sides.

> ...but interop would only be harmed for non-conformant field values.

True, but that's small comfort if they become common in use; it effectively defines a new interoperability profile (see: HTML).

> Not sure what interop problems you are referring to. SH currently only has ASCII, forcing non-ASCII content (when needed) into binary values. That doesn't seem to be an advantage for those who actually need non-ASCII.

The easy availability of non-ASCII content in JSON invites its use when it isn't necessary.

> a) A limit could be enforced *before* feeding the field values into the JSON parser.
> 
> b) "and even if a limit is specified, it’s likely that some header field definition will find a need to violate it" - how is that different in SH?

In SH, if you violate the limits, you won't work with most SH implementations (provided we can get good conformance; that's why having a test suite and detailed algorithms are important) -- or at least enough to dissuade violating the limits. With JSON, there are strong incentives to allow "all of JSON" in.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Jun 2018 12:12:43 +0200

> The easy availability of non-ASCII content in JSON invites its use when it isn't necessary.

So what are the interop problems you were referring to?

> In SH, if you violate the limits, you won't work with most SH implementations (provided we can get good conformance; that's why having a test suite and detailed algorithms are important) -- or at l…

I would say that the incentives are exactly the same for the two 
variants. If the spec makes a mandatory requirement on the recipient 
that is easy to implement, why would implementations choose to follow it 
for SH but not for JFV?

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 13 Jun 2018 20:16:27 +1000

> On 13 Jun 2018, at 8:12 pm, Julian Reschke <[email-masked]> wrote:
> 

String normalisation, comparison, homographs, etc.

> I would say that the incentives are exactly the same for the two variants. If the spec makes a mandatory requirement on the recipient that is easy to implement, why would implementations choose to follow it for SH but not for JFV?

Because the option of deploying a JSON implementation *without* those constraints is extremely easy.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 13 Jun 2018 10:17:47 +0000

--------

>that is easy to implement, why would implementations choose to follow it 

Because with SH somebody underqualified won't stop reading at JSON and
think he knows how this song goes and thus miss all the restrictions ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Jun 2018 13:04:11 +0200

> String normalisation, comparison, homographs, etc.

Well, yes. I18N makes things harder. I just don't buy the argument that 
it's a *downside* of JFV over SH. Either you need non-ASCII, in which 
case JFV is clearly better than SH. Or you don't, and then it doesn't 
matter.

> Because the option of deploying a JSON implementation *without* those constraints is extremely easy.

I don't buy that argument.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Jun 2018 13:05:25 +0200

> think he knows how this song goes and thus miss all the restrictions ?

In which case it won't work anyway, because you need to do 
<https://greenbytes.de/tech/webdav/draft-reschke-http-jfv-08.html#rfc.section.4> 
to get it working at all.

Best regards, Julian

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 13 Jun 2018 09:52:45 -0600

On 06/13/2018 05:04 AM, Julian Reschke wrote:
>>> On 13 Jun 2018, at 8:12 pm, Julian Reschke <[email-masked]>
>>> that is easy to implement, why would implementations choose to follow
>>> it for SH but not for JFV?

> I don't buy that argument.

That argument is valid IMO. If I am writing a simple piece of code that
needs to deal with a structured header, I am going to use an existing
parsing/packing library.

* In JFV case, the probability of choosing a JSON library without
restrictions is non-negligible (because there are lots of such
libraries, many are popular/built-in, and I may be using one of them
already).

* The probability of choosing an SH library without restrictions is
about zero (because such libraries will not exist).

Alex.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 13 Jun 2018 23:45:03 +0200

Hi Julian,

On Wed, Jun 13, 2018 at 01:04:11PM +0200, Julian Reschke wrote:
> > > If the spec makes a mandatory requirement on the recipient that is easy to implement, why would implementations choose to follow it for SH but not for JFV?
> > 

Speaking as an implementer of intermediary, it's a huge pain to have to deal
with non-compliance in field. The intermediary always appears after clients
and servers were already working together, and reveals non-compliance by
breaking what used to work. And the users in place don't want to listen to
your arguments nor read the RFC you put under their noze, they simply say
"but it was working fine, it's stupid to be extremist like this". And it's
very likely getting worse with microservices because lots of server-side
components are themselves clients as well, so the client and the server
code are developed by the same person who refuses to read specs and thinks
he/she knows. To give you an example, I've already seen people send base64
in headers with the LF at the end of each line. It used to work only because
of line folding and tolerance for single LF. With haproxy in the middle
replacing the LF with an LWS, it broke. The guy told me "but there's no
problem doing this, the proof is that it works". The reality is that this
person had no clue how to encode his PEM certificate and figured that simply
sending it into a header field was enough to retrieve it unmollested on the
other side.

As we already discussed, while I don't like JSON much I have no strong feeling
against it either and your JFV proposal certainly remains a good one overall.
But we really cannot dismiss this risk of reusing libraries whose scope was
clearly not to be placed into headers nor to respect the imposed constraints
because 99.9% of the time it will work, and some of us will have to deal with
the 0.1% remaining once it's too late to fix the end points :-/

Cheers,
Willy