Subject: Precision of numbers using JSON Header Field Values

From: Kazuho Oku <[email-masked]>
Date: Fri, 15 Jul 2016 13:13:31 +0900

Hello,

I have a question about how numbers are going to be handled using JSON
Header Field Values.

In section 6.1, the draft uses Content-Length header as an example,
and states that in case the header is to be represented using JSON,
the definition should "restrict all numbers to be non-negative
integers without fractions."

Does this mean that numbers should not be expressed using the
exponential notation (which would mean that we would be adding a
restriction to JSON encoder / parser), or does it mean that we should
validate the number _after_ parsing it using method defined in RFC
7159?

The reason I ask is because RFC 7159 does not define the precision of
numbers, and in some cases existing JSON decoders disagree in how the
number expressions are parsed.

For example, using node.js v.6.2.1,
0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100e400
is evaluated as 100. OTOH, using ruby 2.0.0, the same expression is
interpreted as zero.
(please refer to
https://gist.github.com/kazuho/7e72372111fa0655692d9e44be70c7ea for
the full output)

IMO such disagreement is a mine of vulnerabilities.  For example in
case of HTTP/1.1, it could lead to HTTP response splitting.

So personally, I think we should forbid the use of exponential
notation at least in some cases, but I wonder if adding such
restriction is intended or aligns with the motive to use an existing
notation for HTTP headers.

-- 
Kazuho Oku

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 15 Jul 2016 06:37:00 +0200

Hi Kazuho,

On Fri, Jul 15, 2016 at 01:13:31PM +0900, Kazuho Oku wrote:
(...)

Nice!

> notation for HTTP headers.

Do we have a method to suggest to decoders to reject this ? I mean,
we can say it's forbidden, but if recipients systematically pass the
whole string through a non-configurable decoder, the vulnerability
continues to be deployed. That's especially true for all "MUST NOT"
which tend to be considered as granted by senders with no need to
check on the recipient unfortunately :-/

Also are we sure senders will never emit exponents without being
explicitly instructed to do so ? I'm asking because I wouldn't
like to see for example, "Content-length: 1000000500" being
encoded as "Content-length: 1e09" and then decoded as
"Content-length: 1000000512" (for example) because it's parsed
as a float by one implementation which can only use single
precision numbers.

Willy

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 15 Jul 2016 14:01:41 +0900

Hi,

2016-07-15 13:37 GMT+09:00 Willy Tarreau <[email-masked]>:
> Do we have a method to suggest to decoders to reject this ?

Some decoders might have such option, but my understanding is that
such option is not available at least in JavaScript (i.e. ECMAScript
5th edition).

>  I mean,

It might be fair to expect that encoders would be possible to emit
integers up to 2**53-1 precisely (as discussed in section 6 of RFC
7159).

-- 
Kazuho Oku

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 15 Jul 2016 07:07:43 +0000

--------
In message <CANatvzwDMF50KcFGR72SUtLCJc6fqJJ2pZaCOK9qv2RK5vZ+=[email-masked]>
, Kazuho Oku writes:

>0.0000000000000000000000000000000000000000000000000000000000000000000000000=
>000000000000000000000000000000000000000000000000000000000000000000000000000=
>000000000000000000000000000000000000000000000000000000000000000000000000000=
>000000000000000000000000000000000000000000000000000000000000000000000000000=
>000000000000000000000000000000000000000000000000000000000000000000000000000=
>000000000000000000000000100e400

Seriously, the problem here is not the implementations trying to
make sense of this number, as much as the fact that it is legal
to transmit it that way.

Given that HTTP is a public access interface, we really should not
allow that kind of inane flexibility.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 15 Jul 2016 09:10:12 +0200

On 2016-07-15 06:13, Kazuho Oku wrote:
> ...

That's a good point, and (as the uniqueness issue) a general weakness of 
JSON.

We could:

1) Mandate that all numeric values need to be transferred as strings 
(loosing some of the benefits), or

2) Require use of I-JSON (https://tools.ietf.org/html/rfc7493) 
generators/parsers (loosing lots of potential implementations).

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 15 Jul 2016 07:22:27 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>generators/parsers (loosing lots of potential implementations).

I-JSON is probably the better of those two alternatives, despite
the fact that the restrictions it mandates can be very hard to
implement correctly and efficiently (in particular the unicode
restrictions).

The 3. alternative which you didn't list gets my vote:

3) Don't use JSON.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 15 Jul 2016 09:28:30 +0200

> The 3. alternative which you didn't list gets my vote:

I do not disagree with that. That's certainly an option. In that case 
we'd be discussing a textual format that can be shared across header 
fields, so that people at least can use a single parser.

Best regards, Julian

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 15 Jul 2016 16:47:58 +0900

Hello,

Thank you for the response.

> (loosing some of the benefits), or

I'm afraid that would not be a practical option.

In order to prevent security issues, we would need to specify that
decoders MUST reject JSON numbers.

However, I assume it would be difficult to implement such a
requirement in certain programming languages that do not provide a
well-defined way to distinguish between a numeric type and a string
type (e.g. Perl, PHP).

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 15 Jul 2016 18:14:13 +0900

> For example, using node.js v.6.2.1,

FYI the ruby's behavior has been fixed (see
https://github.com/ruby/ruby/commit/5292b2727a089821f2bf21dfcc8a7be899eb744c),
but anyways I agree with PHK and Julian that this illustrates the
danger of using a loose specification such as JSON.

-- 
Kazuho Oku

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 16 Jul 2016 07:09:20 +0200

Hi,

> integers up to 2**53-1 precisely (as discussed in section 6 of RFC

OK but if transmitted with a decimal point like in your example above,
are we sure a decoder will still consider that it is an int ? For
example, if I craft "Content-length: 1000000500.99999994" and parse it
using atof(), I'm seeing that printf("%f") emits "1000000501.000000"
while converted to an int it becomes "1000000500". This typically means
that an implementation could use a different value from what it is
passing if it re-encodes it. Or maybe we should explicitly state that
certain header fields are integers and it's the recipient's responsibility
to ensure there's no fractional part. Content-length definitely is one
such sensitive field.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 16 Jul 2016 07:12:53 +0200

On Fri, Jul 15, 2016 at 09:28:30AM +0200, Julian Reschke wrote:
> I do not disagree with that. That's certainly an option. In that case we'd
> be discussing a textual format that can be shared across header fields, so
> that people at least can use a single parser.

While I'm not fond of JSON I think that *if it satisfies our needs* it
provides an overall rich set of possibilities and is widely supported. But
if we're seeing corner cases which require parsers to perform some difficult
verifications maybe we'll cause less trouble by proposing a very specific
and strict language for which parsers will eventually be implemented anyway.

I must say that right now I'm worried with the lack of distinction between
an integer and a float. Some fields like a content-length or a byte-range
definitely require a strict integer.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 16 Jul 2016 10:01:05 +0000

--------

>definitely require a strict integer.

[1]

Well, that is true, but *in principle* there is nothing wrong with
transmitting it as "XXXXXX.000000" or even "1.2345e4", as long as
the receiver checks that the result is an integer.

And it's not like that is an issue JSON creates, we already perform
that very check today, but it is bundled up in a much stronger and
general check:  The field can only contain digits.

The trouble is the "we expect people to use generic JSON
parsers" assumption being tacitly made here.

Some fraction of generic JSON parsers store numbers as numbers in
their internal data structure[2].

If you use one of those, an hostile sender can force you to do
floating point math on any and all numeric fields.

We can of course write "If you use a generic JSON parser, make sure
it doesn't store numbers as numbers" but that makes the word "generic"
surplusage, doesn't it ?

We need to think about this problem at the meta level:

1) How are the content of these "structured headers" described in RFCs?

2) How does that description get turned into code which validates
   an input candidate ?

I belive answering that with "1: English Prose, 2: Programmers do
that" will cause far too much security vulnerabilities and interop
problems[3]

We _have_ to find a better answer than that.

Once we have I suspect the on-wire format will follow easily from that.

Poul-Henning

[1] Should array indices start at 0 or 1? My compromise of 0.5 was
    rejected without, I thought, proper consideration.
    	-- Stan Kelly-Bootle

[2] I-JSON imposes a 53 bit restiction on integers, to make it safe
    to store them in IEEE 754 binary64 floating point values.

[3] In exactly the same way "We'll just use JSON" has everywhere
    else.  I guess we'd "be following industry practice".

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 17 Jul 2016 07:40:57 +0200

> the receiver checks that the result is an integer.

Then we'll have to suggest how to do it safely. Something like the
following depending on the language and expected ranges and which
however does not cover the lack of precision if bits are missing
in the mantissa to properly represent the int :

     if ((double)value != (double)(uint64_t)value)
          fail();

Eg, a few tests :

  input=1.99999994 float=2.000000 int=1 equal?=0
  input=1111111111999999 float=1111111111999999.000000 int=1111111111999999 equal?=1
  input=11111111119999999 float=11111111120000000.000000 int=11111111120000000 equal?=1
  input=111111111199999991 float=111111111199999984.000000 int=111111111199999984 equal?=1

> parsers" assumption being tacitly made here.

Absolutely. The difference here is that users totally trust the libs
they're relying on without knowing their limits, and generally have
no easy access to the original contents to verify its format.

> floating point math on any and all numeric fields.

That's my fear.

> it doesn't store numbers as numbers" but that makes the word "generic"

We know we can very easily cause users to shoot themselves in the foot
by lack of knowledge about corner cases, so instead of saying "don't
use this or that", we need instead to help them verify that the
underlying framework didn't lie to them. That makes their choice of
components easier and safer (especially in large teams where implementors
do not chose the lower layer components which have been imposed by others).

> 2) How does that description get turned into code which validates

or 3) do we want to limit authorized values to the subset known to
safely pass through various implementations ? That's about similar to
the analysis work we had to do for 7230 after all. For example, stating
that any integer value cannot be larger than (2^53)-1 (about 9.10^15)
could be fine. It means that a content-length or byte-ranges larger
than 8 PiB will have to be rejected.

> Once we have I suspect the on-wire format will follow easily from that.

I think that's exactly what Julian has been doing over the last years
and what led him to conclude that JSON was not a bad idea in the end.
It might appear as a bad solution but the least bad of all other ones.

But if we start to define some small variations or impose restrictions
on how JSON parsers have to work, maybe we'll take less time to define
exactly what we want to see there and how it should behave regarding
some corner case behaviours (eg: how to handle duplicate keys). My
guess is that it will look very similar to JSON with possibly very
small variations and that modifying a few JSON implementations to
respect this format would not be that hard.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 17 Jul 2016 07:14:22 +0000

--------
In message <[email-masked]>, Willy Tarreau writes:

>It [JSON] might appear as a bad solution but the least bad of all other ones.

The first half I can certainly agree with.

But the second half is far from obvious to me, and I nowhere have I
seen anything like a survey or analysis saying so.

For me it is a question of consequences.

I don't want us to be stuck with mandatory JSON if/when we attempt
a HTTP/3 protocol, unless we already now address and somehow neuter/
manage the worst of the trouble JSON comes with[1].

If we limit JSON to only end-to-end headers, H3 will only need to
be able to move them, and never have to parse them.

But if we allow JSON to creep into 'transport headers', as Julians
"Accept:" example seems to both imply and promise, we will be stuck
with JSON *forever*.

Do you feel like parsing JSON in a 100Gbit/sec load-balancer ?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 17 Jul 2016 07:18:22 +0000

So how does it work when a JSON formatted header is listed in Vary ?

1) Another User-Agent explosion, "Don't Do That Then".

2) We specify mandatory rules for normalized JSON, further restricting
   which "generic JSON libraries" people can use.

3) We specify how proxies must normalize JSON for Vary processing.

What's the plan ?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 17 Jul 2016 07:26:35 +0000

--------

>manage the worst of the trouble JSON comes with[1].

Sorry, footnote 1 (JSON/Vary) became a separate email, but I forgot
to remove the marker.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 17 Jul 2016 09:43:15 +0200

> 3) We specify how proxies must normalize JSON for Vary processing.

There's no plan yet, but it's an interesting point. If the cache *knows* 
that something is JSON it could certainly do more if it wants.

There's also a potential relation with the Key spec...

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 18 Jul 2016 14:14:43 +0200

Hi Poul-Henning,

> Do you feel like parsing JSON in a 100Gbit/sec load-balancer ?

Well, as I mentionned at the workshop in last July, I don't like JSON
but I have no technical argument against it and just for this I want to
give it some consideration without any form of bias. This is what is
being done in this thread and in Julian's initial draft. If at the end
it happens it's really not suited, then we'll have a lot of elements
to know why it is not and what needs to be avoided in the solution we
design as a replacement. For now, I'm seeing some difficulties with
numbers representation and with the ordering of duplicate fields. It is
possible that addressing them will come with a high cost or that some
smart people will propose very simple solutions, I don't know.

What is sure is that I don't feel like manually dealing with the tricks
of this thread on a 100Gbps LB, but I'm much more scared by the idea that
these tricks will simply be overlooked at many other places and result
in security vulnerabilities or at least poor interoperability. I'd
strongly prefer an efficient binary encoding that would pass well over
H2 / H3 whatever, but still we know that H1 will exist for many years,
that applications deal with headers as text and we'll need a text-based
representation at some points along the chain. Maybe we'll end up with
something "looking like" JSON for text versions, will be easy to serialize
to efficient and safe binary for more recent protocols etc... I don't know
but the discussion should help us spot the unfixable problems if any. And
given the number of degrees of freedom of the current HTTP headers grammar,
I'm inclined to think that only two reported issues for now is a low score
to attack this proposal.

Regards,
Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 18 Jul 2016 12:23:36 +0000

--------
In message <[email-masked]>, Willy Tarreau writes:

>I'd strongly prefer an efficient binary encoding that would pass well over
>representation at some points along the chain.

I 100% agree and that is why I think our focus should be on finding
the binary encoding.

We have well proven methods for turning binary into ascii, but the
other way becomes a nightmare every single time.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Jul 2016 14:28:00 +0200

> other way becomes a nightmare every single time.

That's a fine thing to do, but how would it help for HTTP/2?

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 18 Jul 2016 13:00:38 +0000

--------
In message <[email-masked]>, Julian Reschke writes
>That's a fine thing to do, but how would it help for HTTP/2?

We left a lot of low hanging fruit on the tree with HPACK, notably
the Date header.

If we decide to do structured headers, they might be another good
reason for HPACK2, which could then know about these fields and
do the smart thing.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Jul 2016 16:05:44 +0200

> reason for HPACK2, which could then know about these fields and

Agreed, but changing HPACK implies revising HTTP/2, thus HTTP/3, right??

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 18 Jul 2016 14:38:00 +0000

--------
In message <[email-masked]>, Julian Reschke writes
>Agreed, but changing HPACK implies revising HTTP/2, thus HTTP/3, right??

Maybe, but for HTTP/3 we should have even higher ambitions, so maybe
a better HPACK for H2 is a good partial goal.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Jul 2016 16:51:26 +0200

> a better HPACK for H2 is a good partial goal.

AFAIU, there can not be such as thing as a "better HPACK" for HTTP/2, 
because we can't revise HPACK without also revising HTTP/2.

Best regards, Julian

---

From: Patrick McManus <[email-masked]>
Date: Mon, 18 Jul 2016 17:28:20 +0200

every h2 extension bilaterally negotiated via settings is a revision to h2.
hpack can do that right?

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Jul 2016 18:05:23 +0200

On 2016-07-18 17:28, Patrick McManus wrote:
> every h2 extension bilaterally negotiated via settings is a revision to
> h2. hpack can do that right?
>
> ...

I guess so. Thanks for clarifying!

Best regards, Julian

---

From: Mike Bishop <[email-masked]>
Date: Mon, 18 Jul 2016 16:13:00 +0000

Yes, but you can’t get it in the first RTT unless it’s implied by the ALPN token.  Depending on the scope of the changes, you could enable it with SETTINGS, though, yes.  (Additional operations allowed => good; changes to existing values in header table => bad)

From: Patrick McManus [mailto:[email-masked]]
Sent: Monday, July 18, 2016 5:28 PM
To: Julian Reschke <[email-masked]>
Cc: Poul-Henning Kamp <[email-masked]>; Willy Tarreau <[email-masked]>; Kazuho Oku <[email-masked]>; HTTP Working Group <[email-masked]>
Subject: Re: Precision of numbers using JSON Header Field Values

every h2 extension bilaterally negotiated via settings is a revision to h2. hpack can do that right?

On Mon, Jul 18, 2016 at 4:51 PM, Julian Reschke <[email-masked]<mailto:[email-masked]>> wrote:
On 2016-07-18 16:38, Poul-Henning Kamp wrote:
--------
In message <[email-masked]<mailto:[email-masked]>>, Julian Reschke writes
:
On 2016-07-18 15:00, Poul-Henning Kamp wrote:
...
That's a fine thing to do, but how would it help for HTTP/2?

AFAIU, there can not be such as thing as a "better HPACK" for HTTP/2, because we can't revise HPACK without also revising HTTP/2.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 18 Jul 2016 17:05:23 +0000

--------
In message <[email-masked]>, Julian Reschke writes

>because we can't revise HPACK without also revising HTTP/2.

That obviously depends on what you mean "revising HTTP/2"...

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 19 Jul 2016 09:39:23 +1000

> because we can't revise HPACK without also revising HTTP/2.

I may be misremembering, but weren't a few "difficult" issues (like
efficient binary headers) kept out of H2 precisely because H3 could
be, relatively speaking, just around the corner? H2 has blazed a
trail; people are coming on board with the idea of new web protocols,
ALPN is a demonstrable success, the idea of 'evergreen servers' seems
to be emerging. This all says to me that the jump from H2 to H3 should
be much smoother than 1.x to 2 (and that hasn't exactly been rough.)

Also, since we dropped the Minor protocol version number, doesn't that
suggest that the step from H2 to H3 could be quite.. well.. minor? In
fact, the smaller the better -- we're still learning how to
rearchitect our sites for H2's streaming model; if we keep that model
but improve some background properties there's a gain to be had by
upgrading, with no hidden cost.

No need to reach for the stars, every improvement is an improvement
(if it's incremental *enough*.)

> Best regards, Julian

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 19 Jul 2016 09:52:14 +1000

Sorry for replying to myself, I forgot to say...

> (if it's incremental *enough*.)

Back on topic, all that is to say that this is completely irrelevant
to 1.x, and probably irrelevant to 2.

I don't think JSON is the best fit for structured headers, but
something like it could work. If it's ASCII it will already work ok
with HPACK's Huffman coding. If we target something that fits with a
pure binary H3 format then all the better.

Cheers