Subject: Re: [Json] Consensus on JSON-text (WAS: JSON: remove gap between Ecma-404 and IETF draft)

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 28 Nov 2013 09:37:32 -0800

On Nov 27, 2013, at 7:29 PM, Paul Hoffman wrote:

> <no hat>
> 
> On Nov 27, 2013, at 5:00 PM, Alex Russell <[email-masked]> wrote:
> 
>> Will you also be citing ECMA-404 normatively to avoid this sort of divergence in the future?
> 
> If you believe that ECMA-404 will change in the future, that would indicate that ECMA might break interoperability with current implementations, even for what they perceive as "good reasons". In general, the IETF tries not to have its long-lived standards normatively latch on to moving targets for this very reason. Even when other SDOs have assured us that they will not make backwards-incompatible changes, they have done so anyway (cue the Klensin-esqe theme music...), and that has caused serious interoperability problems for the IETF specs.

Stability of ECMA-404 should not be a concern. As far as I can observe TC39 has absolutely no interest in every changing or extending the JSON grammar. I'm confident TC39 would record that as a statement of policy if asked. In fact, the reason TC39 chose to issue ECMA-404 was because there was concern that the JSON WG was on a path to modify and possibly extend JSON syntax. The only sort of changes to ECAM-404 I ever expect to see would be technical corrections to errors found in the current specification language or the addition of informative material to help clarify understanding of the specification.  For example, if there was sufficient public interest we might consider adding an informative Annex that restates the grammar using ABNF notation.

The JSON syntax has its roots in the ECMAScript syntax but is not at all linked to the ECMAScript syntax.  The ECMAScript object and array literal constructs were the inspiration for JSON. But even in the beginning, the JSON syntax was only a subset set of the full syntax of those ECMAScript language features.  ECMAScript object literal syntax has been significantly extended in both the ECMA-262 5th Edition(2009) and the forthcoming 6th Edition.  But those extensions have and will not ever be added to the JSON syntax defined in ECMA-404.

Allen Wirfs-Brock
ECMA-262 Project Editor

---

From: John Cowan <[email-masked]>
Date: Wed, 27 Nov 2013 22:49:41 -0500

Paul Hoffman scripsit:

> If you believe that ECMA-404 will change in the future, that
> would indicate that ECMA might break interoperability with current
> implementations, even for what they perceive as "good reasons". In
> general, the IETF tries not to have its long-lived standards normatively
> latch on to moving targets for this very reason. Even when other SDOs
> have assured us that they will not make backwards-incompatible changes,
> they have done so anyway (cue the Klensin-esqe theme music...), and
> that has caused serious interoperability problems for the IETF specs.

Binding specifically to the first edition of ECMA-404 would resolve
that problem.  Binding to a specific edition doesn't work so well with
Unicode, but that is because Unicode is a special case: it expands its
coverage in successive editions to ever-larger portions of the world
of writing systems.  It would have been impossible to do the whole
thing at once, unlike even the largest ordinary technical standard.
ECMA-404 is not in that league.

-- 
Not to perambulate                 John Cowan <[email-masked]>
    the corridors                  http://www.ccil.org/~cowan
during the hours of repose
    in the boots of ascension.       --Sign in Austrian ski-resort hotel

---

From: Noah Mendelsohn <[email-masked]>
Date: Sat, 30 Nov 2013 13:57:14 -0500

On 11/27/2013 11:51 PM, Tim Berners-Lee wrote:

> JSON is interesting in being a subset of ECMAscript.  That is a big
> dependency -- will it be preserved?

> However as it is unwise to feed JSON into an ECMAscript processor for
> security reasons, that dependency may not affect code, just mean that
> JSON and ECMAscript parsers can share parts at  the moment.

There may be many other situations in which there is benefit to having JSON 
be a proper subset of ECMAscript. For example, one can imagine tool chains 
used for development or testing that would depend on the ability to copy 
text fragments between JSON and ECMAscript source documents, and one might 
foresee more "vertical" standards that would depend on the commonality as 
well.

Noah

---

From: Alex Russell <[email-masked]>
Date: Wed, 27 Nov 2013 17:00:30 -0800

Will you also be citing ECMA-404 normatively to avoid this sort of
divergence in the future?

On Wed, Nov 27, 2013 at 4:13 PM, Tim Bray <[email-masked]> wrote:

> To do this, I think the draft requires these changes:
>
> - Remove the trailing section of section 1.2, starting with “ECMAscript
> 5.1 enumerates...” [because the difference no longer exists]
>
> - In section 2:
>
> -- remove “A JSON text is a serialized object or array.”
>
> -- Insert: “A JSON text is a serialized value.  Note that certain previous
> specifications of JSON constrained a JSON text to be an object or an array.
>  Implementations which generate only objects or arrays where a JSON text is
> called for will be interoperable in the sense that all implementations will
> accept these as conforming JSON texts.”
>
> -- Change the JSON-text production to read:
>
> JSON-text  = value
>
>
>
>
>
>
> On Fri, Nov 22, 2013 at 10:21 AM, Matt Miller (mamille2) <
> [email-masked]> wrote:
>
>> There appears to be consensus to change JSON-text to allow for any JSON
>> value -- not just object / array -- while noting that object or array as
>> the top-level is the most interoperable.
>>
>> We will ask the Document Editor to make this change to
>> draft-ietf-json-rfc4627bis.
>>
>>
>> - Paul Hoffman and Matt Miller
>>
>>
>> _______________________________________________
>> json mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/json
>>
>>
>

---

From: Tim Berners-Lee <[email-masked]>
Date: Wed, 27 Nov 2013 23:51:29 -0500

On 2013-11 -27, at 23:00, Tim Bray wrote:

> I listed some arguments against this in http://lists.w3.org/Archives/Public/www-tag/2013Oct/0041.html and at the moment I still believe them. Is there new information?
> 
> On top of that, I have no fear of anyone trying to change JSON in the future; they would be resoundingly ignored by the community of implementers.  I speak as one who would love to add built-in date/time literals but know that it won’t happen.  -T
> 

JSON is interesting in being a subset of ECMAscript.  That is a big dependency -- will it be preserved?
However as it is unwise to feed JSON into an ECMAscript processor for security reasons, that dependency may not affect code, just mean that JSON and ECMAscript parsers can share parts at  the moment.

One could imagine that the arc of ECMAscript's evolution could end up 
having all kinds of impact on the data structure syntax and semantics.
(unordered sets as alternative to lists? who knows).  So in that case one
could imagine pressure to make a new version of JSON to match.

Yes, literal ISO dates and dateTimes -- I added them to my own N3/turtle parsers without much fanfare, wish they had been put in the Turtle language too.  Maybe they will. 

-timbl

> On Wed, Nov 27, 2013 at 5:00 PM, Alex Russell <[email-masked]> wrote:
> - Remove the trailing section of section 1.2, starting with “ECMAscript 5.1 enumerates...” [because the difference no longer exists]
> 
> - In section 2:
> 
> -- Insert: “A JSON text is a serialized value.  Note that certain previous specifications of JSON constrained a JSON text to be an object or an array.  Implementations which generate only objects or arrays where a JSON text is called for will be interoperable in the sense that all implementations will accept these as conforming JSON texts.”
> 
> On Fri, Nov 22, 2013 at 10:21 AM, Matt Miller (mamille2) <[email-masked]> wrote:
> There appears to be consensus to change JSON-text to allow for any JSON value -- not just object / array -- while noting that object or array as the top-level is the most interoperable.
> 
> We will ask the Document Editor to make this change to draft-ietf-json-rfc4627bis.
> 
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 2 Dec 2013 16:30:09 -0500

On Wed, Nov 27, 2013 at 11:51 PM, Tim Berners-Lee <[email-masked]> wrote:

> security reasons, that dependency may not affect code, just mean that JSON
> and ECMAscript parsers can share parts at  the moment.
>

As I see it, encoding X is a subset of encoding Y if and only if an encoder
for X will only produce outputs that are valid inputs of encoding Y.

If an issue was to occur it would be because encoding Y has changed or the
definition of Y has changed.

> could imagine pressure to make a new version of JSON to match.

Since we are talking about a serialization format, the distinction between
unordered sets and lists cannot occur at the wire level and this is where
we need interoperation.

I do in fact have a schema compiler for JSON that allows an interface to
specify a set of entries rather than a list. But they are only ever
serialized as lists.

Yes, literal ISO dates and dateTimes -- I added them to my own N3/turtle
> parsers without much fanfare, wish they had been put in the Turtle language
> too.  Maybe they will.
>

And you probably do exactly what I do and represent a DateTime
representation as a subset of String just as byte, int16, int32, int64,
uint* are all subsets of Integer.

One of the things I think we have learned from JSON is that a
self-describing format only needs to specify the abstract type of the datum
and not the representation.

For convenience, I allow a schema to specify the size of an integer and
whether it is signed or unsigned so that the code generator can create
appropriate code bindings. But none of that shows up on the wire, nor is
that particularly helpful.

What we are doing at this point is to fix a version of the JSON encoding in
time so that when a client and server are negotiating use of JSON encoding,
both sides know what is being negotiated.

So hopefully JSON does not change in future, only the description of JSON.

That is not the same as saying that JSON meets all possible current and
future protocol needs. It does not. It is not possible to pass binary data
efficiently for a start and using decimal for floating point representation
is likely to make the format unacceptable for serious data applications
since it introduces conversion errors.

The next question is whether those unmet needs should be addressed by an
entirely new encoding with a completely new syntax and structure or whether
we could extend the JSON model. My view is that we should do the second.

XML is fine for documents but it is not meeting my needs as a protocol
designer. I find XML Schema to be unnecessarily confusing and baroque, the
schema validation features supported don't help me in application
protocols. XML does not support binary encoding of cryptographic data or
floating point.

There are many data encodings on offer but I would like to be able to write
one decoder that can consume a data stream that contains basic JSON data
and JSON with extensions. This makes negotiating an encoding in a Web
service easy, the consumer states which encodings are acceptable and the
sender makes sure what is sent is compatible, downgrading the encoding to
the level accepted if necessary.

-- 
Website: http://hallambaker.com/

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 2 Dec 2013 18:58:30 -0500

On Mon, Dec 2, 2013 at 5:15 PM, Ashok Malhotra <[email-masked]>wrote:

>
> On 12/2/2013 4:30 PM, Phillip Hallam-Baker wrote:
>
>> XML does not support binary encoding of cryptographic data or floating
>> point.
>>
> XML Schema supports hexBinary and base64binary datatypes.
> It also supports float and double datatypes.
>
> All the best, Ashok
>

Those are text encodings of binary data. They are not a binary encoding.
XML is a text encoding, not a binary encoding.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 2 Dec 2013 19:07:30 -0500

On Mon, Dec 2, 2013 at 5:48 PM, Nico Williams <[email-masked]> wrote:

> On Mon, Dec 02, 2013 at 04:30:09PM -0500, Phillip Hallam-Baker wrote:
> But it can be part of the on-the-wire description.  See below.
>
> Self-description is a continuum.  Some ASN.1 encoding rules can encode
> quite a bit of a schema on the wire -- clearly there's a point at which
> the resulting redundancy causes problems.  But it's also true that
> having a large subset of the schema in the serialization can be useful
> (e.g., for generic "dump" tools).
>
> Given the prevalence of languages like Python, a "set" type will no
> doubt seem useful to some!  Heck, the ability to use non-string values
> as keys (names) for objects would be nice too -- anyone who's spent much
> time with Python and JSON has wished for these things.  JSON alone is
> insufficiently expressive for "pickling" Python values; JSON with a fair
> bit of convention layered on gets closer to being good enough for
> pickling Python values.
>

Well that is a rather different situation, the language allows for
polymorphic typing and does not have a clue what the native data type of
the entries it exchanges is.

I agree that there is a utility in that information for pickling data
values but it is not an approach I would like to see in an application
protocol.

The way I would approach this if necessary would be to add in a tag that
could be used to annotate the data with additional type hints.

---

From: Ashok Malhotra <[email-masked]>
Date: Mon, 02 Dec 2013 17:15:12 -0500

On 12/2/2013 4:30 PM, Phillip Hallam-Baker wrote:
> XML does not support binary encoding of cryptographic data or floating point. 

All the best, Ashok

---

From: John Cowan <[email-masked]>
Date: Mon, 2 Dec 2013 19:24:52 -0500

Nico Williams scripsit:

> "datetime" and such are interpretations of more basic datatypes, 

An interval of time is not a string, any more than a number is a string.
They are both *representable* by strings, but everything is: you can
represent a human being or the planet Saturn by a string.

> they belong in pre-agreed/documented schema rather than on the wire.

The decision to do so is arbitrary.

-- 
John Cowan   <[email-masked]>   http://www.ccil.org/~cowan
One time I called in to the central system and started working on a big
thick 'sed' and 'awk' heavy duty data bashing script.  One of the geologists
came by, looked over my shoulder and said 'Oh, that happens to me too.
Try hanging up and phoning in again.'  --Beverly Erlebacher

---

From: John Cowan <[email-masked]>
Date: Mon, 2 Dec 2013 22:53:57 -0500

Nico Williams scripsit:

> Interval definitely sounds like a tuple, but, sure, JSON texts can
> represent arrays as text...

I was thinking about intervals like 2013-12.

-- 
The Unicode Standard does not encode            John Cowan
idiosyncratic, personal, novel, or private      http://www.ccil.org/~cowan
use characters, nor does it encode logos
or graphics.                                    [email-masked]

---

From: Nico Williams <[email-masked]>
Date: Mon, 2 Dec 2013 16:48:06 -0600

Context will affect how much of the schema you or I will find desirable
to see appear redundantly on the wire.  But mostly I agree with you:
"datetime" and such are interpretations of more basic datatypes, and so
they belong in pre-agreed/documented schema rather than on the wire.
Indeed, datetime/timestamp could be either strings or numeric, and still
be understood correctly in context.

> the level accepted if necessary.

Yes.  Though, really, the main thing that's missing is chunked
(indefinite-length) unescaped binary data.  That's the thing that's
difficult/expenside to deal with in JSON (or XML, for that matter).
Bulk data transfers do matter.  (And if one is going to add that to any
serialization, then plain binary-coded integers and IEEE754 doubles may
be much better, perf-wise, than decimal encodings.)

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Mon, 2 Dec 2013 20:23:06 -0600

> The decision to do so is arbitrary.

Of course it is.  How much to describe on the wire vs. schema is... a
continuum.  Given the JSON we have though... it's best to deal with
datetime via schema.

My point was and still is that the one thing that's sorely missing is
an indefinite-length unescaped binary data encoding, and which is not
nearly as complex, notionally, IMO, as other types.

Nico
--

---

From: Alex Russell <[email-masked]>
Date: Mon, 2 Dec 2013 17:26:17 -0800

On Mon, Dec 2, 2013 at 1:30 PM, Phillip Hallam-Baker <[email-masked]>wrote:

> As I see it, encoding X is a subset of encoding Y if and only if an
> encoder for X will only produce outputs that are valid inputs of encoding Y.
>
> What we are doing at this point is to fix a version of the JSON encoding
> in time so that when a client and server are negotiating use of JSON
> encoding, both sides know what is being negotiated.
>

This seems entirely reasonable, so long as this extension is not called
"JSON" (assuming it adds new grammar productions). Hence the request to
normatively cite ECMA-404 and re-name any IETF-produced spec to reflect
that it is intended to be a media-type registration (or extension document
for protocol authoring, etc.), not an alternate (and therefore perhaps
extensible) definition of JSON.

> There are many data encodings on offer but I would like to be able to
> write one decoder that can consume a data stream that contains basic JSON
> data and JSON with extensions. This makes negotiating an encoding in a Web