Subject: JSON: remove gap between Ecma-404 and IETF draft

From: Anne van Kesteren <[email-masked]>
Date: Tue, 12 Nov 2013 15:01:30 +0800

To improve JSON interoperability the IETF should not define a more
restricted version of JSON than defined by Ecma-404.

Parsers exist that can parse "42" today and parsers that cannot parse
"42" today can be meaningfully upgraded to do so too. This would not
break those parsers, unless they depend on parsing 42 as an error,
which is a far more unlikely scenario than parsing it as 42 given
precedence.

(Worth pondering about: what to do about a leading BOM, which
XMLHttpRequest and browsers allow, but neither IETF nor Ecma-404
allow.)

-- 
http://annevankesteren.nl/

---

From: Anne van Kesteren <[email-masked]>
Date: Tue, 12 Nov 2013 15:09:58 +0800

To be clear, this is a Last Call comment on
http://tools.ietf.org/html/draft-ietf-json-rfc4627bis-07 The JSON Data
Interchange Format (draft-ietf-json-rfc4627bis-07).

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 12 Nov 2013 18:20:39 +0900

On 2013/11/12 16:09, Anne van Kesteren wrote:
> On Tue, Nov 12, 2013 at 3:01 PM, Anne van Kesteren<[email-masked]>  wrote:

If XMLHttpRequest has reasons to continue allowing it, I'd suggest that:
1) It strongly discurages it, and
2) It defines processing as something roughly like
    a) If the first few bytes look like a BOM, ignore them
    b) Process the rest according to rfc4627bis or ECMA-404 (whichever 
works better if they are not in full alignment).

That will make sure that variation is confined as locally as possible.

Regards,   Martin.

---

From: Anne van Kesteren <[email-masked]>
Date: Tue, 12 Nov 2013 17:25:53 +0800

On Tue, Nov 12, 2013 at 5:20 PM, "Martin J. Dürst"
<[email-masked]> wrote:
>    b) Process the rest according to rfc4627bis or ECMA-404 (whichever works
> better if they are not in full alignment).
>

So that is roughly how it is defined. Using the web's "utf-8 text
resource decode" method that removes a BOM and then passing the rest
to something equivalent to JSON.parse(). However, if we are defining a
new text transport format I think it would make sense to allow a
leading BOM similar to how text/css, text/html, etc. allow for that.

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 12 Nov 2013 07:29:00 -0800

[[ Adding the JSON WG to this thread ]]

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 13 Nov 2013 20:24:20 +0000

>> which is a far more unlikely scenario than parsing it as 42 given

I see Anne's input on the top-level grammar as interesting and useful.  I
believe that we could choose to be reasonable here by changing the ABNF:

JSON-text = value

and then adding text about interoperability in the same way that we have
approached numbers, strings, and duplicate keys.  Protocols that come
after this draft is published as an RFC can safely decide to allow any top
level value.  Parsers that claim compatibility with the new doc will allow
any value.  For widest interoperability, however, you can't assume that an
unknown receiver will correctly process anything but an object or an array
at the top level.

We would also need to change section 8.1 according to the mechanism that
was previously proposed:

00 00 00 xx  UTF-32BE
    00 xx ?? xx  UTF-16BE
    xx 00 00 00  UTF-32LE
    xx 00 xx ?? UTF-16LE
    xx xx ?? ?? UTF-8

in order to account for strings at the top level whose first character has
a codepoint greater than 127.

It would be awful for there to remain two mutually-incompatible versions
of JSON going forward, and I think we should take this opportunity to
address the biggest incompatibility.

>>XMLHttpRequest and browsers allow, but neither IETF nor Ecma-404

If 404 doesn't allow it, I don't see a strong need to add it.  Parsers can
always be more forgiving of what they will parse than what the spec says,
particularly since section 9 says "A JSON parser MAY accept non-JSON forms
or extensions".

-- 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Nov 2013 12:32:30 -0800

On 13 Nov 2013, at 12:24, Joe Hildebrand (jhildebr) <[email-masked]> wrote:

> changing the ABNF:

(would need to add the whitespace — that is not visible in the ECMA 404 racetracks because they use an implicit whitespace rule.)

There has been a long discussion in the WG about this potential change, I don’t want to repeat it here.
It may be worthwhile to dig up pointers to how that resulted in not making the change, for observers outside the WG.

Grüße, Carsten

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 13 Nov 2013 20:43:10 +0000

On 11/13/13 1:32 PM, "Carsten Bormann" <[email-masked]> wrote:

>(would need to add the whitespace ‹ that is not visible in the ECMA 404
>racetracks because they use an implicit whitespace rule.)

Good catch.

>There has been a long discussion in the WG about this potential change, I
>don¹t want to repeat it here.
>It may be worthwhile to dig up pointers to how that resulted in not
>making the change, for observers outside the WG.

I would argue that we didn't have adequate representation from TC39 in
that discussion, and that their getting involved is as good a reason as
any to double-check that we've still got the same consensus.  I know that
I for one decided to stop pushing for this when it looked like there were
a few more participants that wanted the status quo.

-- 
Joe Hildebrand

---

From: Tony Hansen <[email-masked]>
Date: Wed, 13 Nov 2013 16:04:54 -0500

On 11/13/13, 3:43 PM, Joe Hildebrand (jhildebr) wrote:
> that discussion, and that their getting involved is as good a reason 
> as any to double-check that we've still got the same consensus. I know 
> that I for one decided to stop pushing for this when it looked like 
> there were a few more participants that wanted the status quo. 

I personally am on the side of making this change. I think it'd be an 
extreme shame if this incompatibility remained in the document.

     Tony Hansen

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 13 Nov 2013 13:27:58 -0800

<no hat>

> a codepoint greater than 127.

A string at the top level of a JSON text still needs to start with an ASCII " character, so the logic is still fine, I believe.

Carsten's point about whitespace is more problematic. Does the ECMA-404 definition of a JSON text allow it to start with one (or more) whitespace characters? The text in that document says:
. . .
A JSON text is a sequence of tokens formed from Unicode code points that conforms to the JSON value grammar. The set of tokens includes six structural tokens, strings, numbers, and three literal name tokens.
. . .
Insignificant whitespace is allowed before or after any token.
. . .

It would be nice if ECMA-404 was clearer on this, given that the racetrack illustrations show everything other than the whitespace. In specific, it would be good to know whether or not the racetrack for "value" in Section 5 is meant to have optional whitespace at the left and right to match the above text. If TC39 could say for certain on that, it would be useful to the community.

--Paul Hoffman

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 13 Nov 2013 14:36:37 -0700

On 11/13/13 2:27 PM, "Paul Hoffman" <[email-masked]> wrote:

>A string at the top level of a JSON text still needs to start with an
>ASCII " character, so the logic is still fine, I believe.

Without top level strings, the first *two* characters of any JSON text are
always ASCII.  This:

"?"  (that's U+0022 U+0100 U+0022)

would encode the first two characters in UTF-16BE as:

00 22 01 00

8.1 currently says:

00 00 00 xx UTF-32BE
00 xx 00 xx UTF-16BE
xx 00 00 00 UTF-32LE
xx 00 xx 00 UTF-16LE
xx xx xx xx UTF-8

So the JSON text above would not match any of the table entries, causing
an error.

-- 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Nov 2013 13:40:01 -0800

On 13 Nov 2013, at 13:27, Paul Hoffman <[email-masked]> wrote:

> It would be nice if ECMA-404 was clearer on this

I think the only way to read the English prose in ECMA-404 is to always allow insignificant whitespace around every token, including the single token that makes up an out-of-container JSON value.
That is also consistent with the most widely deployed extensions of JSON for out-of-container values, including the one in recent JavaScript.

(Yes, it is easier to be unambiguous with ABNF.)

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Nov 2013 13:47:09 -0800

It is probably worth pointing out that most of this discussion will not reach es-discuss.
You may want to relay the salient points if we want their input.

Grüße, Carsten

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 13 Nov 2013 21:54:29 +0000

On 11/13/13 2:40 PM, "Carsten Bormann" <[email-masked]> wrote:

>I think the only way to read the English prose in ECMA-404 is to always
>allow insignificant whitespace around every token, including the single
>token that makes up an out-of-container JSON value.
>That is also consistent with the most widely deployed extensions of JSON
>for out-of-container values, including the one in recent JavaScript.
>

Note further that since ECMA-404 says:

"The whitespace characters are: character tabulation (U+0009), line feed
(U+000A), carriage return (U+000D), and space (U+0020)"

that the encoding detection does not need to be modified again with this
change.

-- 
Joe Hildebrand

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 13 Nov 2013 14:18:42 -0800

> would encode the first two characters in UTF-16BE as:

Yep, you're right, and ignore my previous message. I can't count to two these days....

--Paul Hoffman

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 13 Nov 2013 23:51:44 +0000

On 11/13/13 3:47 PM, "John Cowan" <[email-masked]> wrote:

>It's not clear that 404 disallows it, since 404 is defined in terms of
>characters, and a BOM is not a character but an out-of-band signal.

Agree.  However, that signal would be a part of the 4627bis octet stream,
so a little interop guidance would likely be useful.  Something like:

"Some producers of JSON produce JSON-text that starts with a redundant
U+FEFF (ZERO WIDTH NO-BREAK SPACE, previously known as BYTE ORDER MARK)
with the ostensible purpose of signaling the encoding of the text to
follow.  Since JSON has other mechanisms to determine encoding, this is
not required.  Receiving applications MAY safely ignore this initial
character without generating an error.  Implementations that do not send
U+FEFF are interoperable in the sense that all software implementations
which receive the un-prefixed text will not generate parse errors."

-- 
Joe Hildebrand

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 13 Nov 2013 19:15:28 -0800

<no hat>

On Nov 13, 2013, at 6:35 PM, Allen Wirfs-Brock <[email-masked]> wrote:

> 
> On Nov 13, 2013, at 3:51 PM, Joe Hildebrand (jhildebr) wrote:
> 
> However, for example, a conforming implementation of the ECMAScript JSON.parse function would reject any string passed to it that starts with a U+FFEF code point because the unquoted occurrence of that code point does not conform to the ECMA-252, 5th Edition or Ecma-404 JSON grammar.
> 
> In order to be successfully processed, that code point would have to be stripped from the string prior to calling JSON.parse.

The question was specifically about ECMA-404, not ECMA-252. It would be great to hear from TC39 whether or not ECMA-404 allows or disallows it.

--Paul Hoffman

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 14 Nov 2013 20:14:29 +0900

Hello Henry, others,

On 2013/11/14 18:44, Henry S. Thompson wrote:
> John Cowan writes:
>
>> Joe Hildebrand (jhildebr) scripsit:
>>
>>> If 404 doesn't allow [a BOM], I don't see a strong need to add it.
>>> Parsers can always be more forgiving of what they will parse than what
>>> the spec says, particularly since section 9 says "A JSON parser MAY
>>> accept non-JSON forms or extensions".
>>
> I think this is a crucial observation.

Yes, and I think it's based on the experience with XML. But while this 
experience may be applicable to JSON, Anne's original comment about the 
BOM and XMLHttpRequest suggests that 404 actually currently does not 
tolerate a BOM, and that implementations (except for XMLHttpRequest) 
also don't.

To give some historic background, the BOM for UTF-8 wasn't in the first 
edition of XML 
(http://www.w3.org/TR/1998/REC-xml-19980210#sec-guessing). It only later 
came in because Microsoft used it for notepad to be able to quickly 
distinguish between UTF-8 and the legacy system encoding. Because many 
people were writing some XML by hand, and some of them were using 
notepad, the pressure on XML to accept a BOM at the start of an UTF-8 
file mounted, and it was included in the second edition of the XML 
Recommendation (http://www.w3.org/TR/2000/REC-xml-20001006#sec-guessing).

Compared to XML, JSON may be much less edited by hand, or much less 
edited on notepad, or otherwise just have a different history from XML, 
but we have to make sure.

Regards,   Martin.

> I note that XML approaches
> this problem in what might be a useful way.  The XML ABNF makes no
> mention of BOM, it's not part of any XML document as such.  But it
> _is_ allowed.  The relevant wording [1] is:
>
>    Entities ... may begin with the Byte Order Mark described by Annex H
>    of [ISO/IEC 10646:2000], section 16.8 of [Unicode] (the ZERO WIDTH
>    NO-BREAK SPACE character, #xFEFF). _This is an encoding signature,_
>    _not part of either the markup or the character data of the XML_
>    _document._ XML processors must be able to use this character to
>    differentiate between UTF-8 and UTF-16 encoded documents. [emphasis
>    added]
>
> ht
>
> [1] http://www.w3.org/TR/REC-xml/#charencoding

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Wed, 13 Nov 2013 13:39:16 -0800

> It would be nice if ECMA-404 was clearer on this, given that the racetrack illustrations show everything other than the whitespace. In specific, it would be good to know whether or not the racetrac…

Yes, leading white space is allowed:

"The set of tokens includes the six structural tokens, *strings*,  *numbers*, ..."  (emphasis added)

"Insignificant whitespace is allowed before or after any token"

The elements matched by the value production are all tokens (or productions that begin and end with a token) so whitespace can occur to the left or right of any value

Allen

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Wed, 13 Nov 2013 18:35:06 -0800

Allen

> which receive the un-prefixed text will not generate parse errors."

Isn't it an interooperbility issue that many receiving applications do not ignore it.

Allen

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 14 Nov 2013 07:48:30 -0800

> The question was specifically about ECMA-404, not ECMA-252. It would be great to hear from TC39 whether or not ECMA-404 allows or disallows it.

ECMA=nnn is the correct designation for Ecma standards.  When speaking about the organization "Ecma" is the usage.

Allen

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 14 Nov 2013 07:54:56 -0800

On Nov 14, 2013, at 7:33 AM, John Cowan wrote:

> 
> Per contra, ECMA-404 refers only to text(ual content).  The BOM is
> meaningful when transforming byte sequences into code point sequences,
> but ECMA-404 deals in the latter only.  So it is the furthest thing
> from surprising that it makes no mention of BOMs, and has nothing to
> say about their use outside text.

Exactly ECMA-404 only defines a textual content format.  That was intentional.

Allen

---

From: Mark Davis ☕ <[email-masked]>
Date: Wed, 13 Nov 2013 16:50:44 -0800

On Wed, Nov 13, 2013 at 3:51 PM, Joe Hildebrand (jhildebr) <
[email-masked]> wrote:

> that all software implementations

perhaps:

...​all conformant software ...​

Mark <https://google.com/+MarkDavis>

*— Il meglio è l’inimico del bene —*

---

From: [email-masked] (Henry S. Thompson)
Date: Thu, 14 Nov 2013 09:44:00 +0000

> characters, and a BOM is not a character but an out-of-band signal.

I think this is a crucial observation.  I note that XML approaches
this problem in what might be a useful way.  The XML ABNF makes no
mention of BOM, it's not part of any XML document as such.  But it
_is_ allowed.  The relevant wording [1] is:

ht

[1] http://www.w3.org/TR/REC-xml/#charencoding
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: John Cowan <[email-masked]>
Date: Wed, 13 Nov 2013 17:47:38 -0500

Joe Hildebrand (jhildebr) scripsit:

> I see Anne's input on the top-level grammar as interesting and useful.
> I believe that we could choose to be reasonable here by changing
> the ABNF:
>
> JSON-text = value
>
> and then adding text about interoperability in the same way that we
> have approached numbers, strings, and duplicate keys.

+1 for choosing to be reasonable.

-- 
John Cowan    [email-masked]    http://ccil.org/~cowan
This great college [Trinity], of this ancient university [Cambridge],
has seen some strange sights. It has seen Wordsworth drunk and Porson
sober. And here am I, a better poet than Porson, and a better scholar
than Wordsworth, somewhere betwixt and between.  --A.E. Housman

---

From: John Cowan <[email-masked]>
Date: Wed, 13 Nov 2013 20:49:33 -0500

> U+FEFF (ZERO WIDTH NO-BREAK SPACE, previously known as BYTE ORDER MARK)

For "previously" read "also".  Indeed, the use of U+FEFF as a
zero-width no-break space is deprecated in favor of U+2600 WORD JOINER.
See <http://www.unicode.org/faq/utf_bom.html#BOM>.

-- 
John Cowan                                <[email-masked]>
Yakka foob mog.  Grug pubbawup zink wattoom gazork.  Chumble spuzz.
    --Calvin, giving Newton's First Law "in his own words"

---

From: John Cowan <[email-masked]>
Date: Thu, 14 Nov 2013 10:33:41 -0500

Paul Hoffman scripsit:

> The question was specifically about ECMA-404, not ECMA-252. It would be
> great to hear from TC39 whether or not ECMA-404 allows or disallows it.

The point of issuing standards is so that all may use them, so there is
no need to ask anyone.  A quick examination of ECMA-404 makes it clear
that there are no references to BOMs, whether under that name, or "byte
order mark", or "U+FEFF".

But that is not determinative for our purposes, because of this statement
from the Unicode Standard (I cite section 16.8 of Unicode 6.2, but
substantially equivalent statements can be found back to Unicode 3.2):

    Systems that use the byte order mark must recognize when an
    initial U+FEFF signals the byte order. In those cases, it is
    not part of the textual content and should be removed before
    processing, because otherwise it may be mistaken for a legitimate
    zero width no-break space.

-- 
Overhead, without any fuss, the stars were going out.
        --Arthur C. Clarke, "The Nine Billion Names of God"
                John Cowan <[email-masked]>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 18 Nov 2013 20:26:44 +0900

On 2013/11/18 20:11, Henry S. Thompson wrote:
> Pete Cordell writes:
>
>> Given the history below, would it be sensible to accept BOMs for UTF-8
>> encoding, but not for UTF-16 and UTF-32?  In other words, are BOMs needed
>> and/or used in the wild for UTF-16 and UTF-32?
>>
>> Maybe the text can say something like "SHOULD accept BOMs for UTF-8,
>> and MAY accept BOMs for UTF-16 and / or UTF-32"?
>
> My sense is that you'll see more UTF-16 BOMs than anything else.

Yes indeed. BOM means Byte Order Mark. It's crucial for over-the-wire 
UTF-16. (It's irrelevant for in-memory UTF-16, but that's not what we 
are discussing.) To bring up the XML example again, XML actually 
strictly requires a BOM for UTF-16. The IETF definition of UTF-16 does 
not require a BOM for UTF-16. See http://tools.ietf.org/html/rfc2781, in 
particular http://tools.ietf.org/html/rfc2781#section-3.2, 
http://tools.ietf.org/html/rfc2781#section-3.3, and 
http://tools.ietf.org/html/rfc2781#section-4.

For UTF-8, the BOM is not a Byte Order Mark, because such a mark isn't 
necessary at all. It may serve as a signature, but is not necessary, and 
in some circumstances counterproductive.

As for what to say about whether to accept BOMs or not, I'd really want 
to know what the various existing parsers do. If they accept BOMs, then 
we can say they should accept BOMs. If they don't accept BOMs, then we 
should say that they don't.

Regards,   Martin.

> UTF-32 support seems to be waning (at least in the browsers), but
> UTF-16 is in pretty widespread use.  John, do you think you can fool
> google into counting BOMs for us?

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 18 Nov 2013 13:35:00 +0100

>should say that they don't.

Unicode signatures are not useful for application/json resources and are
likely to break exisiting and future code, it is not at all uncommon to
construct JSON text by concatenating, say, string literals with some web
service response without passing the data through a JSON parser. And as
RFC 4627 makes no mention of them, there is little reason to think that
implementations tolerate them.

Perl's JSON module gives me

  malformed JSON string, neither array, object, number, string
  or atom, at character offset 0 (before "\x{ef}\x{bb}\x{bf}[]")

Python's json module gives me

  ValueError: No JSON object could be decoded

Go's "encoding/json" module gives me

  invalid character 'ï' looking for beginning of value

http://shadowregistry.org/js/misc/#t2ea25a961255bb1202da9497a1942e09 is
another example of what kinds of bugs await us if we were to specify the
use of Unicode signatures for JSON, essentially

  new DOMParser().parseFromString("\uBBEF\u3CBF\u7979\u3E2F","text/xml")

Now U+BBEF U+3CBF U+7979 U+3E2F is not an XML document but Firefox and
Internet Explorer treat it as if it were equivalent to "<yy/>".
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 18 Nov 2013 14:48:19 +0100

* Henry S. Thompson wrote:
>I'm curious to know what level you're invoking the parser at.  As
>implied by my previous post about the Python 'requests' package, it
>handles application/json resources by stripping any initial BOM it
>finds -- you can try this with
>
>>>> import requests
>>>> r=requests.get("http://www.ltg.ed.ac.uk/ov-test/b16le.json")
>>>> r.json()

The Perl code was

  perl -MJSON -MEncode -e
    "my $s = encode_utf8(chr 0xFEFF) . '[]'; JSON->new->decode($s)"

The Python code was

  import json
  json.loads(u"\uFEFF[]".encode('utf-8'))

The Go code was

  package main
  
  import "encoding/json"
  import "fmt"
  
  func main() {
    r := "\uFEFF[]"
  
    var f interface{}
    err := json.Unmarshal([]byte(r), &f)
    
    fmt.Println(err)
  }

In other words, always passing a UTF-8 encoded byte string to the byte
string parsing part of the JSON implementation. RFC 4627 is the only
specification for the application/json on-the-wire format and it does
not mention anything about Unicode signatures. Looking for certain byte
sequences at the beginning and treating them as a Unicode signature is
the same as looking for `/* ... */` and treating it as a comment.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tim Bray <[email-masked]>
Date: Mon, 18 Nov 2013 07:54:58 -0800

This feels backward, because BOMs are actually useful for UTF-16 and
UTF-32, but essentially useless for UTF-8.

On Mon, Nov 18, 2013 at 2:05 AM, Pete Cordell <[email-masked]>wrote:

> Maybe the text can say something like "SHOULD accept BOMs for UTF-8, and
> MAY accept BOMs for UTF-16 and / or UTF-32"?
>
> Thanks,
>
> Pete Cordell
> Codalogic Ltd
> C++ tools for C++ programmers, http://codalogic.com
> Read & write XML in C++, http://www.xml2cpp.com
> ----- Original Message ----- From: ""Martin J. Dürst"" <
> [email-masked]>
> To: "Henry S. Thompson" <[email-masked]>
> Cc: "John Cowan" <[email-masked]>; "IETF Discussion"
> <[email-masked]>; "Paul Hoffman" <[email-masked]>; "JSON WG"
> <[email-masked]>; "Joe Hildebrand (jhildebr)" <[email-masked]>; "Anne
> van
> Kesteren" <[email-masked]>; <[email-masked]>; "es-discuss"
> <[email-masked]>
> Sent: Thursday, November 14, 2013 11:14 AM
> Subject: Re: [Json] JSON: remove gap between Ecma-404 and IETF draft
>
>
>  Hello Henry, others,
>>
>> tolerate a BOM, and that implementations (except for XMLHttpRequest) also
>> don't.
>>
>> edition of XML (http://www.w3.org/TR/1998/REC-xml-19980210#sec-guessing).
>> It only later came in because Microsoft used it for notepad to be able to
>> quickly distinguish between UTF-8 and the legacy system encoding. Because
>> many people were writing some XML by hand, and some of them were using
>> notepad, the pressure on XML to accept a BOM at the start of an UTF-8 file
>> mounted, and it was included in the second edition of the XML
>> Compared to XML, JSON may be much less edited by hand, or much less edited
>> on notepad, or otherwise just have a different history from XML, but we
>> have to make sure.
>>
>> Regards,   Martin.
>>
>>
>> _______________________________________________
>> json mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/json
>>
>
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json
>

---

From: John Cowan <[email-masked]>
Date: Mon, 18 Nov 2013 11:19:54 -0500

> My sense is that you'll see more UTF-16 BOMs than anything else.

I agree.

> google into counting BOMs for us?

No, because Google transcodes everything into UTF-8 as soon as it starts
to process it.  What I can say (auct. Mark Davis) is that UTF-16 documents
in all formats represent much less than 0.1% of the searchable Web.
By contrast, UTF-8 (including ASCII) amounts to 80% of it.  This reflects
actual rather than declared encodings, and is as of January 2012.

-- 
So they play that [tune] on                     John Cowan
their fascist banjos, eh?                       [email-masked]
        --Great-Souled Sam                      http://www.ccil.org/~cowan

---

From: John Cowan <[email-masked]>
Date: Mon, 18 Nov 2013 12:07:09 -0500

Pete Cordell scripsit:

> Not useless if you're trying to tell the difference between a hand
> editted Windows cp-1252 (or whatever it's called) encoded text file
> and a UTF-8 encoded text file.

In principle you cannot tell, but in practice it's possible to discriminate
betwen the two with extremely high reliability.

-- 
"Well, I'm back."  --Sam        John Cowan <[email-masked]>

---

From: [email-masked] (Henry S. Thompson)
Date: Mon, 18 Nov 2013 11:11:01 +0000

ht

---

From: [email-masked] (Henry S. Thompson)
Date: Mon, 18 Nov 2013 12:59:18 +0000

>>> r=requests.get("http://www.ltg.ed.ac.uk/ov-test/b16le.json")

Signatures are not part of the text of a document, as the UNICODE spec
makes clear, so asking what happens when you pass a string beginning
with a BOM to a parser is not really the right question in this
context, is it?

As I tried to say in an earlier post, there's a distinction which
needs to be carefully insisted on between, on the one hand, languages
and their parsers, where I agree signatures/BOMs have no place, and,
on the other hand, (media-typed) resources/entities/payloads and _their_
processing, where a discussion of BOMs/signatures _is_ appropriate
and, often, necessary.

BTW I agree that the status of the UTF-8 BOM as signature is slightly
hazy, but again the UNICODE spec itself [1] says

  "this sequence can serve as signature for UTF-8 encoded text where
   the character set is unmarked"

ht

[1] http://www.unicode.org/versions/Unicode6.2.0/ch16.pdf

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Nov 2013 17:22:42 +0100

On 2013-11-18 17:08, Pete Cordell wrote:
> ----- Original Message From: "Tim Bray" <[email-masked]>
> editted Windows cp-1252 (or whatever it's called) encoded text file and
> a UTF-8 encoded text file.

Ye.

> I don't think we need them for any other reason, but I think some
> international Windows users would be thankful if you allowed them for
> that case.

So I gather "non-international Windows" users never need non-ASCII 
characters? :-) (Yes, that explains lots of I18N brokenness originating 
from certain countries...)

Best regards, Julian

---

From: "Pete Cordell" <[email-masked]>
Date: Mon, 18 Nov 2013 10:05:07 -0000

Maybe the text can say something like "SHOULD accept BOMs for UTF-8, and MAY 
accept BOMs for UTF-16 and / or UTF-32"?

Thanks,

Pete Cordell
Codalogic Ltd
----- Original Message ----- 
From: ""Martin J. Dürst"" <[email-masked]>
To: "Henry S. Thompson" <[email-masked]>
Cc: "John Cowan" <[email-masked]>; "IETF Discussion"
<[email-masked]>; "Joe Hildebrand (jhildebr)" <[email-masked]>; "Anne van

---

From: "Pete Cordell" <[email-masked]>
Date: Mon, 18 Nov 2013 13:36:13 -0000

----- Original Message ----- 
> UTF-16. (It's irrelevant for in-memory UTF-16, but that's not what we are 
> discussing.)

The in-memory case is not entirely irrelevant because a number of JSON 
messages will be constructed in memory and then squirted to line.

I did a little experiment with Visual Studio.  It will allow me to save in 
UTF-8 with or without a BOM (like thing).  Saving in UTF-16 (Or was it 
UCS2?) is always with a BOM.  There didn't seem to be a UTF-32 option.

JSON doesn't need BOMs.  However, there are cases where people might hand 
edit messages, and if they choose to save in UTF-16 they will likely have a 
BOM.

Is it acceptable to tell people not to save hand editted files in UTF-16, 
suggesting UTF-8 (possibly with an encoded BOM) as an alternative?

I would imagine that if someone did have a hand editted UTF-8 file on 
Windows then the allowance of a BOM would help their sanity immeasurably, 
but it's not something I have firsthand knowledge of.

I believe Unix/Linux works with UTF-8 without BOMs.  Is this the case?

Pete Cordell
Codalogic Ltd

---

From: "Pete Cordell" <[email-masked]>
Date: Mon, 18 Nov 2013 16:08:53 -0000

> UTF-32, but essentially useless for UTF-8.

Not useless if you're trying to tell the difference between a hand editted 
Windows cp-1252 (or whatever it's called) encoded text file and a UTF-8 
encoded text file.

I don't think we need them for any other reason, but I think some 
international Windows users would be thankful if you allowed them for that 
case.

On Mon, Nov 18, 2013 at 2:05 AM, Pete Cordell <[email-masked]>wrote:

>> file
>> edited

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 18 Nov 2013 15:56:11 -0500

> but it's not something I have firsthand knowledge of.

I believe the opposite is true.

The failure of Windows to correctly process documents without BOM markers
is a constant pain trying to use .NET to parse XML.

The ability to compose a JSON message by wrapping another JSON message is
essential. That is, it has to be possible to write something like

printf ("{\"Object\", %s}", Text);

I use the .NET platform heavily. Please do not let Microsoft off the hook
here. The cost of doing so is having to write code to kick out spurious BOM
sequences occurring at any random point in the text. Which becomes really
painful when having to deal with strings where there might actually be a
reason to put the BOM in.

The benefit of not doing so is that it might encourage Microsoft to fix
their tools so that they don't insert spurious BOM sequences in documents
where doing so breaks them.

-- 
Website: http://hallambaker.com/

---

From: John Cowan <[email-masked]>
Date: Mon, 18 Nov 2013 13:41:53 -0500

Pete Cordell scripsit:

> Do you mean that the presence of a UTF-8 BOF sequence doesn't prove
> that it's not Windows cp-1252 or do you mean you can tell apart a
> UTF-8 and cp-1252 file without BOMs?

I meant the latter, but the former is true, too.  A plain text document
beginning "ï»¿" in Windows-1252 will appear to begin with an 8-BOM
in the absence of out of band information.

> If the latter, do the relevant tools take the time to distinguish
> the 2 without BOMs?

Some tools do, some don't.  The IRC client I use, XChat, attempts to
convert input as UTF-8, and if that fails, converts it as Latin-1.
I have not yet seen it produce mojibake.

-- 
John Cowan   [email-masked]  http://www.ccil.org/~cowan
Most languages are dramatically underdescribed, and at least one is
dramatically overdescribed.  Still other languages are simultaneously
overdescribed and underdescribed.  Welsh pertains to the third category.
        --Alan King

---

From: "Pete Cordell" <[email-masked]>
Date: Mon, 18 Nov 2013 18:19:46 -0000

----- Original Message From: "John Cowan"

> betwen the two with extremely high reliability.

John,

Do you mean that the presence of a UTF-8 BOF sequence doesn't prove that 
it's not Windows cp-1252 or do you mean you can tell apart a UTF-8 and 
cp-1252 file without BOMs?

If the latter, do the relevant tools take the time to distinguish the 2 
without BOMs?

Thanks,

Pete Cordell
Codalogic Ltd

---

From: Tatu Saloranta <[email-masked]>
Date: Mon, 18 Nov 2013 10:59:48 -0800

Dominant Java implementations support UTF-16 with BOM; either directly or
through Java's Reader implementations that handle BOMs.
String concatenation case seems irrelevant, since BOMs are not included in
in-memory representation anyway, as opposed to byte stream serialization.

-+ Tatu +-

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 19 Nov 2013 13:32:37 +0900

Okay, here are some more tests.

http://www.sw.it.aoyama.ac.jp/2013/pub/json_tests/test1_utf8_nobom.json
http://www.sw.it.aoyama.ac.jp/2013/pub/json_tests/test2_utf8_bom.json

They are self-describing JSON files served with application/json, the 
first without a BOM, and the second with a BOM.

They contain some Japanese, and a tiny bit of Spanish.

[see more below]

>>>> r=requests.get("http://www.ltg.ed.ac.uk/ov-test/b16le.json")

I get a 404 on this example. I can put up UTF-16 examples, too.

Regards,   Martin.

---

From: t.p. <[email-masked]>
Date: Tue, 19 Nov 2013 10:10:47 +0000

----- Original Message -----
From: "Martin J. Dürst" <[email-masked]>
To: "Henry S. Thompson" <[email-masked]>
Cc: "John Cowan" <[email-masked]>; "IETF Discussion"
<[email-masked]>; "Pete Cordell" <[email-masked]>; "JSON WG"
<[email-masked]>; "Anne van Kesteren" <[email-masked]>;
<[email-masked]>; "es-discuss" <[email-masked]>
Sent: Monday, November 18, 2013 11:26 AM

> in some circumstances counterproductive.

Martin

We had a similar discussion with syslog back in 2005, the issue being
that UTF-8 was new and different and how to tell whether it was being
used or not, and what made it into RFC5424 was
"  If a syslog application encodes MSG in UTF-8, the string MUST start
   with the Unicode byte order mask (BOM), which for UTF-8 is ABNF
   %xEF.BB.BF.  "
which remains a MUST to this day.  There are no relevant Errata.

Tom Petch

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 19 Nov 2013 12:02:55 +0100

>is a constant pain trying to use .NET to parse XML.

Test case please?

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 19 Nov 2013 20:09:30 +0900

On 2013/11/19 19:10, t.p. wrote:
> From: "Martin J. Dürst"<[email-masked]>

> which remains a MUST to this day.  There are no relevant Errata.

This is something that seems to have made quite a lot of sense for 
syslog. I can understand that if before 2005, syslog was used with 
legacy encodings (iso-8859-1, Shift_JIS and similar), and there was 
otherwise no easy way to label the UTF-8 strings.

But another solution (for syslog, that is) would also have been 
possible. As John already pointed out, UTF-8 is very easy to detect 
heuristically: If a byte sequence follows the UTF-8 byte pattern, it's 
most definitely UTF-8 and not something else. For more background, 
please see http://www.sw.it.aoyama.ac.jp/2012/pub/IUC11-UTF-8.pdf, where 
that idea came up first.

As for JSON, it doesn't have the problem of legacy encodings. JSON by 
definition is encoded in an Unicode encoding form, and it's easy to 
distinguish these because of the restrictions on character sequences in 
JSON. And this can be done without a BOM (or with a BOM).

What's most important now is to know what receivers actually accept. We 
are not in a design phase, we are just updating the definition of JSON 
and making sure we fix problems if there are problems, but we have to 
use the installed base for the main guidance, not other protocols or 
formats.

Regards,   Martin.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 19 Nov 2013 13:31:02 +0100

>in-memory representation anyway, as opposed to byte stream serialization.

HTTP implementations cannot correctly determine whether an entity body
is text in a single character encoding and if so what that encoding is,
accordingly the dominant API deals in byte[] arrays, not text Strings;
furthermore, many programming languages default to byte[] arrays for
string literals. That often combines into forms of

  byte[] json = sprintf('{"x": %s, "y": %s}', GET(...), GET(...));

which works fine if all three byte[] arrays are UTF-8 encoded and use
no Unicode signature, which is the case 99% of the time.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Tue, 19 Nov 2013 08:59:36 -0800

On Nov 19, 2013, at 3:09 AM, Martin J. Dürst wrote:
> ...
> As for JSON, it doesn't have the problem of legacy encodings. JSON by definition is encoded in an Unicode encoding form, and it's easy to distinguish these because of the restrictions on character sequences in JSON. And this can be done without a BOM (or with a BOM).
> 
> What's most important now is to know what receivers actually accept. We are not in a design phase, we are just updating the definition of JSON and making sure we fix problems if there are problems, but we have to use the installed base for the main guidance, not other protocols or formats.

There can be no doubt that the most widely deployed JSON parsers are those that are built intp the browser javascript implementations.  The ECMAScript 5 specification for JSON.parse that they implement says BOM is an illegal character.  But what do the browser actually implement?  This:

//FireFox 25 scratchpad execution:
JSON.parse('\ufeff {"abc": 0} ')
/*
Exception: JSON.parse: unexpected character
@Scratchpad/1:1
*/

JSON.parse('\ufeff {"abc": 0} ')
/*
Exception: JSON.parse: unexpected character
@Scratchpad/1:1
*/

JSON.parse('\ufeff {"abc": 0} ')
/*
Exception: JSON.parse: unexpected character
@Scratchpad/1:1
*/
JSON.parse('\ufeff {"abc": 0} ')
/*
Exception: JSON.parse: unexpected character
@Scratchpad/1:1
*/
JSON.parse('\ufeff {"abc": 0} ')
/*
Exception: JSON.parse: unexpected character
@Scratchpad/1:1
*/

//Safari 5.1.9 JS console
JSON.parse('\ufeff {"abc": 0} ')
message: "JSON Parse error: Unrecognized token '?'"

//Chrome 31 JS console
JSON.parse('\ufeff {"abc": 0} ')
SyntaxError: Unexpected token 
message: "Unexpected token "

Unfortunately, I don't have access to IE right now,  but the trend is clear

Allen

---

From: Chris Lilley <[email-masked]>
Date: Tue, 19 Nov 2013 08:13:37 +0100

Hello Bjoern,

Monday, November 18, 2013, 2:48:19 PM, you wrote:

> string parsing part of the JSON implementation.

Yes, a byte stream will contain a BOM if one is present.

> RFC 4627 is the only

No, because /* */ are characters and are found in a character stream.
And a character stream does not contain a BOM as a BOM is not
character data.

RFC 4627 doesn't need to say this, because Unicode says it.

If JSON mixes up characters and bytes there will of course be
confusion. But hopefully it doesn't, as this is not 1990 anymore.

-- 
Best regards,
 Chris                            mailto:[email-masked]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 20 Nov 2013 15:18:45 +0900

Hello Henry, others,

On 2013/11/20 3:55, Henry S. Thompson wrote:
> Allen Wirfs-Brock writes:
>
>> There can be no doubt that the most widely deployed JSON parsers are
>> those that are built intp the browser javascript implementations.
>> The ECMAScript 5 specification for JSON.parse that they implement
>> says BOM is an illegal character.  But what do the browser actually
>> implement?  This:
>
> No, try e.g. jsonviewer.stack.hu [1] (works in Chrome, Safari, Opera,
> not in IE or Firefox)

In Firefox, I got some garbled characters, in particular some question 
marks for each of the two bytes of the BOM and one question mark for the 
e-acute. Because of the type of the errors, I strongly suspect it is 
related to what we are trying to investigate, and so I don't think this 
can be taken as evidence one way or another.

or feed [2] to www.jsoneditoronline.org (Use
> Open/Url) (works in Chrome, IE, Firefox, ran out of time to test more).

The fact that some libraries or Web sites accept a BOM for JSON isn't a 
proof that all (well, let's say the majority) accept a BOM.

> As previously discussed, _no-one_ is arguing that BOMs are in the JSON
> language as such.  JSON parsers shouldn't accept BOMs.
>
> BOMs are, to quote the UNICODE spec, "not part of the text".  It is
> appropriate that specs concerned with JSON-on-the-wire, for example
> the media type registration for 'application/json', _should_ discuss
> the BOM, and it's open to them, _without changing the language at
> all_, to say that BOMs are acceptable but, again, are not part of the
> text which the parser has to accept.

I agree that *from a theoretical viewpoint*, this is correct. But theory 
isn't everything. As I have written before (and you have cited in 
another thread, for another spec):

   What's most important now is to know what receivers actually
   accept. We are not in a design phase, we are just updating the
   definition ... and making sure we fix problems if there are
   problems, but we have to use the installed base for the main
   guidance

For our update from RFC 4627, the null hypothesis is that there are no 
BOMs (neither for UTF-8 nor for UTF-16). The patterns given in 
http://tools.ietf.org/html/rfc4627#section-3 cannot apply to characters, 
they can only apply to bytes. If we want to allow a spec in 
application/json, then we have to have strong evidence that almost all 
parsers can deal with BOMs, not just fragmentary evidence that some 
parsers don't choke on a BOM.

Please note that there's some parallel to XML, in that neither Unicode 
(for the encoding form) nor the IETF (for the 'charset') require a BOM 
for "UTF-16", but XML nevertheless strictly requires it.

Regards,   Martin.

> ht
>
> [1] http://jsonviewer.stack.hu/#http://www.ltg.ed.ac.uk/~ht/ov-test/b16le.json

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 19 Nov 2013 18:55:29 +0000

No, try e.g. jsonviewer.stack.hu [1] (works in Chrome, Safari, Opera,
not in IE or Firefox) or feed [2] to www.jsoneditoronline.org (Use
Open/Url) (works in Chrome, IE, Firefox, ran out of time to test more).

ht

---

From: [email-masked] (Henry S. Thompson)
Date: Wed, 20 Nov 2013 10:29:35 +0000

Martin J. Dürst writes:

> Hello Henry, others,
>
> The fact that some libraries or Web sites accept a BOM for JSON isn't
> a proof that all (well, let's say the majority) accept a BOM.

I wasn't suggesting that it did, rather that the kind of testing that
needed to be done was testing that included the interface between
transport and parser, not parser alone, since, as we are all agreed,
the BOM is not allowed by the language itself.

> I agree that *from a theoretical viewpoint*, this is correct. But
> theory isn't everything. As I have written before (and you have cited
> in another thread, for another spec):
>
> http://tools.ietf.org/html/rfc4627#section-3 cannot apply to
> characters, they can only apply to bytes. If we want to allow a spec
> in application/json, then we have to have strong evidence that almost
> all parsers can deal with BOMs, not just fragmentary evidence that
> some parsers don't choke on a BOM.

I'm not suggesting parsers should allow BOMs, if by parser you mean
what implements the grammar of the language.  And I'm certainly not
suggesting that two examples pulled at random from a Google search
constitute strong evidence about practice at large.  All I was trying
to suggest was that showing that the language parser as such in
Firefox etc.didn't allow BOMs, as the OP had done, was the right kind
of evidence _either_.

> for "UTF-16", but XML nevertheless strictly requires it.

I agree that XML is a useful point of comparison, in particular
because it too does not allow a BOM as part of an XML document, but
rather treats it as an aspect of packaging/transport external to the
XML document, which seems to me to be the kind of approach to BOMs the
JSON WG might consider.

ht

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 19 Nov 2013 20:24:36 +0000

I've been back and forth on this several times, however I think I've come
to a conclusion for myself unless new arguments or data comes in.

I don't see any value in a BOM ever being included intentionally.  RFC4627
does not specify anything about BOMs.  RFC4627 explicitly has a mechanism
for determining encoding that does not use BOMs.  Although ECMA-262
supports more whitespace than RFC4627, <BOM> is not one of the supported
whitespace characters in my reading.  Same with ECMA-404.  Many existing
implementations do not parse BOM-prepended text in practice (thanks
Allen). 

I understand the use case of manually creating a document in notepad and
getting a BOM you didn't expect, but I don't think that's a compelling
enough reason to break backward-compatibility by adding effectively-new
blessings for BOMs.

Section 9 provides complete air-cover for any parser writer that wants to
support BOMs anyway:

"A JSON parser MAY accept non-JSON forms or extensions."

What they do with them, particularly when they don't match the encoding
detection of section 8.1 would of course be unspecified, but those
documents would have been rejected by a strictly-conforming parser anyway.

As such, I think our best course of action is to change nothing with
respect to BOMs, leaving them as non-interoperable syntax extensions.

On 11/19/13 5:59 PM, "Allen Wirfs-Brock" <[email-masked]> wrote:

>those that are built intp the browser javascript implementations.  The
>ECMAScript 5 specification for JSON.parse that they implement says BOM is
>an illegal character.  But what do the
> browser actually implement?  This:
>
>
>*/JSON.parse('\ufeff {"abc": 0} ')
>/*
>*/JSON.parse('\ufeff {"abc": 0} ')
>/*
>1. message: "JSON
> Parse error: Unrecognized token '?'"
>
>
>
>//Chrome 31 JS console
>JSON.parse('\ufeff
> {"abc": 0} ')
>1. 
>
>
>1. message: "Unexpected
> token "
>Unfortunately,
> I don't have access to IE right now,  but the trend is clear
>
>
>Allen
>
>
>
>
>
>
>
>

-- 
Joe Hildebrand

---

From: Tatu Saloranta <[email-masked]>
Date: Tue, 19 Nov 2013 10:30:47 -0800

> no Unicode signature, which is the case 99% of the time.

My point was just that although it appears that many scripting languages
may not deal with BOM properly, same is not true on all platforms. Proper
JSON APIs on JVM do accept both String and byte[] based input; byte[] being
preferred since it is more efficient, and reliably with auto-detection,
assuming that -- as per JSON specification -- the only single-byte encoding
used is UTF-8.

-+ Tatu +-

---

From: Henri Sivonen <[email-masked]>
Date: Thu, 21 Nov 2013 15:28:43 +0200

On Thu, Nov 21, 2013 at 7:53 AM, Allen Wirfs-Brock
<[email-masked]> wrote:
> Just to be clear about this.  My tests directly tested JavaScript built-in
> JSON parsers WRT to BOM support in three major browsers.  The tests directly
> invoked the built-in JSON.parse functions and directly passed to them a
> source strings that was explicitly constructed to contain a BOM code point .
> This was done to ensure that the all transport layers  (and any transcodings
> they might perform) were bypassed and that we were actually testing the real
> built-in JSON parse functions.

It would be surprising if JSON.parse() accepted a BOM, since it
doesn't take bytes as input.

However, XHR's responseType = "json" exercises browsers in a way where
the input is bytes from the network. From the perspective of JSON
support in XHR,
http://lists.w3.org/Archives/Public/www-tag/2013Nov/0149.html (which
didn't reach the es-discuss part of this thread previously) applies.

-- 
Henri Sivonen
[email-masked]
http://hsivonen.fi/

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Thu, 21 Nov 2013 09:01:01 -0800

> doesn't take bytes as input.

ECMAScript's JSON.parse accepts an ECMAScript string value as its input.  ECMAScript strings are sequences of 16-bit values.  JSON.parse (and most other ECMAScript functions) interpret those values  as Unicode code units.  The value U+FEFF can appear at any position within a string. When defining a string as an ECMAScript literal, a sequence like \ufeff is an escape sequence that means place the code unit value 0xefff into the string at this position in the sequence. Also note that the actual strings passed below to JSON.parse contain the actual code point value U+FEFF not the escape sequence that was used to express it.  To include the actual escape sequence characters in the string it would have to be expressed as '\\feff'.

JSON.parse('\ufeff ["XYZ"]');  //note outer quotes delimit an ECMAScript string, the inner quotes are a JSON string.  

throws a runtime SyntaxError exception because the JSON grammar does not allow U+FEFF to appear that position

JSON.parse('["\ufeffXYZ"]');

operates without error and returns a Array containing a four element ECMAScript string.   This works because the JSON grammar allows any code unit except for " and \ and the ASCII control characters to appear literally in a JSON string. 

> didn't reach the es-discuss part of this thread previously) applies.

Right, JSON use via XHR is a different usage scenario and that probably involves encoding and decoding steps. It has very little to do with the JSON syntax, as defined in ECMA-404. It's all about how the bits that represent a string are interchanged, not the eventual semantic processing of the string (ie, processing by JSON.parse or some other JSON parser)

Allen

---

From: Allen Wirfs-Brock <[email-masked]>
Date: Wed, 20 Nov 2013 21:53:22 -0800

On Nov 19, 2013, at 10:18 PM, Martin J. Dürst wrote:

> In Firefox, I got some garbled characters, in particular some question marks for each of the two bytes of the BOM and one question mark for the e-acute. Because of the type of the errors, I strongly suspect it is related to what we are trying to investigate, and so I don't think this can be taken as evidence one way or another.
> 
> The fact that some libraries or Web sites accept a BOM for JSON isn't a proof that all (well, let's say the majority) accept a BOM.

Just to be clear about this.  My tests directly tested JavaScript built-in JSON parsers WRT to BOM support in three major browsers.  The tests directly invoked the built-in JSON.parse functions and directly passed to them a source strings that was explicitly constructed to contain a BOM code point .  This was done to ensure that the all transport layers  (and any transcodings they might perform) were bypassed and that we were actually testing the real built-in JSON parse functions.

Neither of the sites referenced about perform a comparable test.  They take user inputed text when is then pass through whose knows what layers of browser and application preprocessing and then they present something derived from that original user input to a JSON parser.  In both bases the actual parser does not appear to be the the built-in JavaScript JSON.parse function that I was testing.

json.view.stack.hu uses Ext.util.JSON.decode whose document describe it as "Modified version of Douglas Crockford"s json.js".   In other words not the built-in JSON.parse function

www.jsoneditoronlineorg uses a library called JSONLint in preference to the built-in JSON.parse function.  It does not conform to the ECMAScript 5 JSON.parse specification.

So testing using either of these sites say nothing relevant to about my observation concern BOM handling by the most widely deployed JSON parsers (the ones that are built into browser JavaScript implementations) 

Allen

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 21 Nov 2013 18:41:08 +0100

>ECMAScript's JSON.parse accepts an ECMAScript string value as its input.
>ECMAScript strings are sequences of 16-bit values.  JSON.parse (and most
>other ECMAScript functions) interpret those values  as Unicode code 
>units.  The value U+FEFF can appear at any position within a string. 
>When defining a string as an ECMAScript literal, a sequence like \ufeff 
>is an escape sequence that means place the code unit value 0xefff into 
>the string at this position in the sequence. Also note that the actual 
>strings passed below to JSON.parse contain the actual code point value 
>U+FEFF not the escape sequence that was used to express it.  To include 
>the actual escape sequence characters in the string it would have to be 
>expressed as '\\feff'.

A byte order mark indicates the order of bytes in a sequence of bytes.
An ecmascript string is not a sequence of bytes and therefore it cannot
have a byte order mark inside it. Your test is not for BOM support but
for an egregious semantic error in the implementation of JSON.parse.

  http://shadowregistry.org/js/misc/#t2ea25a961255bb1202da9497a1942e09

That is a similar test. It makes Firefox see UTF-8 BOMs in ecmascript
strings. Firefox is not supposed to look for UTF-8 BOMs in ecmascript
strings because ecmascript strings are not sequences of bytes at that
level of reasoning.

Is there any chance, by the way, to change `JSON.stringify` so it does
not output strings that cannot be encoded using UTF-8? Specifically,

  JSON.stringify(JSON.parse("\"\uD800\""))

would need to escape the surrogate instead of emitting it literally.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 21 Nov 2013 20:37:49 +0100

* John Cowan wrote:
>Bjoern Hoehrmann scripsit:
>
>No, there isn't.  We've been down this road repeatedly.  People can and
>do use JSON strings to encode arbitrary sequences of unsigned 16-bit integers.

The output of JSON.stringify("\uD800") contains no backslash character,
if you call `utf8_encode(JSON.stringify("\uD800"))` you get an exception
because UTF-8 cannot encode the lone surrogate and `utf8_encode` does
not know it could encode it as `\uD800` without loss of information. If
`JSON.stringify` produced an escape sequence instead, there would be no
problem passing the output to `utf8_encode`.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: John Cowan <[email-masked]>
Date: Thu, 21 Nov 2013 14:15:33 -0500

-- 
You let them out again, Old Man Willow!                 John Cowan
What you be a-thinking of?  You should not be waking!   [email-masked]
Eat earth!  Dig deep!  Drink water!  Go to sleep!
Bombadil is talking.                                    http://ccil.org/~cowan

---

From: Nico Williams <[email-masked]>
Date: Thu, 21 Nov 2013 13:58:13 -0600

> problem passing the output to `utf8_encode`.

That's just one implementation.  We had hundreds of e-mails in this
list about this.  Well over a thousand to cover several issues like
this.  I think the only area where we have [roughly] consensus to
revisit the previous consensus is the top-level value restriction,
which has led to the whole UTF and byte-order detection sub-thread
(which we had, also, had before).  We're on much stronger ground to
revisit this one matter than the whole unpaired surrogates matter, and
we're much much less likely to change our consensus on that because
one proposal is about relaxing JSON to match ECMAScript's definition,
while yours is to do the opposite.

Nico
--

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 22 Nov 2013 19:43:05 +0100

* Matt Miller (mamille2) wrote:
>There does not appear to be any consensus on explicitly allowing or 
>disallowing of a Byte Order Mark (BOM).  Neither RFC4627 nor the current 
>draft mention BOM anywhere, and the modus operandi of the JSON Working 
>Group has been to leave text unchanged unless there was wide support.

To be clear, that means application/json entities that start with a byte
sequence that matches U+FEFF encoded in UTF-8/16/32 is malformed because
the ABNF does not allow a U+FEFF at that position (and interpreting such
a sequence as anything other than ordinary character data requires
explicit specification). I do think an informational note saying as much
could be useful.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Fri, 22 Nov 2013 18:21:21 +0000

There appears to be consensus to change JSON-text to allow for any JSON value -- not just object / array -- while noting that object or array as the top-level is the most interoperable.

We will ask the Document Editor to make this change to draft-ietf-json-rfc4627bis.

- Paul Hoffman and Matt Miller

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Fri, 22 Nov 2013 18:29:01 +0000

There does not appear to be any consensus on explicitly allowing or disallowing of a Byte Order Mark (BOM).  Neither RFC4627 nor the current draft mention BOM anywhere, and the modus operandi of the JSON Working Group has been to leave text unchanged unless there was wide support.

---

From: Tim Bray <[email-masked]>
Date: Wed, 27 Nov 2013 16:13:40 -0800

To do this, I think the draft requires these changes:

- Remove the trailing section of section 1.2, starting with “ECMAscript 5.1
enumerates...” [because the difference no longer exists]

- In section 2:

-- remove “A JSON text is a serialized object or array.”

-- Insert: “A JSON text is a serialized value.  Note that certain previous
specifications of JSON constrained a JSON text to be an object or an array.
 Implementations which generate only objects or arrays where a JSON text is
called for will be interoperable in the sense that all implementations will
accept these as conforming JSON texts.”

-- Change the JSON-text production to read:

JSON-text  = value

On Fri, Nov 22, 2013 at 10:21 AM, Matt Miller (mamille2) <[email-masked]
> wrote:

> There appears to be consensus to change JSON-text to allow for any JSON
> value -- not just object / array -- while noting that object or array as
> the top-level is the most interoperable.
>

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 27 Nov 2013 19:29:18 -0800

<no hat>

On Nov 27, 2013, at 5:00 PM, Alex Russell <[email-masked]> wrote:

> Will you also be citing ECMA-404 normatively to avoid this sort of divergence in the future?

If you believe that ECMA-404 will change in the future, that would indicate that ECMA might break interoperability with current implementations, even for what they perceive as "good reasons". In general, the IETF tries not to have its long-lived standards normatively latch on to moving targets for this very reason. Even when other SDOs have assured us that they will not make backwards-incompatible changes, they have done so anyway (cue the Klensin-esqe theme music...), and that has caused serious interoperability problems for the IETF specs.

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Nov 2013 04:51:01 +0100

> JSON-text  = value

Add optional whitespace around that.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Wed, 27 Nov 2013 20:00:12 -0800

I listed some arguments against this in
http://lists.w3.org/Archives/Public/www-tag/2013Oct/0041.html and at the
moment I still believe them. Is there new information?

On top of that, I have no fear of anyone trying to change JSON in the
future; they would be resoundingly ignored by the community of
implementers.  I speak as one who would love to add built-in date/time
literals but know that it won’t happen.  -T

On Wed, Nov 27, 2013 at 5:00 PM, Alex Russell <[email-masked]>wrote:

> On Wed, Nov 27, 2013 at 4:13 PM, Tim Bray <[email-masked]> wrote:
>
>> - Remove the trailing section of section 1.2, starting with “ECMAscript
>> 5.1 enumerates...” [because the difference no longer exists]
>>
>> - In section 2:
>>
>> -- Insert: “A JSON text is a serialized value.  Note that certain
>> previous specifications of JSON constrained a JSON text to be an object or
>> an array.  Implementations which generate only objects or arrays where a
>> JSON text is called for will be interoperable in the sense that all
>> implementations will accept these as conforming JSON texts.”
>>
>> On Fri, Nov 22, 2013 at 10:21 AM, Matt Miller (mamille2) <
>> [email-masked]> wrote:
>>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 27 Nov 2013 23:55:01 -0500

> literals but know that it won’t happen.  -T

The JavaScript variant may stay the same but people will add features if
they find they are necessary to meet their requirements.

As was previously established, many implementations already support a list
of values so as to enable use in append only logs. I have added length
encoded blocks so as to avoid repeatedly base64 encoding binary blobs.

The Javascript world can ignore me, I try my best to ignore the javascript
world. But if people are implementing one of my protocols they will find
that while it will work perfectly OK with JSON, implementations will be
more efficient if they support JSON-A or JSON-B.

Suggesting that people will never change a spec is silly. Of course specs
will be changed.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 28 Nov 2013 07:44:23 +0000

>-- Change the JSON-text production to read:

We also need to make an explicit decision about whitespace-tolerance.

-- 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Nov 2013 09:41:04 +0100

> We also need to make an explicit decision about whitespace-tolerance.

If the point is to track the changes made by ECMA-404 that decision has been made for us (see end of section 4 in 404).
(Note also that JSON is whitespace-tolerant today, so not having that for the new non-containers at top-levels but everywhere else would be rather surprising.*)

Minimal change appears to be:

JSON-text = ws value ws

Grüße, Carsten

*) The other interesting practical usage here is removing newlines from the allowed whitespace for a single JSON-text and joining sequences of these by newlines to form streams of JSON values.
But that is outside the scope of application/json and application/__+json, an instance of which is exactly one JSON-text.
(truenull3true is a valid stream of JSON texts if you don’t require adding whitespace as delimiters.
Implementations that employ a typical form of tokenization without knowing what those tokens can be in JSON won’t like that.  In any case, whitespace is needed to separate numbers.)

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Nov 2013 17:16:31 +0100

On 28 Nov 2013, at 05:51, Tim Berners-Lee <[email-masked]> wrote:

> JSON is interesting in being a subset of ECMAscript.  

That was a clever device used by Douglas Crockford to minimize the bike shedding that was inevitable when he designed the data format.  That device is no longer necessary, indeed, it has become a liability as ECMAscript moved forward.

> That is a big dependency -- will it be preserved?

There is no dependency any more, and the subset relationship has already been destroyed by the introduction of Line/Paragraph separators (U+2028/U+2029) in recent versions of ECMAscript (these are valid in JSON strings but not in ECMAscript 5 strings).

It is best to acknowledge the role of JavaScript as the inspiration for the JSON data interchange format (and one of its biggest customers), but otherwise consider the two specifications independent of each other.

Grüße, Carsten