Subject: json-string for HTTP header field parameter values

From: "Manger, James H" <[email-masked]>
Date: Mon, 31 Oct 2011 00:07:22 +1100

HTTP currently uses token and quoted-string for various header field parameter values, and recommends these syntaxes for new headers. However neither supports Unicode, which isn't really acceptable today.

I would like to recommend the JSON string syntax for new header field parameter values. JSON is very widely used on the web, particularly by protocols built on HTTP. There are JSON implementations for basically every computer language. JSON support the full range of Unicode characters. Developers love it.

A JSON string: is enclosed in double quotes; uses \" and \\ to represent " and \; uses six other \x sequences for other chars; and allows \uXXXX as an escape sequence for any Unicode character [json.org, RFC4627]. An HTTP header profile of JSON string would require any chars outside the printable ASCII set to be escaped.

RFC5987 "Character Set and Language Encoding for HTTP Header Field Parameters" already offer one way to represent any Unicode string in a HTTP header parameter value, eg foo*=UTF-8''coll%C3%A8gues. However this is not very appealing when defining a new parameter. HTTPbis-p2 already recommends new parameters allow the token and quoted-string syntaxes so supporting RFC5987 for Unicode means implementations have to support 2 parameter names (foo and foo*), 3 syntaxes, and 2 escaping mechanisms (\x in quoted-string, and %xx in RFC5987) -- all for a brand new parameter. Yuck.

I think the considerations for new headers (issue #231), and advice on defining auth scheme parameters (issue #320), should consider how to support Unicode parameter values -- and json-string would be a good way to do that.

P.S. json-string could also work in practice in places where quoted-string is defined (such as for parameters of new authentication schemes), since no actual quoted-string value will ever have escaped 'u' as '\u' so '\uXXXX' could be safely interpreted as per JSON instead of as 'uXXXX' as per quoted-string rules.

--
James Manger

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 31 Oct 2011 14:03:25 +1100

Hi James,

> HTTP currently uses token and quoted-string for various header field parameter values, and recommends these syntaxes for new headers. However neither supports Unicode, which isn't really acceptable…

Well, it uses a variety of things. One of our outstanding tasks is to collect a set of header microsyntax that's in common use; e.g., the parameterised string, which is already used widely.

> I would like to recommend the JSON string syntax for new header field parameter values. JSON is very widely used on the web, particularly by protocols built on HTTP. There are JSON implementations…

Yes, developers love it; just as they loved XML five years ago. I'm a little wary of following fashion, given the relatively long time scales of HTTP evolution. While I think most will agree that the current header syntax is far from perfect, one of its main problems is that there's a lot of differences between existing header parsing. Adding a new formatting convention will not help that situation.

> A JSON string: is enclosed in double quotes; uses \" and \\ to represent " and \; uses six other \x sequences for other chars; and allows \uXXXX as an escape sequence for any Unicode character [jso…

Just an observation; this form of Unicode support is an encoding -- just as 5987 is an encoding. You can have aesthetic concerns about one or the other, but they fulfil the same job, and aren't seriously visible to users (who see the decoded artefact) or developers (who use a library, in both cases).

> RFC5987 "Character Set and Language Encoding for HTTP Header Field Parameters" already offer one way to represent any Unicode string in a HTTP header parameter value, eg foo*=UTF-8''coll%C3%A8gues.…

New headers (or new parameters to existing ones) don't have to support two parameter names -- if they want to specify that only foo* be present, so be it, AIUI. That puts it to one syntax and one escaping mechanism.

> I think the considerations for new headers (issue #231), and advice on defining auth scheme parameters (issue #320), should consider how to support Unicode parameter values -- and json-string would…

Considering that there's effectively zero experience with creating JSON-based HTTP headers, it's something we're not likely to do in HTTPbis. However, nothing's stopping you from defining a header with JSON syntax; 231 is trying to encourage people to think about interoperability, leveraging existing libraries, convergence, etc. when they create a header. 

There are some things to consider if you do use JSON; e.g.

Foo: { "a": 1,
Foo: "b", 2 }

is dangerous, because it can be collapsed by an intermediary to:

Foo: {"a": 1,, "b", 2}

... which is invalid JSON. That means that authors would have to be very careful to do:

Foo: {"a": 1
Foo: "b", 2}

if they want to use multiple lines (which is somewhat likely, given that implementations often have length limits for HTTP headers).

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: "Manger, James H" <[email-masked]>
Date: Mon, 31 Oct 2011 15:14:02 +1100

> There are some things to consider if you do use JSON; e.g.

I am only suggesting the string syntax from JSON, not arbitrary JSON (objects, arrays, numbers etc).

--
James Manger

---

From: David Morris <[email-masked]>
Date: Sun, 30 Oct 2011 21:33:35 -0700 (PDT)

Using the JSON string syntax w/o designing in support for JSON objects 
doesn't make sense to me. I suspect it would be quite confusing and many
would assume that JSON objects were supported.

---

From: "Thomson, Martin" <[email-masked]>
Date: Mon, 31 Oct 2011 13:55:48 +0800

On 2011-10-31 at 15:33:35, David Morris wrote:
> doesn't make sense to me. I suspect it would be quite confusing and 
> many would assume that JSON objects were supported.

I wouldn't have made that conclusion.  You wouldn't even have to mention JSON in any specification, just to add the syntactical constructs so that the string syntax becomes identical.

But Mark said it best.  The encoding in 5987 is sufficient.  I tend to think that it loses a lot in the human-readable area, which is still important to me, but it is hard to justify adding another mechanism where an adequate one already exists.

Sadly, the aesthetic concern only has any sway once all the other concerns are addressed.

--Martin

---

From: "Manger, James H" <[email-masked]>
Date: Mon, 31 Oct 2011 17:34:45 +1100

> But Mark said it best.  The encoding in 5987 is sufficient.
> I tend to think that it loses a lot in the human-readable area,
> which is still important to me, but it is hard to justify
> adding another mechanism where an adequate one already exists.

If the RFC5987 encoding is sufficient, then httpbis should recommend RFC5987 syntax instead of token and quoted-string (for new headers and auth scheme parameters). It is having to add RFC5987 as an additional syntax just to get Unicode support that is most objectionable.

Mark said:
>> New headers (or new parameters to existing ones) don't have to
>> support two parameter names -- if they want to specify that only
>> foo* be present, so be it, AIUI. That puts it to one syntax and
>> one escaping mechanism.

That might work. I doubt groups defining new parameters will be that happy that some names have to end with a * (and values start with UTF-8''), particularly when Unicode support is nice but not crucial (at least initially). It would be better if Unicode wasn't a 2nd-class citizen. That is, it would be better if the recommended way to include a string parameter value automatically supported Unicode.

--
James Manger

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 31 Oct 2011 19:54:52 +1100

> If the RFC5987 encoding is sufficient, then httpbis should recommend RFC5987 syntax instead of token and quoted-string (for new headers and auth scheme parameters). It is having to add RFC5987 as a…

Unicode isn't necessary for the majority of headers; it's only when there's user-visible data that it's necessary. Mandating 5987 -- or any encoding -- across the board would be overkill.

> That might work. I doubt groups defining new parameters will be that happy that some names have to end with a * (and values start with UTF-8''), particularly when Unicode support is nice but not cr…

See above. HTTP headers aren't intended to be exposed to end users, so I'm not sure why a * matters that much, in the big scheme of things.

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Mon, 31 Oct 2011 13:47:18 +0100

On 2011-10-30 14:07, Manger, James H wrote:
 > ...

Here's a list of problems I see with this proposal:

- Scope - the proposed syntax overloads quoted-string, potentially 
changing the interpretation of existing content, thus I don't think we 
*can* do this as part of HTTPbis.

- Adding a different type of quoted-string might make things more 
confusing; for the RFC5987 encoding it's at least easy to understand 
when it's in use.

- the JSON \u format doesn't really use Unicode but UCS-2 code points; 
which means that senders and receivers will need to understand surrogate 
pairs; see also <https://tools.ietf.org/html/rfc5137#section-5.1> for 
context (that proposal adds additional delimiters to avoid the variable 
length issue)

Best regards, Julian