Subject: Re: [jose] [Acme] Signed JSON document / Json Content Metaheader / JSON Container

From: Anders Rundgren <[email-masked]>
Date: Wed, 28 Jan 2015 22:04:38 +0100

Hi Lists,

As expected the flood of various JSON clear-text signature schemes would come sooner rather than later :-)

Which one is best?  Well that would be comparing oranges with apples and personally I prefer strawberries!

Cheers,
Anders

On 2015-01-28 19:14, Phillip Hallam-Baker wrote:
> All,
>
> I would like to propose a spec similar to the JSON log file spec but with the purpose of being a container for JSON signatures. The immediate application for this might be for ACME or other JSON/Rest Web services.
>
> The elevator pitch for the idea is that a JSON Container would be a JSON Metadata blob with the content being described appended to the end in a way that makes the separation between content and metadata simple and unambiguous.
>
> <JSON-BLOB> [Separator] <content-blob>
>
> Advantages:
>
> * Can read the metadata for a file etc with a plain ASCII editor or command line tools like cat, more, etc.
>
> * Avoids the need to BASE64 armor the content. So if the content is JSON or other ASCII/Unicode text it remains readable in an ASCII/Unicode editor.
>
> * Can add signatures, digests and other metadata items to content in a simple regular fashion.
>
> * Content and metadata can be transported as a simple regular blob that is fully abstracted from and independent of the transport and without the need for canonicalization, XPath or other complications.
>
>
> Right now, the IETF spec for describing content metadata is based on RFC822. And by 'based on', I mean that it is a find the needle in the RFC haystack approach. RFC822 style content headers as currently used have several disadvantages:
>
> * It is a flat structure, headers can have attributes but trying to add more structure is painful.
>
> * It is a historical artifact and content metadata headers are mixed in with protocol metadata headers without rhyme or reason.
>
> * It is not JSON and JSON is the spec we are now converging on for this type of work. It has all the expressive capabilities of XML and ASM.1 without the insanities and stupid. It is as easy to read and write as RFC822 but without the limitations.
>
>
> The spec itself would be simple, its basically draft-ietf-json-text-sequence applied to a sequence of one json object and one content blob.
>
> JSON sequence specifies that "A JSON text sequence consists of any number of JSON texts, all encoded in UTF-8, each prefixed by an ASCII Record Separator (0x1E), and each ending with an ASCII Line Feed character (0x1A)"
>
> This is not ideal for JSON Container. We would prefer that the character which is illegal in a JSON document be the one that is the separator between the header and content.
>
> All indexes used in the metadata header are calculated from the Record Separator byte, the first byte following being byte 0 and so on.
>
>
> Applying this in a Web Service is very simple, our messages now have the form:
>
> POST / HTTP/1.1
> Host: example.com <http://example.com>
> Content-Type: application/json-container
> Content-Length: 666
>
> { "Signature" : "wefwkjefkljwehfjklwhejkflh" }
>
> <0x1E>{ "Service-Type" : "acme.ws <http://acme.ws>",
>     "Transaction-ID" : "2h23roih23oih23orh",
>     "Register" : { ....<web service parameters here> ... } }
>
> Where <0x1E> is the record separator character.
>
> The scope of the signature is the content, the whole content, AND NOTHING BUT THE CONTENT. Signing headers is a mugs game. Put all the information that matters into one blob and sign the blob. Do not present developers with an IKEA self-assembly job.
>
> So to prevent protocol substitution attacks, the service identifier and the method name wrap the method parameters. In a response, the service identifier, response code and transaction ID wrap the response data for the same reason.
>
>
> Applying the scheme to stored data in a file system is a little different. First we have to develop a file extension convention:
>
> file.jpg  becomes file.jpg.jsonc
>
> Many jsonc aware applications will have to share data with applications that are not aware. So we need the option of storing metadata in a separate file when that is convenient:
>
> The metadata for file.jpg is stored in file.jsonm
>
> In some circumstances we might want to keep the metadata for all the files in a directory or tree in one file, a metadata container. This could be jsonmc.jsonmc appearing in either the directory a file is in or one of the directories above it.
>
> A metadata container would be a JSON container and begin with the usual metadata header for that file followed by a JSON sequence of entries per file. If this contains a signature, the file entries can just be digests.
>
>
> Notice how we have just developed a format that allows us to sign a complete code distribution including content data very easily. This is obviously out of scope for ACME but the fact that an approach transfers so neatly to a completely different problem suggests that it is the right track.
>
>
>
>
> _______________________________________________
> Acme mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/acme
>

---

From: Anders Rundgren <[email-masked]>
Date: Thu, 29 Jan 2015 06:38:45 +0100

On 2015-01-29 05:25, Fraser Tweedale wrote:
Hi Fraser,

 > I agree the JSON ship has sailed, but there *are* drawbacks.

The JSON ship may have sailed but it is actually more like a dinghy when
look closer making a turn or two fairly realistic.

Using predictable serialization which is anything but rocket-science,
"canonicalization" can [often] be reduced to "stringify":

https://openkeystore.googlecode.com/svn/resources/trunk/docs/jcs.html

https://code.google.com/p/openkeystore/source/browse/python/trunk/src/org/webpki/json
https://code.google.com/p/openkeystore/source/browse/resources/trunk/docs/JCSDemo.cs

Anders

> On Wed, Jan 28, 2015 at 11:03:05PM -0500, Phillip Hallam-Baker wrote:
>> On Wed, Jan 28, 2015 at 9:57 PM, Nat Sakimura <[email-masked]> wrote:
>>
>>> On a side note: if such a spec is to be defined here, IMHO, it should use
>>> the algorithms and probably header parameters specified by JWA, etc. It
>>> should limit the scope to payload processing and expression of the entire
>>> thing in JSON Log like format, and leave the rest to JOSE.
>>>
>>
>> Absolutely. In fact that is why I am not raising it in JOSE as that just
>> provides the format for the main crypto attributes.
>>
>>
>>
>>
>>
>>
>>> On Thu Jan 29 2015 at 11:51:24 Manger, James <
>>> [email-masked]> wrote:
>>>
>>>> A signed JAR file meets some of these requirements.
>>>>
>>>> Metadata and signatures are in extra files in the ZIP archive:
>>>> META-INF/MANIFEST.MF, META-INF/MYKEY.SF, META-INF/MYKEY.RSA.
>>>>
>>>> Content is the other files in the archive.
>>>>
>>>> It is not JSON of course, and the signature & certs are packaged in
>>>> ASN.1, but it is a useful comparison. It avoids BASE64 on the content; can
>>>> adds signatures, digests, and other metadata; can transport content and
>>>> metadata as a regular blob (*.jar file); can sign complete code
>>>> distributions.
>>>>
>>>>
>>>>
>> I have used signed jar files. But Sun rather poisoned the well there by
>> suing Microsoft over control of Java followed up by further lawsuits from
>> Oracle.
>>
>> I can't imagine anyone is going to accept Jar or anything involving
>> assinine.1 as a wire format for packaging. Those days are long past. The
>> way you get coherence is to pick one encoding and stick to it. JSON seems
>> to have been the one we picked. It has all the functionality offered by the
>> alternatives and none of the drawbacks.
>
> There are ∞ valid ways to serialise a JSON object.  If a JSON object
> is signed over, it must be canonicalised, or signed over and
> transported in serialised form.  JOSE takes the latter approach
> (base64url-encoded serialised JSON object, inside a JSON object),
> while JWK thumbprint draft uses canonicalisation.  ASN.1 encodings
> and deterministic binary serialisations can avoid the need for these
> sorts of hacks.
>
> I agree the JSON ship has sailed, but there *are* drawbacks.
>
> Regards,
> Fraser
>

---

From: Justin Richer <[email-masked]>
Date: Thu, 29 Jan 2015 08:14:42 -0500

Relying on side-effects of a handful of contemporary implementations is 
dangerous at best and absolutely foolhardy at worst, especially when it 
comes to security systems. You *need* to have a formal canonicalization, 
normalization, or serialization in order for these things to work in 
practice.

Otherwise, you're betting on luck, and that's just daft.

  -- Justin

On 1/29/2015 12:38 AM, Anders Rundgren wrote:
> jose mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jose

---

From: Anders Rundgren <[email-masked]>
Date: Thu, 29 Jan 2015 15:26:04 +0100

> normalization, or serialization in order for these things to work in

You are right: JCS isn't guaranteed to run on any JSON implementation.

However, JCS doesn't rely on side-effects, it has a formal normalization
scheme (the description may be crap but there is one at least), which is
explicitly implemented in two (yes, only two) reference implementations.

Then I have with the help of available documentation and communication
with the authors, verified that two "commercial" JSON implementations
also can support "predictive serialization" which BTW is a requested
item by the general JSON community for the simple reason that humans
(unlike computers) find it somewhat odd that if you define properties
A,B,C they come out as A,C,B.

The only (known NB) "fly in the soup" is that floating-point serialization
usually requires some specific measures to work as mandated by the JCS
normalization scheme.  If this is a show-stopper and can't be corrected,
your best option is using JWS of course.  The idea was never ever to
"compete" but to "complement".  I have adjusted the JCS during the voyage
so that it nowadays reuses JOSE algorithms, curves, public key objects, and
signature blob format.  "Crypto compatible".

> Otherwise, you're betting on luck, and that's just daft.

Since I don't consider myself a gambler, I guess I have to settle on "daft" then :-)

Anders

---

From: Fraser Tweedale <[email-masked]>
Date: Thu, 29 Jan 2015 16:59:04 +1000

> "canonicalization" can [often] be reduced to "stringify":
This means that you either:

    a) have a JSON library that serialises objects in precisely the
       manner required, or affords users the degree of control needed
       to achieve it, or
    b) must write the serialisation function yourself

Although b) is not rocket-science, the format is hostile to code
reuse.  JOSE made a different tradeoff - you can use any JSON
library but the format is more complex.  One drawback is traded off
against another.

JSON has benefits compared with ASN.1, but using it presents you
with an unavoidable choice between significant drawbacks.  This fact
ought not be glibly ignored.

Regards,
Fraser

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 29 Jan 2015 08:53:58 -0500

> Otherwise, you're betting on luck, and that's just daft.

-1E200

Canonicalization is the stupidest idea in computer security. It is never
ever necessary and never ever implemented reliably.

A digital signature signs a sequence of bits. So if you ever want to check
a signature again, make sure you keep hold of your original sequence of
bits. Simple!

I see people say that canonicalization is 'essential' in every discussion
of signatures. What I have never seen is an example of something that is a
reasonable thing to do that goes wrong if you don't have C15N.

And by reasonable, I do not mean 'take a cert, store it in X.500 directory,
reassemble'. DER encoding is the stupidest stupid of all the steaming piles
of stupid in ASN.1. BER meets the needs of X.509 just as well, as was
proved by the fact that the Web ran quite happily on BER encoded certs
until some spoilsport let on what we had been doing.

The reason I am proposing JSON Container is precisely to avoid the need for
canonicalization. That does not work in XML Dig sig and it won't work for
JSON.

My proposal has three parts:

1) A blob of data where the only requirement is that it must be a valid
JSON encoding. Changing this is permitted. Want to add another signature,
go ahead! So not only are syntactic differences allowed, semantic changes
are allowed as well.

2) A separator marker to unambiguously define the end of (1) and the start
of (3)

3) The sequence of bits that was signed.

A signature in (1) cannot refer to any part of (1), it can only reference
(3) and is by default the whole of (3) but could be a well defined range
inside (3) instead.

That is it, no JSON canonicalization or unique serialization required. This
is a proposal for wrapping arbitrary content data with a wrapper that
contains JSON metadata which might include signatures. There is no reason
for either the signature or the data to be in a canonical form.

If you want to have signed metadata, you would need to first wrap the
content with a JSON Container with the metadata and then wrap that with a
second container with the signature.

---

From: David Waite <[email-masked]>
Date: Thu, 29 Jan 2015 13:11:46 -0700

I believe Justin was saying you need canonicalization in order to make an embedded signature work in an interoperable way, since the data structure will be read, manipulated and serialized out before the signature is verified.  I don’t think he was advocating an embedded signature approach.

If the consensus of this WG was that canonicalization was not a barrier to interoperability and/or a security risk, I believe JWS would likely embed signatures and require canonicalization. Instead, the current approach is that of a separate signature against binary data that in some cases happens to be parsable JSON.

WRT JSON container, it in my opinion looks like it would be best served as an alternate serialization of JWS. Instead of the compact JWS form of  

	b64(metadata) “.” b64(data) “.” b64(signature),

 you would likely have 

	<magic number?> <metadata> 0x1E <fixed width or width-prefixed signature> 0x1E <data>

 - where metadata is defined to be JSON, and data can be whatever your heart desires.

-DW

> On Jan 29, 2015, at 6:53 AM, Phillip Hallam-Baker <[email-masked]> wrote:
> 
> On Thu, Jan 29, 2015 at 8:14 AM, Justin Richer <[email-masked] <mailto:[email-masked]>> wrote:
> Relying on side-effects of a handful of contemporary implementations is dangerous at best and absolutely foolhardy at worst, especially when it comes to security systems. You *need* to have a formal canonicalization, normalization, or serialization in order for these things to work in practice.
> 
> Canonicalization is the stupidest idea in computer security. It is never ever necessary and never ever implemented reliably. 
> 
> A digital signature signs a sequence of bits. So if you ever want to check a signature again, make sure you keep hold of your original sequence of bits. Simple!
> 
> I see people say that canonicalization is 'essential' in every discussion of signatures. What I have never seen is an example of something that is a reasonable thing to do that goes wrong if you don't have C15N.
> 
> And by reasonable, I do not mean 'take a cert, store it in X.500 directory, reassemble'. DER encoding is the stupidest stupid of all the steaming piles of stupid in ASN.1. BER meets the needs of X.509 just as well, as was proved by the fact that the Web ran quite happily on BER encoded certs until some spoilsport let on what we had been doing.
> 
> 
> The reason I am proposing JSON Container is precisely to avoid the need for canonicalization. That does not work in XML Dig sig and it won't work for JSON.
> 
> 1) A blob of data where the only requirement is that it must be a valid JSON encoding. Changing this is permitted. Want to add another signature, go ahead! So not only are syntactic differences allowed, semantic changes are allowed as well.
> 
> 2) A separator marker to unambiguously define the end of (1) and the start of (3)
> 
> A signature in (1) cannot refer to any part of (1), it can only reference (3) and is by default the whole of (3) but could be a well defined range inside (3) instead.
> 
> That is it, no JSON canonicalization or unique serialization required. This is a proposal for wrapping arbitrary content data with a wrapper that contains JSON metadata which might include signatures. There is no reason for either the signature or the data to be in a canonical form. 
> 
> If you want to have signed metadata, you would need to first wrap the content with a JSON Container with the metadata and then wrap that with a second container with the signature.

---

From: "Richer, Justin P." <[email-masked]>
Date: Thu, 29 Jan 2015 21:28:46 +0000

I think you missed my point. I wasn't saying that I *liked* canonicalization and normalization but rather that you *need* either it or something *equivalent* in order to get the same set of bits out of either end. Like you, what I was saying was that you need *something* that makes your bit stream stable. I'm actually in favor of the JOSE approach, which is exactly to keep the original bits around by protecting them with Base64url over the network so that they don't get munched. As an implementor, this is fantastic, as it makes my code much simpler in terms of both generation and consumption. I have the JSON objects where I need them and the raw streams where I need them, and I can easily keep them separated with a reasonable assumption about them not being confused with each other. For heaven's sake, don't do canonicalization. But at the same time, don't assume that a JSON blob is going to be kept pristine as a string by any JSON-processing system.

So we're actually in violent agreement on this point. In my view, JOSE's approach of keeping the original bits around is the desirable one by far.

 -- Justin

On Jan 29, 2015, at 8:53 AM, Phillip Hallam-Baker <[email-masked]<mailto:[email-masked]>> wrote:

On Thu, Jan 29, 2015 at 8:14 AM, Justin Richer <[email-masked]<mailto:[email-masked]>> wrote:
Relying on side-effects of a handful of contemporary implementations is dangerous at best and absolutely foolhardy at worst, especially when it comes to security systems. You *need* to have a formal canonicalization, normalization, or serialization in order for these things to work in practice.

-1E200

If you want to have signed metadata, you would need to first wrap the content with a JSON Container with the metadata and then wrap that with a second container with the signature.
_______________________________________________
jose mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/jose

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 29 Jan 2015 19:47:52 -0500

On Thu, Jan 29, 2015 at 4:28 PM, Richer, Justin P. <[email-masked]>
wrote:

>  I think you missed my point. I wasn't saying that I *liked*
> canonicalization and normalization but rather that you *need* either it or
> something *equivalent* in order to get the same set of bits out of either
> end. Like you, what I was saying was that you need *something* that makes
> your bit stream stable.
>

Sure but that does not need to be in the signature format. I would prefer
to simply tell people that they have to get this package from A to B with
absolutely no modifications or it won't work.

> I'm actually in favor of the JOSE approach, which is exactly to keep the
> original bits around by protecting them with Base64url over the network so
> that they don't get munched. As an implementor, this is fantastic, as it
> makes my code much simpler in terms of both generation and consumption.
>

I don't like that for the general case though as the main advantage of JSON
is that it is human readable and I can't translate back and forth between
Base64 in my head any more.

> I have the JSON objects where I need them and the raw streams where I need
> them, and I can easily keep them separated with a reasonable assumption
> about them not being confused with each other. For heaven's sake, don't do
> canonicalization. But at the same time, don't assume that a JSON blob is
> going to be kept pristine as a string by any JSON-processing system.
>

That is one of the reasons for insisting on a sharp division between 'that
which is signed' and 'the JSON bit'.

> So we're actually in violent agreement on this point. In my view, JOSE's
> approach of keeping the original bits around is the desirable one by far.
>

I thought you might intend something like that but its not what I think you
said and it gave me an opportunity to use the phrase steaming piles of
stupid :-)