Subject: Re: [jose] Direct Compact Serialization

From: ⌘ Matt Miller <[email-masked]>
Date: Fri, 3 Apr 2015 10:30:47 -0600

On 3/25/15 10:14 AM, Phillip Hallam-Baker wrote:
> The revised draft is now up. It is essentially a one pager. Which 
> turns out to be five with the boiler plate and IANA section.
> 
> http://tools.ietf.org/html/draft-hallambaker-joseunencoded-01
> 
> 2. Serialization
> 
> In the JWS Direct Serialization, no JWS Unprotected Header is
> used. In this case, the JOSE Header and the JWS Protected Header
> are the same.
> 
> In the JWS Direct Serialization, a JWS is represented as the 
> concatenation:
> 
> UTF8(JWS Protected Header)) || '.' || (JWS Payload) || '.' ||
> 
> (JWS Signature)
> 
> The calculation of the signature is performed over the octet
> sequence that corresponds to the concatenation:
> 
> UTF8(JWS Protected Header)) || '.' || (JWS Payload) || '.'
> 
> _______________________________________________ jose mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/jose
> 

Here is my review of draft-hallambaker-joseunencoded.

In summary, I think this draft has serious issues.

I assume the following:

    UTF8(JWS Protected Header)) || '.' || (JWS Payload) || '.' || (JWS
Signature)

Means that the JWS Protocted Header is the JSON-Object structure
(possibly, or not, including insignificant-to-JSON whitespace), and
the JWS Payload and JWS Signature are the exact octet values -- no
escaping or other armoring (e.g., base64url encoded).

Using '.' as a delimiter without preparation of the inputs can lead to
ambiguous results.  A '.' in the JWS Payload makes it impossible to
determine where the payload ends and the signature or MAC begins.
Similarly, a '.' in the JWS Signature causes the same problem for
searching from the end.  A '.' in the JWS Protected Header requires
non-trivial preprocessing of the input in order to determine where the
JSON structured JWS Protected Header ends and the JWS Payload begins.
 Some form of TLV-like container would help alleviate this problem,
such as CBOR.

Using the JWS Protected Header without any form of canonicalization or
other armoring can lead to signature/MAC verification failures due to
differences in how various JSON implementations parse and serialize.
Some generators add extra whitespace; many parsers don't retain the
order of name/value paris within a JSON-Object; others; and there are
other subtle differences that can cause serious problems with
on-the-wire consistency.  This problem might be mitigated by requiring
consumers to retain the original "UTF8(JWS Protected Header)" value as
well as requiring producers to remove all insignificant whitespace
before using the UTF8-encoded header to generate the signature/MAC and
emitting the Compact Direct Serialization.  Verifying this on the
consumer side would be expensive.  I'll note that CBOR provides
canonicalizaation guidance; this JSON problem seems to be far better
mitigated there.

- -- 
- - m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 3 Apr 2015 10:26:15 -0700

I share Matt's reservations about the delimiting here.  That part is
fundamentally broken.

I don't share his concerns about the canonicalization.  That denies
PHB's basic thesis, which I agree with.  If you have a transport that
is 8-bit clean, you can trivially preserve the JSON, as serialized and
you have exactly as many issues as JOSE has already.  Saving the
base64 encoding probably saves more bytes than moving to a completely
new format.

If you really cared about saving bytes, you would drop the text labels
and move to a schema-based binary encoding, or at least integer keys.
A format without OIDs could be extremely compact.  But part of the
point of JOSE is to avoid that muck, and the gains aren't significant.

I don't see a significant gain in moving to COSE, unless you have a
lot more base64 to remove from the protected header.  But if you are
shipping a lot of bytes in the header, I'd suggest that you might be
past the point where you should be concerning yourself with saving
space.

On 3 April 2015 at 09:30, ⌘ Matt Miller <[email-masked]> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
> Comment: GPGTools - https://gpgtools.org
>
> iQEcBAEBCgAGBQJVHsA3AAoJEDWi+S0W7cO1qh4H/2rw97xHZeKU///CBHMKB7v8
> Z+vKEoHFVzW9dbfnJyDdSRiWUbaSw+M+mZhfjgm8rJq4h/dOZzt0+7+kw+EZnegH
> to6q+x+2DZyO2tiPlVwT/TygUOVkU5q12dmwCS6IIVITrA9vfiWonCMRYV1mPx1d
> S74LIx8n2TTbJ8rxNWw7ALO7VT++X+flSKSEONxYyRks3tDT8bmYOqjUaqwnUiiO
> GoyAki663KsbY75C+vdqJ0fKgh2ZRLG50PFe1xXD92vI/Vvj09VclZYqWSwpxZaC
> 5hVTZ1yGJTCrkpEjppmDmpty9y0j6sbcpYBd0ZdUYSXRR7fu+yHX68j6P6wMmC0=
> =dSL4
> -----END PGP SIGNATURE-----
>
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jose

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 6 Apr 2015 19:20:24 +0000

>I share Matt's reservations about the delimiting here.  That part is

As do I.  json-text-sequences (RFC 7464) was designed for this sort of thing, as Justin Richer mentioned. You could at least take the idea from there of using an ASCII RS as the separator, since it is not valid JSON.

>you have exactly as many issues as JOSE has already.  

It would have to be both 8-bit clean, tolerate newlines, and if the payload isn't detached, you're severely limited in what could be sent.  Limited to the point where I'm worried the approach would work terribly well in practice.

Do we have one of these transports in mind, by the way?

>Saving the base64 encoding probably saves more bytes than moving to a completely
>new format.

Probably, but I'm also worried the need to keep the original text around after parsing to avoid the need for canonicalization.  The extra memory associated with that isn't a good fit for smaller devices, and is an attractive nuisance for developer shortcuts.

>and move to a schema-based binary encoding, or at least integer keys.

ASN.1 is a schema-based binary encoding, so we already have one of those in CMS.  Integer keys help, but help a lot more in an encoding that uses 1 byte for the oft-used small integers (as CBOR does).

>point of JOSE is to avoid that muck, and the gains aren't significant.

Aren't OID's just a particular approach for minting unique byte strings?  Registries and tag: URNs, are all other approaches in this space.

>past the point where you should be concerning yourself with saving

I only see a gain in moving to COSE if you're going to use it throughout.  For non-clean transports like http: URLs, you can always base64 at the end, and save some processing time and space.

-- 
Joe Hildebrand

---

From: Martin Thomson <[email-masked]>
Date: Mon, 6 Apr 2015 13:23:58 -0700

> Do we have one of these transports in mind, by the way?

At least one: HTTP

CoAP too, I guess.

> Probably, but I'm also worried the need to keep the original text around after parsing to avoid the need for canonicalization.  The extra memory associated with that isn't a good fit for smaller de…

I get the nuisance factor, but shortcuts only lead to signature
verification failure (including failure to verify).  Given that jose
has the exact same problem, I'm not super-concerned.

> ASN.1 is a schema-based binary encoding, so we already have one of those in CMS.  Integer keys help, but help a lot more in an encoding that uses 1 byte for the oft-used small integers (as CBOR doe…

Don't make me get out the schema vs. encoding bat.  PER allows types
to consume <1 byte if you want to get aggressively parsimonious.

> Aren't OID's just a particular approach for minting unique byte strings?  Registries and tag: URNs, are all other approaches in this space.

Indeed.  The reason I mentioned them is that they are especially
wasteful of bytes.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 14 Apr 2015 10:21:48 -0400

> As do I.  json-text-sequences (RFC 7464) was designed for this sort of thing, as Justin Richer mentioned. You could at least take the idea from there of using an ASCII RS as the separator, since it…

I think the RS character makes better sense. The only reason I did not
use it before was to try to make the changes as minimal as possible
and see what people argued for.

> It would have to be both 8-bit clean, tolerate newlines, and if the payload isn't detached, you're severely limited in what could be sent.  Limited to the point where I'm worried the approach would…

HTTP is 8-bit clean. Always has been. Anything is not MUST break. End of story.

I am not worried at all about signatures breaking when they go through
non compliant HTTP proxies. That is what the signature is there for.

Streaming is a much harder proposition. Because many implementations
of the HTTP streaming spec are broken and don't handle trailer headers
that I added for the express purpose of supporting streamed
signatures.

It is only useful to put the header at the end if you have streaming
capability. And right now you would have to have two nested chunking
schemes or write a new chunking scheme for HTTP.

Either is possible of course. But once you get into chunking schemes
then you might as well use JSON-B which supports chunked binary blobs
as a replacement for base-64 encoded text. Or some other binary
encoding of JSON.

Fortunately it is not one that is in the slightest bit relevant to
ACME or 95% of Web Service transactions. The messages are all compact
enough to process in one chunk.

> Do we have one of these transports in mind, by the way?

HTTP

> Probably, but I'm also worried the need to keep the original text around after parsing to avoid the need for canonicalization.  The extra memory associated with that isn't a good fit for smaller de…

I don't see the problem. The proper tool chain is

1) Parse signature header, identify data bytes
2) Check signature
3) Pass data bytes to JSON parser.

There is no need to keep the data bytes after the signature is checked
unless they are going to be passed on to another system. If you have a
constrained device it should not be an intermediate party in a three
party transaction.

> ASN.1 is a schema-based binary encoding, so we already have one of those in CMS.  Integer keys help, but help a lot more in an encoding that uses 1 byte for the oft-used small integers (as CBOR doe…

Or use a dictionary to compress the tags with an option to
pre-exchange which JSON-C does.

The dictionary can then be compiled incrementally or exchanged inline
as the circumstances require.

> I only see a gain in moving to COSE if you're going to use it throughout.  For non-clean transports like http: URLs, you can always base64 at the end, and save some processing time and space.

If you are using a data encoding that has direct support for binary,
there is no need for this particular scheme.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 14 Apr 2015 09:52:04 -0700

> that I added for the express purpose of supporting streamed

That's not quite right.  HTTP implementations frequently stream
request and response bodies.  The problem that people encounter is
where intermediaries (which include the stack you are using) buffer.
They are entitled to do so.  That causes some people heartburn, and it
breaks all the broken things that people think they should be able to
do.  e.g., Server Sent Events (http://www.w3.org/TR/eventsource/)

Trailers are broken in the sense that they can't be relied upon.  That
much is true.