Subject: Re: [jose] #26: Allow for signature payload to not be base64 encoded

From: Richard Barnes <[email-masked]>
Date: Tue, 2 Jul 2013 18:20:21 -0400

Glad that helped.  Detailed examples below.

Couple of high-level points to help the examples make sense:
1. Define a "cenc" header parameter, for "Content ENCoding", with one
value, "b64u".  When present in the JSON serialization, the "payload" field
is base64url encoded; when it is absent, the "payload" field represents the
payload directly (as a UTF-8 string).
2. For the compact serialization, "cenc" is always "b64u".
3. When protected a header are present (always for the compact
serialization), the JWS Signing Input is base64(protected).base64(payload),
as it is in -11.  When no protected header is present, the JWS Signing
Input is simply the octets of the payload.

With that in mind, let's look at a few test cases.  We'll use the following
inputs:

Header: {"alg":"HS256","kid":"2"}
== 7b22616c67223a224853323536222c226b6964223a2232227d (hex UTF-8)
== eyJhbGciOiJIUzI1NiIsImtpZCI6IjIifQ (base64 UTF-8)

Key (kid=2): 0x0001020304050607 (8 octets)

Payload: "café"
== 636166c3a9 (hex UTF-8)
== Y2Fmw6k (hex UTF-8)

With those inputs, let's consider 4 cases that show the possibilities
discussed above.

### CASE 1: Compact / protected / b64u

eyJhbGciOiJIUzI1NiIsImtpZCI6IjIifQ
.Y2Fmw6k
.QUoMVpjxLXjH1ZAX9tw4QEPnL6G_Hvd4EjD-JsHilyk

No change here.  The compact case just represents the special case where
the payload is base64-encoded and all the headers are protected.  Thus, we
use the base64.base64 form for the JSON Signing Input.
JWS Signing Input: eyJhbGciOiJIUzI1NiIsImtpZCI6IjIifQ.Y2Fmw6k (as a UTF-8
string)
== 65794a68624763694f694a49557a49314e694973496d74705a434936496a496966512e5932466d77366b
(hex)

### CASE 2: JSON / protected / b64u

{
  "unprotected":{"cenc":"b64u"},
  "protected":"eyJhbGciOiJIUzI1NiIsImtpZCI6IjIifQ",
  "payload":"Y2Fmw6k",
  "signature":"QUoMVpjxLXjH1ZAX9tw4QEPnL6G_Hvd4EjD-JsHilyk"
}

This is just the JSON equivalent to Case 1.

### CASE 3: JSON / unprotected / b64u

{
  "unprotected":{"alg":"HS256","kid":"2","cenc":"b64u"},
  "payload":"Y2Fmw6k",
  "signature":"X-GwEVfz9WEi0UhrmK1muKXecqxMipngSL_ZenXw5Lo"
}

Here we've removed the integrity protection from the headers (because who
needs it? ;) ).  So the signature is different, because JWS Signing Input
here is just the payload.
JWS Signing Input: café (as a UTF-8 string)
== 636166c3a9 (hex)

### CASE 4: JSON / unprotected / plain

{
  "unprotected":{"alg":"HS256","kid":"2"},
  "payload":"café",
  "signature":"X-GwEVfz9WEi0UhrmK1muKXecqxMipngSL_ZenXw5Lo"
}

This object is equivalent to Case 3, but with the payload expressed
directly as a JSON string instead of base64-encoded.  And, if you ask me,
the prettiest and most human-readable of the whole menagerie here :)

I've written a script to generate these test cases, in case you want to
play with this some more:
<http://test.polycrypt.net/temp/hello-jose.html>

Hope this helps,
--Richard

On Fri, Jun 28, 2013 at 7:04 PM, Dick Hardt <[email-masked]> wrote:

> Hi Richard, that helps.
>
> Our you looking at a way of effectively signing a JSON payload? A
> complete, real world example would help alot.
>
>
> On Fri, Jun 28, 2013 at 3:57 PM, Richard Barnes <[email-masked]> wrote:
>
>> Hey Dick,
>>
>> I guess I should have been clearer that I was only talking about the JSON
>> serialization.  My proposed change would have no impact on the compact
>> serialization.  Hopefully that answers your questions about separators,
>> etc.; they payload would just be a normal JSON string.
>>
>> --Richard
>>
>>
>> On Thu, Jun 27, 2013 at 7:56 PM, Dick Hardt <[email-masked]> wrote:
>>
>>>
>>>
>>>
>>> On Thu, Jun 27, 2013 at 3:24 PM, Richard Barnes <[email-masked]> wrote:
>>>
>>>> I think there are two separable issues here, both of which Mike and I
>>>> probably have divergent opinions on :)
>>>> 1. Syntax: Should it be possible for the "payload" field JWS to be a
>>>> JSON string (instead of base64) when the payload is simply a UTF-8 string?
>>>>
>>>
>>> One of the goals of the token was that it not require encoding for it to
>>> be transported in a request. IE that it be safe and SIMPLE to include in an
>>> HTTP header or in a URL
>>>
>>>
>>>> 2. Crypto: Must the JWS Signing Input be base64url encoded.
>>>>
>>>> The current answers to these questions are:
>>>> 1. It is not possible.  Payload is always base64url encoded.
>>>> 2. It is always base64url encoded.
>>>>
>>>> The answers should be:
>>>> 1. The payload should not be base64url encoded unless it cannot be
>>>> represented as a JSON string.
>>>>
>>>
>>> How does an implementation determine which parts are separated? Are '.'
>>> escaped? Does the implementation need to parse the JSON to determine a
>>> valid separator? How does it know the end of the JSON? That sounds really
>>> hard to implement, or I am misunderstanding what you are proposing.
>>>
>>>
>>>
>>>> 2. The JWS Signing Input should not be base64url encoded unless there
>>>> is a JWS Protected Header
>>>>
>>>> Neither of these are complicated changes:
>>>> 1. Add a "b64" header parameter that indicates that the payload is
>>>> base64url-encoded binary, as opposed to a UTF-8 string.  Specify that this
>>>> parameter is on by default in compact-formatted JWS objects.
>>>>
>>>
>>> And how is the header separated from the payload?
>>>
>>>
>>>>  2. Modify the signing/verification instructions to switch based on
>>>> the presence of a JWS Protected Header: "If the JWS Protected Header is
>>>> absent or empty, then the JWS Signing Input is simply the JWS Payload (not
>>>> encoded).  If the JWS Protected Header has a non-empty value, then the the
>>>> JWS Signing Input is the concatenation of the Encoded JWS Header, a period
>>>> ('.') character, and the Encoded JWS Payload"
>>>>
>>>> There are two reasons for these, both of which are important to make
>>>> sure that this spec is applicable to a broad variety of use cases:
>>>> 1. Compactness.  As Jim notes, quoted strings are shorter than
>>>> base64url-encoded strings
>>>>
>>>
>>> The tokens are already compact enough.
>>>
>>>
>>>>  2. Crypto-compatibility: Avoiding base64-encoding means that there's
>>>> nothing JWS-specific about the signature value.  So for example, you could
>>>> translate a JWS with no protected attributes to a CMS SignedData object
>>>> with no protected attributes.
>>>>
>>>
>>> Not a useful feature from my point of view.
>>>
>>> -- Dick
>>>
>>
>>
>
> _______________________________________________
> jose mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jose
>
>

---

From: Dick Hardt <[email-masked]>
Date: Tue, 2 Jul 2013 17:14:46 -0700

Hi Richard, thanks for the example, some comments and questions:

To clarify, in the JSON (non-compact) version, the payload is restricted to being a string? ie. it cannot be a JSON object? If so, that seems really limiting.

I don't understand why "cenc" needs to be added to the compact serialization. The recipient will need to know if it is in the compact format or in JSON before parsing.

My gut tells me that signing the header is useful and will thwart some attacks -- but I am not an expert in that area. Is there some research that says it makes no difference? I think it would help catch implementation errors.

-- Dick

On Jul 2, 2013, at 3:20 PM, Richard Barnes <[email-masked]> wrote:

> 1. Define a "cenc" header parameter, for "Content ENCoding", with one value, "b64u".  When present in the JSON serialization, the "payload" field is base64url encoded; when it is absent, the "payload" field represents the payload directly (as a UTF-8 string).
> 3. When protected a header are present (always for the compact serialization), the JWS Signing Input is base64(protected).base64(payload), as it is in -11.  When no protected header is present, the JWS Signing Input is simply the octets of the payload.
> 
> With that in mind, let's look at a few test cases.  We'll use the following inputs:
> 
> With those inputs, let's consider 4 cases that show the possibilities discussed above.
> 
> No change here.  The compact case just represents the special case where the payload is base64-encoded and all the headers are protected.  Thus, we use the base64.base64 form for the JSON Signing Input.
> JWS Signing Input: eyJhbGciOiJIUzI1NiIsImtpZCI6IjIifQ.Y2Fmw6k (as a UTF-8 string)
> == 65794a68624763694f694a49557a49314e694973496d74705a434936496a496966512e5932466d77366b (hex)
> 
> Here we've removed the integrity protection from the headers (because who needs it? ;) ).  So the signature is different, because JWS Signing Input here is just the payload.
> This object is equivalent to Case 3, but with the payload expressed directly as a JSON string instead of base64-encoded.  And, if you ask me, the prettiest and most human-readable of the whole menagerie here :)
> 
> I've written a script to generate these test cases, in case you want to play with this some more:
> Our you looking at a way of effectively signing a JSON payload? A complete, real world example would help alot.
> 
> 
> I guess I should have been clearer that I was only talking about the JSON serialization.  My proposed change would have no impact on the compact serialization.  Hopefully that answers your questions about separators, etc.; they payload would just be a normal JSON string.
> 
> --Richard
> 
> 
> I think there are two separable issues here, both of which Mike and I probably have divergent opinions on :)
> 1. Syntax: Should it be possible for the "payload" field JWS to be a JSON string (instead of base64) when the payload is simply a UTF-8 string?
> 
> One of the goals of the token was that it not require encoding for it to be transported in a request. IE that it be safe and SIMPLE to include in an HTTP header or in a URL
>  
> 1. The payload should not be base64url encoded unless it cannot be represented as a JSON string.
> 
> How does an implementation determine which parts are separated? Are '.' escaped? Does the implementation need to parse the JSON to determine a valid separator? How does it know the end of the JSON? That sounds really hard to implement, or I am misunderstanding what you are proposing.
> 
>  
> 2. The JWS Signing Input should not be base64url encoded unless there is a JWS Protected Header
> 
> 1. Add a "b64" header parameter that indicates that the payload is base64url-encoded binary, as opposed to a UTF-8 string.  Specify that this parameter is on by default in compact-formatted JWS objects.
> 
> 2. Modify the signing/verification instructions to switch based on the presence of a JWS Protected Header: "If the JWS Protected Header is absent or empty, then the JWS Signing Input is simply the JWS Payload (not encoded).  If the JWS Protected Header has a non-empty value, then the the JWS Signing Input is the concatenation of the Encoded JWS Header, a period ('.') character, and the Encoded JWS Payload"
> 
> There are two reasons for these, both of which are important to make sure that this spec is applicable to a broad variety of use cases:
> 1. Compactness.  As Jim notes, quoted strings are shorter than base64url-encoded strings
> 
> 2. Crypto-compatibility: Avoiding base64-encoding means that there's nothing JWS-specific about the signature value.  So for example, you could translate a JWS with no protected attributes to a CMS SignedData object with no protected attributes.
>

---

From: Richard Barnes <[email-masked]>
Date: Wed, 3 Jul 2013 12:04:43 -0400

On Tue, Jul 2, 2013 at 8:14 PM, Dick Hardt <[email-masked]> wrote:

> To clarify, in the JSON (non-compact) version, the payload is restricted
> to being a string? ie. it cannot be a JSON object? If so, that seems really
> limiting.
>

The payload can be anything that can be expressed as a JSON string.  The
payload needs to be serialized, not as a JSON entity (object, array,
number), because it's going to be input to a signature verification
operation (so it needs to be canonical).  Every JSON string has a unique
representation in UTF-8, so we can use that for anything that can be put
into a string.

> I don't understand why "cenc" needs to be added to the compact
> serialization. The recipient will need to know if it is in the compact
> format or in JSON before parsing.
>

You're correct: "cenc" isn't expressed in the compact serialization, it's
part of the definition that the payload is always base64-encoded.  I just
included it in Case 2 to show the equivalent JSON form (since it's *not*
fixed in JSON).

> My gut tells me that signing the header is useful and will thwart some
> attacks -- but I am not an expert in that area. Is there some research that
> says it makes no difference? I think it would help catch implementation
> errors.
>

We had a thread on CFRG on this a while back [1].  The upshot is that you
can't prevent an attacker from making someone execute the wrong algorithm,
because he has to execute the algorithm in order to realize that the header
is bad.  The one real attack we have is that RSA-PSS does not protect the
hash value, so if an attacker can, say, find a SHA-1 collision that
produces the same signature as the "RS256" output, then he can change the
algorithm to exploit that attack.  So we need to have the ability to
protect the "alg" header for that case (cf. as RFC 6211 does for CMS [2]);
everything else is warm fuzzies.

--Richard

[1] <http://www.ietf.org/mail-archive/web/cfrg/current/msg03381.html>
[2] <https://tools.ietf.org/html/rfc6211>

>>> I guess I should have been clearer that I was only talking about the
>>> JSON serialization.  My proposed change would have no impact on the compact
>>> On Thu, Jun 27, 2013 at 7:56 PM, Dick Hardt <[email-masked]>wrote:
>>>
>>>>
>>>>
>>>>
>>>> One of the goals of the token was that it not require encoding for it
>>>> to be transported in a request. IE that it be safe and SIMPLE to include in
>>>> an HTTP header or in a URL
>>>>
>>>>

---

From: Dick Hardt <[email-masked]>
Date: Wed, 3 Jul 2013 10:55:21 -0700

> representation in UTF-8, so we can use that for anything that can be put

So it is just a string that has had control characters escaped
(",\,/,backspace, formfeed, newline, carriage return, horizontal tab)

This is in sharp contrast to the compact form that takes a JSON object.

I have a hard time understanding how this even fits into the WG Charter as
your proposal is not transporting JSON. (yes, a JSON string is part of
JSON, but it does not have the value proposition that a JSON object has)

Am I missing something?

-- Dick

---

From: Richard Barnes <[email-masked]>
Date: Wed, 3 Jul 2013 17:15:53 -0400

> JSON, but it does not have the value proposition that a JSON object has)

Yeah, I think you might be a little confused.  Right now, the payload can
be any octets -- JSON, UTF-8, EBCDIC, JPEG, QuickTime, whatever. That's
true for both the JSON and compact serializations. And that's why in
general, the payload has to be base64-encoded to be able to carry it in
JSON.

This proposal is just saying that when you have content that can be
represented as UTF-8 (e.g., JSON or HTML), you don't have to base64 encode
it, you can just stick it in a JSON string.

--Richard

---

From: Dick Hardt <[email-masked]>
Date: Wed, 3 Jul 2013 14:23:33 -0700

I'm guessing that per: http://datatracker.ietf.org/wg/jose/charter/

- A Standards Track document specifying a representation of encrypted
data using JSON data structures, where the data to be protected
includes (but is not limited to) JSON data structures.

You are looking to use JSON as the wrapper for a JSON string.

Interesting, but I think that should be a different document as it solves
very different use cases and the processing is very different.

On Wed, Jul 3, 2013 at 2:15 PM, Richard Barnes <[email-masked]> wrote:

>>>> To clarify, in the JSON (non-compact) version, the payload is
>>>> restricted to being a string? ie. it cannot be a JSON object? If so, that
>>>> seems really limiting.
>>>>
>>>
>> I have a hard time understanding how this even fits into the WG Charter
>> as your proposal is not transporting JSON. (yes, a JSON string is part of

---

From: John Bradley <[email-masked]>
Date: Wed, 3 Jul 2013 17:24:59 -0400

For a JWT the content is a JSON object.  For JOSE that restriction doesn't apply, and the content could be any binary data.  That is one of the reasons people want to put additional JSON in the envelope as they may not be able to that easily as part of the body.

Just for the record I am one of the people on the side of integrity protecting headers unless there is a strong reason not to as is the case with multiple recipients and counter mode encryption.

John B.

On 2013-07-03, at 5:15 PM, Richard Barnes <[email-masked]> wrote:

> The payload can be anything that can be expressed as a JSON string.  The payload needs to be serialized, not as a JSON entity (object, array, number), because it's going to be input to a signature verification operation (so it needs to be canonical).  Every JSON string has a unique representation in UTF-8, so we can use that for anything that can be put into a string.
> 
> So it is just a string that has had control characters escaped (",\,/,backspace, formfeed, newline, carriage return, horizontal tab)
> 
> I have a hard time understanding how this even fits into the WG Charter as your proposal is not transporting JSON. (yes, a JSON string is part of JSON, but it does not have the value proposition that a JSON object has)
> 
> Am I missing something?
> 
> -- Dick
> 
> Yeah, I think you might be a little confused.  Right now, the payload can be any octets -- JSON, UTF-8, EBCDIC, JPEG, QuickTime, whatever. That's true for both the JSON and compact serializations. And that's why in general, the payload has to be base64-encoded to be able to carry it in JSON.  
> 
> This proposal is just saying that when you have content that can be represented as UTF-8 (e.g., JSON or HTML), you don't have to base64 encode it, you can just stick it in a JSON string.
> 
> --Richard
>

---

From: Mike Jones <[email-masked]>
Date: Wed, 3 Jul 2013 21:33:09 +0000

As I'd written earlier, *if* the working group decides to also support a payload representation that is not base64url encoded (and I think that's a big *if*), I believe that it should apply only to the JSON Serialization and use an alternative member name, so that the current JSON Serialization objects are unchanged.
For instance, these three alternative representations could all represent the same JWS Payload:
    "payload": "SGVsbG8gd29ybGQ"
    "verbatim": "Hello world"
    "verbatim": "Hello\u0020world"

As I see it, we shouldn't change the meaning of "payload" at this point, without as Karen put it, "a strong rationale and a ground swell of working group support".

                                                                -- Mike

From: John Bradley [mailto:[email-masked]]
Sent: Wednesday, July 03, 2013 2:25 PM
To: Richard Barnes
Cc: Dick Hardt; Jim Schaad; n-sakimura; Mike Jones; [email-masked]; [email-masked]
Subject: Re: [jose] #26: Allow for signature payload to not be base64 encoded

John B.

On 2013-07-03, at 5:15 PM, Richard Barnes <[email-masked]<mailto:[email-masked]>> wrote:

On Wed, Jul 3, 2013 at 1:55 PM, Dick Hardt <[email-masked]<mailto:[email-masked]>> wrote:
On Wed, Jul 3, 2013 at 9:04 AM, Richard Barnes <[email-masked]<mailto:[email-masked]>> wrote:
On Tue, Jul 2, 2013 at 8:14 PM, Dick Hardt <[email-masked]<mailto:[email-masked]>> wrote:
Hi Richard, thanks for the example, some comments and questions:

Am I missing something?

-- Dick

--Richard

_______________________________________________
jose mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/jose

---

From: Mike Jones <[email-masked]>
Date: Wed, 3 Jul 2013 21:40:19 +0000

John, since you're raising the topic of integrity protecting JWS header values, I'd be interested in your reactions to my note encoded below.

                                                                Cheers,
                                                                -- Mike

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Mike Jones
Sent: Saturday, June 29, 2013 3:43 AM
To: [email-masked]
Subject: Re: [jose] #24: Move JWS headers into signature block

Perhaps I'm in an odd frame of mind tonight, because I wouldn't normally even consider re-raising a closed issue, but Ben Laurie's advice "why not just protect everything" kept running my mind and I realized that the current JWS JSON Serialization doesn't allow us to do that in the general case.  Specifically, we don't allow a per-signature "protected" headers field, which would be necessary to protect the cryptographic parameters if different signatures use different algorithms.

So I'd at least like others' thoughts on whether we want to "fill in the matrix" for the JWS JSON Serialization and allow header parameters to be specified both in protected and unprotected forms, both on a shared and per-signature basis.  We currently support 3 of these 4 header parameter locations.

Note that we would not do this for JWE, since (as extensively discussed) per-recipient protected content is problematic.

For the signature input, if both shared and per-signature protected headers were present, we'd need to concatenate the two base64url encoded representations together with a separator character between (I'm thinking comma (',') because it is distinct from period ('.'), which is also used as a separator in the signature input).

I'm fine with this issue remaining closed, but I wanted to at least run this possibility by the working group for their input, since it hadn't been discussed previously.

                                                                Cheers,

                                                                -- Mike

...

John B.

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 4 Jul 2013 16:32:32 +1000

Separate names to hold the payload as a base64url-encoding ("payload") or as a string ("verbatim") is a better approach than a "cenc" header field. At some point we might add a third name ("payload_uri") to identify the content without including it in the JSON.

One hassle we are having is that transmitting the payload in B64, or as a string, or externally, or by reference isn’t merely a serialization choice, but actually affects the bytes that are signed (so it potentially changes the security properties so it requires lots of care). Understandably, but unfortunately, many are reluctant to make changes to tackle this at this point.

A related issue is that the "alg" value is no longer sufficient to indicate the bytes that are signed. This is one reason why a “mode” concept would be helpful (along with a "mode" field, with defaults for compatibility with current drafts).

--
James Manger

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mike Jones
Sent: Thursday, 4 July 2013 7:33 AM
To: John Bradley; Richard Barnes
Cc: Jim Schaad; n-sakimura; [email-masked]; Dick Hardt
Subject: Re: [jose] #26: Allow for signature payload to not be base64 encoded

As I’d written earlier, *if* the working group decides to also support a payload representation that is not base64url encoded (and I think that’s a big *if*), I believe that it should apply only to the JSON Serialization and use an alternative member name, so that the current JSON Serialization objects are unchanged.

As I see it, we shouldn’t change the meaning of “payload” at this point, without as Karen put it, “a strong rationale and a ground swell of working group support”.

                                                                -- Mike