Subject: Re: [http-auth] HTTP Signatures specification updated

From: Julian Reschke <[email-masked]>
Date: Mon, 03 Feb 2014 17:40:49 +0100

Manu Sporny:
> In order to make Web Payments (and REST APIs in general) more secure on
> the Web, this group has been working on extensions to the HTTP protocol
> so that you can digitally sign HTTP headers and content.
>
> We just pushed an update to the specification and believe that it's
> ready for public comment. The specification is already deployed in
> production at Joyent and Digital Bazaar:
>
> http://tools.ietf.org/html/draft-cavage-http-signatures-01
>
> This technology is useful when you just need to make a secure request to
> your financial provider to transmit money from you to someone else, make
> a payment, create an invoice, etc. without having to perform a login
> handshake (OAuth, user+pass, etc.).
> ...

Hi Manu,

find below my feedback:

Technical:

1) The spec seems to be confused about whether it defines an HTTP 
authentication scheme or not. If it does, it needs to follow the 
requirements in 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p7-auth-25.html#authentication.scheme.registry; 
it it does not, it shouldn't use the "Authorization" header field.

2)
    REQUIRED.  The `algorithm` parameter is used if the client and server
    agree on a non-standard digital signature algorithm.  The full list
    of supported signature mechanisms is listed below.

-  This should be aligned with other IETF specs using algorithm names.

3)
    OPTIONAL.  The `headers` parameter is used to specify the list of
    HTTP headers used to sign the request.  If specified, it should be a
    quoted list of HTTP header names, separated by a single space
    character.  By default, only one HTTP header is signed, which is the
    `Date` header.  Note that the list MUST be specified in the order the
    values are concatenated together during signing.  To include the HTTP
    request line in the signature calculation, use the special `request-
    line` value.  While this is overloading the definition of `headers`
    in HTTP linguism, the request-line is defined in RFC 2616 [RFC2616],
    and as the outlier from headers in useful signature calculation, it
    is deemed simpler to use `request-line` than to add a separate
    parameter for it.

- just state that the list order is significant for the signature algorithm

- for request-line, just pick a name that is guaranteed not to conflict 
(such as "(request-line)"). That being said, the request line is gone 
from HTTP/2, and it might be good to break the information down into 
components that are more generic.

4)
    OPTIONAL.  The `extensions` parameter is used to include additional
    information which is covered by the request.  The content and format
    of the string is out of scope for this document, and expected to be
    specified by implementors.

- the BNF says 'ext'

- this will make it incredible hard to have more than one extension. why 
not allow extension specs to use (and register) new parameter names?

5)
    REQUIRED.  The `signature` parameter is a `Base64` encoded digital
    signature generated by the client.  The client uses the `algorithm`
    and `headers` request parameters to form a canonicalized `signing
    string`.  This `signing string` is then signed with the key
    associated with `keyId` and the algorithm corresponding to
    `algorithm`.  The `signature` parameter is then set to the `Base64`
    encoding of the signature.

- Which base64? Please pick a concrete on, such as the one defined in 
<http://tools.ietf.org/html/rfc4648#section-4>

6)
    1.  If the header name is not `request-line` then append the
        lowercased header name followed with an ASCII colon `:` and an
        ASCII space ` `.

- And the field value?

- Also: what if there are multiple instances of a given field name?

- What about leading and trailing whitespace in the header field value?

7)
    Digest: SHA-256=X48E9qOokqqrvdts8nOJRJN3OWDUoyWxBf7kbu9DBPE=

- If this is the Digest header field defined in 
<http://tools.ietf.org/html/rfc3230#section-4.3.2> then the spec should 
say so.

8) Error handling: as this is relevant for security, the spec needs to 
state how to handle ambiguous values, such as multiple occurrences of 
the same parameter name in "Authorization".

Editorial:

- In the boilerplate, please say where to send feedback.

- Please cite the HTTPbis drafts instead of RFC 2616.

- Please do not cite RFC 2617 except when talking about Basic and Digest

- s/header/header field/

- please use symbolic references (xml2rfc PI symrefs="yes")

- in xml2rfc, you get appendices by putting <section> elements into the 
<back> element

- if BNF notation is needed, please use ABNF as defined in RFC 5234.

- (IANAL) the statement about Public Domain may or may not be in 
conflict with the Copyright Statement on the front page

- OAuth reference missing

- the reference to RFC 3339 seems to be unused

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 4 Feb 2014 13:54:22 +1100

Since we're giving feedback on this:

* There are many ways that signatures could be useful in HTTP, and this is just one way. So, "HTTP Signatures" is a terrible name for this; it's way to generic. Please change it. Suggestion: "The 'Signature' HTTP Authentication Scheme' (but see Julian's comment below).

* Abstract - The term 'origin' is defined by RFC6454, and I don't think that's how you're using it here; at best, you're authenticating the client *to* the origin. 

* 1. Introduction - this is a protocol extension, not a protocol in and of itself.

* 1. Introduction - "However, it is common place..." -- this sentence is probably going to get you in more trouble than it's worth :)  Recommend dropping it. Also, you don't cover why Digest auth isn't good.

* 2.1.1.2 algorithm - "The full list of supported signature mechanisms is listed below."  This sort of thing is usually delegated to a registry, so that you don't have to revise the whole spec to modify the list.  Also, "supported" is odd here, because this is a protocol specification, not a product specification.

* 2.1.1.3 headers - Signing the *whole* request-line is a very bad idea, because the HTTP version is a hop-by-hop artefact, not end-to-end. Also, the spacing of the request-line can be irregular. I'd suggest you define how to canonicalise the request method + URI (ideally, the effective request URI, *not* the request-target; see <http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-25#section-5.5>).

* 2.1.2 Signature String Construction - You need to define how to canonicalise headers before processing, taking into account that headers can recur, have varying whitespace between the field-name, colon and value, whitespace at the EOL and varying line termination.  

* You don't define a corresponding challenge. Your use cases might not require a 401 + WWW-Authenticate, but have you considered that some will want this?

A few more comments below.

On 4 Feb 2014, at 3:40 am, Julian Reschke <[email-masked]> wrote:

> 1) The spec seems to be confused about whether it defines an HTTP authentication scheme or not. If it does, it needs to follow the requirements in <http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p7-auth-25.html#authentication.scheme.registry; it it does not, it shouldn't use the "Authorization" header field.

See especially <http://tools.ietf.org/html/draft-ietf-httpbis-p7-auth-25#section-5.1.2>. 

> - for request-line, just pick a name that is guaranteed not to conflict (such as "(request-line)"). That being said, the request line is gone from HTTP/2, and it might be good to break the information down into components that are more generic.
> 
> 4)
> - this will make it incredible hard to have more than one extension. why not allow extension specs to use (and register) new parameter names?
> 
> 
> 5)
> - Which base64? Please pick a concrete on, such as the one defined in <http://tools.ietf.org/html/rfc4648#section-4>
> 
> 6)
> - If this is the Digest header field defined in <http://tools.ietf.org/html/rfc3230#section-4.3.2> then the spec should say so.

This confused me too.

> 
> 8) Error handling: as this is relevant for security, the spec needs to state how to handle ambiguous values, such as multiple occurrences of the same parameter name in "Authorization".
> 
> 
> Editorial:
> 
> - in xml2rfc, you get appendices by putting <section> elements into the <back> element
> 
> - (IANAL) the statement about Public Domain may or may not be in conflict with the Copyright Statement on the front page
> 
> _______________________________________________
> http-auth mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/http-auth

--
Mark Nottingham   http://www.mnot.net/

---

From: Manu Sporny <[email-masked]>
Date: Wed, 05 Feb 2014 16:13:20 -0500

> find below my feedback:

Hi Julian, thanks for the feedback, responses to you feedback with some
additional questions below.

> if it does not, it shouldn't use the "Authorization" header field.

The specification started out as a way for clients to authenticate
themselves to the origin. We've had a few requests to use the same
general approach for the origin to authenticate certain parts of the
HTTP message back to the client (which can be useful over non-TLS
connections, or if you want to have extra security on top of TLS for
Pervasive Monitoring scenarios).

If we wanted to use the mechanism for both client-to-origin signatures
and origin-to-client signatures, what would you suggest that we do? We
could introduce a new header "Signature", that can be used by both
clients and origins, but didn't know if that would be the proper
approach. Thoughts?

For the time being, I think I've followed the requirements in the
draft-ietf-httpbis-p7-auth-25 spec. It's unclear exactly how I should
specify the registration. The registry doesn't exist yet. So I've added
an IANA Considerations appendix noting the requirement for a
registration to be made if the spec makes it through the IETF process:

https://web-payments.org/specs/source/http-signatures/#appendix-d

It would be helpful if the text in ietf-httpbis-p7-auth-25 gave an
example of how one does this, or linked to an RFC that describes how to
do this.

> 2) REQUIRED.  The `algorithm` parameter is used if the client and
> server agree on a non-standard digital signature algorithm.  The full
> list of supported signature mechanisms is listed below.
> 

The algorithm names are aligned w/ RFC6376:

http://tools.ietf.org/html/rfc6376

JOSE chose to use different algorithm names. Which IETF specs should we
align with, they're both current, no?

> 3) OPTIONAL.  The `headers` parameter is used to specify the list of 
> `Date` header.  Note that the list MUST be specified in the order
> the values are concatenated together during signing.  To include the
> HTTP request line in the signature calculation, use the special
> `request- line` value.  While this is overloading the definition of
> `headers` in HTTP linguism, the request-line is defined in RFC 2616
> [RFC2616], and as the outlier from headers in useful signature
> calculation, it is deemed simpler to use `request-line` than to add a
> separate parameter for it.
> 

Done. I moved the explanatory text further down in the document.

https://web-payments.org/specs/source/http-signatures/#headers

> - for request-line, just pick a name that is guaranteed not to
> conflict (such as "(request-line)"). That being said, the request
> line is gone from HTTP/2, and it might be good to break the
> information down into components that are more generic.

Would still calling it "(request-line)" be fine and the state that for
HTTP2, the request line is built using the method, path, and the string
"HTTP/2"? For example:

GET /foo HTTP/2

would be the request line generated for creating and validating signatures?

> 4) OPTIONAL.  The `extensions` parameter is used to include
> additional information which is covered by the request.  The content
> and format of the string is out of scope for this document, and
> expected to be specified by implementors.
> 
> - the BNF says 'ext'

Removed the parameter based on your feedback below:

> - this will make it incredible hard to have more than one extension.
> why not allow extension specs to use (and register) new parameter
> names?

That's a better idea, fixed:

https://web-payments.org/specs/source/http-signatures/#appendix-b

> 5) REQUIRED.  The `signature` parameter is a `Base64` encoded
> digital signature generated by the client.  The client uses the
> `algorithm` and `headers` request parameters to form a canonicalized
> `signing string`.  This `signing string` is then signed with the key 
> - Which base64? Please pick a concrete on, such as the one defined
> in <http://tools.ietf.org/html/rfc4648#section-4>

I picked the one you mentioned:

https://web-payments.org/specs/source/http-signatures/#signature

> 6) 1.  If the header name is not `request-line` then append the 
> lowercased header name followed with an ASCII colon `:` and an ASCII
> space ` `.
> 
> - And the field value?

Fixed:

https://web-payments.org/specs/source/http-signatures/#canonicalization

> - Also: what if there are multiple instances of a given field name?

Added guidance if there are multiple instances:

http://sites.local/web-payments.org/specs/source/http-signatures/#canonicalization

> - What about leading and trailing whitespace in the header field

They are preserved, we want to stay as far away from canonicalizing the
headers as we can. If a library, client, proxy, or server adds/removes
whitespace, then the application is screwed. Canonicalization of header
field values is something we do not want to specify, even if it means
that signatures in some environments will fail.

That said, if this becomes a major issue, then an extension
specification can deal with it (or we can rev the spec to include some
basic canonicalization). Unfortunately, at that point we're firmly in
DKIM-Signature territory. One of the goals of this specification is to
not be more complex than necessary.

I added a note to mention that header field values must not be modified:

https://web-payments.org/specs/source/http-signatures/#canonicalization

> 7) Digest: SHA-256=X48E9qOokqqrvdts8nOJRJN3OWDUoyWxBf7kbu9DBPE=
> 
> <http://tools.ietf.org/html/rfc3230#section-4.3.2> then the spec
> should say so.

Fixed, the spec now refers to that RFC (although, informally as
Digest isn't required for the spec).

https://web-payments.org/specs/source/http-signatures/#canonicalization

> 8) Error handling: as this is relevant for security, the spec needs
> to state how to handle ambiguous values, such as multiple occurrences
> of the same parameter name in "Authorization".

The specification now states what to do w/ multiple occurrences of the
same Signature parameter in the Authorization header:

http://sites.local/web-payments.org/specs/source/http-signatures/#ambiguity

It also describes what to do with multiple header fields:

https://web-payments.org/specs/source/http-signatures/#canonicalization

Is there a list somewhere of other sorts of error handling that should
be taken into account with HTTP?

> - In the boilerplate, please say where to send feedback.

Done:

https://github.com/web-payments/web-payments.org/commit/0da85e2066011ab7e9eb8595ecd136affb6a5346

Although, I don't know if I added the note in the right way. The W3C
spec tool allows you to modify the Status section, I couldn't find out
how to do that w/ the xml2rfc tool.

> - Please cite the HTTPbis drafts instead of RFC 2616.

Fixed:

https://github.com/web-payments/web-payments.org/commit/83c9cba2a3fc99bef0e71bd0fadeb058b7576966

> - Please do not cite RFC 2617 except when talking about Basic and

Done:

https://github.com/web-payments/web-payments.org/commit/6f97a12df79619b8eb264db8ce209ef149c30598

> - s/header/header field/

Fixed:

https://github.com/web-payments/web-payments.org/commit/e8c14fa0b2fdc291473f1aa7302281e4915ee66e

> - please use symbolic references (xml2rfc PI symrefs="yes")

I thought I was, but the xml2rfc tool doesn't seem to be picking it up?

https://github.com/web-payments/web-payments.org/blob/6f97a12df79619b8eb264db8ce209ef149c30598/specs/source/http-signatures/index.xml#L17

> - in xml2rfc, you get appendices by putting <section> elements into

Fixed:

https://github.com/web-payments/web-payments.org/commit/e3ea00aa4e75b6791647baa2a0b5eecaeb87a72b

> - if BNF notation is needed, please use ABNF as defined in RFC 5234.

Fixed, I think:

https://github.com/web-payments/web-payments.org/commit/937875db36e7799fafdfd6b5c4e35fb9e0a7df79

> conflict with the Copyright Statement on the front page

Fixed:

https://github.com/web-payments/web-payments.org/commit/0d21fa41cf9fec2aa06007919aca8bceefa13d28

> - OAuth reference missing

https://github.com/web-payments/web-payments.org/commit/774c7842f5c76e32e565f26303763e1c155242eb

> - the reference to RFC 3339 seems to be unused

Removed the reference.

Thanks a bunch for your input Julian, it really helped clean up the spec!

It would be great to hear your thoughts on whether this spec should
continue to use the Authorization header or if it should try to do
origin-to-client authentication as well. We'd like to do the latter if
there is a simple, clean way to do it. It would be nice if the HTTP
message information passed back from the financial servers that would be
using this spec could be digitally signed as well. Answers to the other
questions asked would also be greatly appreciated. :)

-- manu

-- 
Manu Sporny (skype: msporny, twitter: manusporny, G+: +Manu Sporny)
Founder/CEO - Digital Bazaar, Inc.
blog: The Worlds First Web Payments Workshop
http://www.w3.org/2013/10/payments/

-- 
Manu Sporny (skype: msporny, twitter: manusporny, G+: +Manu Sporny)
Founder/CEO - Digital Bazaar, Inc.
blog: The Worlds First Web Payments Workshop
http://www.w3.org/2013/10/payments/

---

From: Manu Sporny <[email-masked]>
Date: Fri, 07 Feb 2014 12:34:46 -0500

Hey Mark,

Thanks a bunch for your feedback, responses to your comments below:

On 02/03/2014 09:54 PM, Mark Nottingham wrote:
> * There are many ways that signatures could be useful in HTTP, and 
> this is just one way. So, "HTTP Signatures" is a terrible name for 
> this; it's way to generic. Please change it. Suggestion: "The 
> 'Signature' HTTP Authentication Scheme' (but see Julian's comment 
> below).

Changed to what you suggested:

https://github.com/web-payments/web-payments.org/commit/3e22397d0d0222983d9d3a64342654e30eac5854

As I mentioned to Julian, we may change the spec to provide both client
to server and server to client authentication (or verifiability of
header fields where a receiving application can assume authentication
based on the digital signature). We'd love to hear your thoughts on the
best way to do that. For example, should we introduce a new HTTP header
field called "Signature" or "Verification"? Or should we extend the
"Authentication" header field in the response from the server to the client?

So, the name of the spec may change to:

"HTTP Security Rock Goddess version 3.awesome"

... which I'm sure we can all agree is a way better name that anything
that has been proposed to date. :P

> * Abstract - The term 'origin' is defined by RFC6454, and I don't 
> think that's how you're using it here; at best, you're
> authenticating the client *to* the origin.

Changed it to "server" since that's what I think the original author meant:

https://github.com/web-payments/web-payments.org/commit/b6ba31f143105a084094be39644e70b8a76cd339

Fixed:

https://github.com/web-payments/web-payments.org/commit/b48b4a76f84cbe268992afcd877b2f9da60d4e03#diff-d50cc99f156dfafd46bcd16c22ee2187L85

> * 1. Introduction - "However, it is common place..." -- this
> sentence is probably going to get you in more trouble than it's worth
> :) Recommend dropping it. Also, you don't cover why Digest auth
> isn't good.

https://github.com/web-payments/web-payments.org/commit/95d724119b6cc7e797f42841338dbcbbedacd53b

> * 2.1.1.2 algorithm - "The full list of supported signature 
> mechanisms is listed below."  This sort of thing is usually
> delegated to a registry, so that you don't have to revise the whole
> spec to modify the list.  Also, "supported" is odd here, because this
> is a protocol specification, not a product specification.

Done:

https://github.com/web-payments/web-payments.org/commit/a42cdf5119f99cc9025ac8a56ecde70f36dc1b83

and fixed the text here:

https://github.com/web-payments/web-payments.org/commit/9b1f24dfbc7b9da2c2d0d5b48ed636c5ad79a168

> * 2.1.1.3 headers - Signing the *whole* request-line is a very bad 
> idea, because the HTTP version is a hop-by-hop artefact, not 
> end-to-end. Also, the spacing of the request-line can be irregular. 
> I'd suggest you define how to canonicalise the request method + URI 
> (ideally, the effective request URI, *not* the request-target; see 
> <http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-25#section-5.5>).

Fixed the algorithm to generate the request line:

https://github.com/web-payments/web-payments.org/commit/c64c7cf327b7e034ecf4793787f34f888a351385

I'm hesitant to require the use of the effective request URI since the
Host can be digitally signed as well. I realize that there is still an
issue if an HTTP-based request is replayed over HTTPS, but if that's a
concern then the application should be using nonces.

If there is a stronger attack vector than that, we could use the
effective request URI. I'm just concerned that some applications may not
know whether they're going to submit the request over HTTP or HTTPS. I
mean, most of them should know that, but some of them might not? Looking
for guidance here because this is not an area that I know much about
(other than "this sounds like it might be a pain for developers and not
really increase the security of protocol extension enough to warrant the
increase in complexity").

> * 2.1.2 Signature String Construction - You need to define how to 
> canonicalise headers before processing, taking into account that 
> headers can recur, have varying whitespace between the field-name, 
> colon and value, whitespace at the EOL and varying line termination.

Addressed the headers recurring issue here:

https://github.com/web-payments/web-payments.org/commit/e8c14fa0b2fdc291473f1aa7302281e4915ee66e#diff-d50cc99f156dfafd46bcd16c22ee2187R208

We don't specify how to deal with whitespace in the field name because
we don't want to get into canonicalization. We could do a lighter form
of canonicalization (strip all whitespace at beginning and end of
string), but we'd rather the signature just not work for the
applications than have funky whitespace than try to explain how
canonicalization happens. The approach is: We'll sign whatever you give
us for the header field name. How has this approach worked out in the
past? I have a good idea of what your response is going to be. :)

I think we're fairly explicit about line termination being LF (aka
"\n"), what am I missing? We don't explicitly state LF or use hex values
in the spec, do you want us to do that?

> * You don't define a corresponding challenge. Your use cases might 
> not require a 401 + WWW-Authenticate, but have you considered that 
> some will want this?

Yes, we did consider it. We wanted this to be a mostly "you're verified
or you're not" mechanism. We didn't really want any sort of
back-and-forth negotiation. That said, it's a weak argument because you
probably want to be able to notify clients that they could access the
resource if they provided a signature.

If we decide that this is going to use the "Authorization" header (and
not some new kind of header), we'll define the WWW-Authenticate bits of
it. I've raised an issue to track it:

https://github.com/web-payments/web-payments.org/issues/23

>> it does not, it shouldn't use the "Authorization" header field.
> 
> See especially 
> <http://tools.ietf.org/html/draft-ietf-httpbis-p7-auth-25#section-5.1.2>.

Ok, I've gone down the list and attempted to address each item (or make
a note that we need to deal w/ the item in the spec):

> HTTP authentication is presumed to be stateless

Check, it's stateless.

> The authentication parameter "realm" is reserved

We don't use it.

> The "token68" notation was introduced for compatibility with
> existing authentication schemes and can only be used once per
> challenge or credential.

We use the auth-param approach and stay far away from token68.

> When the auth-param syntax is used, all parameters ought to support 
> both token and quoted-string syntax, and syntactical constraints 
> ought to be defined on the field value after parsing (i.e., 
> quoted-string processing).

We were not doing this, it's been clarified in the spec now by removing
the ABNF, which was largely unnecessary:

https://github.com/web-payments/web-payments.org/commit/3b37035ae0c8c94c8705c4cb103fd801daa4b96f

> Definitions of new schemes ought to define the treatment of unknown
> extension parameters.

Fixed:

https://github.com/web-payments/web-payments.org/commit/dfed1f92341722749c2dd653b1658dd092232967

> Authentication schemes need to document whether they are usable in 
> origin-server authentication (i.e., using WWW-Authenticate), and/or
> proxy authentication (i.e., using Proxy-Authenticate).

Waiting on feedback from Julian and you to address this issue. It's
being tracked here:

https://github.com/web-payments/web-payments.org/issues/23

> Therefore, new authentication schemes that choose not to carry 
> credentials in the Authorization header field (e.g., using a newly 
> defined header field) will need to explicitly disallow caching

I'm sure someone more versed in HTTP knows what this means, but I'm
having a hard time figuring out if this applies to us. My reading on it is:

If you chose to use some information that's outside of the Authorization
header that is used by the authorization scheme in some way, you need to
be explicit about whether or not the request should be cached.

I don't think this applies to this spec because we are okay with a
response being cached if the digital signature on the request is the
same (ie: a botnet of clients requesting the exact same document with
the same set of headers signed).

That said, I'm probably not the best person to make this determination,
do you have any thoughts on the matter?

Fixed:

https://github.com/web-payments/web-payments.org/commit/3e3d215d30ae7069703061b5be7db3917888fa00

Thanks a ton for the feedback Mark! It's very much appreciated. If you
could let us know your thoughts on how we should extend the spec to
cover both client to server and server to client verified HTTP headers,
that'd be swell. It doesn't seem like it would be much work, but we're
struggling wrt. a proper approach that would fit in with the larger body
of work on HTTP. :)

-- manu