Subject: MACing HTTP requests/responses (Re: Content-Integrity header)

From: Nico Williams <[email-masked]>
Date: Wed, 11 Jul 2012 13:43:27 -0500

I agree that we need something better, and in particular that we ought
to have a MAC instead of a plain hash.  The problem with a MAC is:
whence the key?  Also, what should the MAC be applied to?

Using a MAC, having a shared session key, ties into HTTP
authentication.  We can definitely have a generic MAC for HTTP and say
that HTTP authentication mechanisms that can should output session
keys.  And the HTTP authentication would also have to take care of MAC
algorithm negotiation.  I'd be quite happy with this approach.

One issue with this approach is: if we always use TLS (but we might
not), why do the extra session crypto?  What do we gain?  Do we need
to worry about content re-writing proxies, say, as in some 3G
networks?  If we always use TLS then it suffices to ensure that a) TLS
provides confidentiality protection, b) the server cert remains the
same for the length of the login session, c) we have a unique,
unpredictable session ID in the headers (what we might call a cookie,
though we don't want it to be a cookie as such).

In one post you talked about sequencing and replay protection for
chunks.  Adding that to the MAC really gets us close to the MIC token
features/design from RFCs 1964/4121 (Kerberos GSS mech).  We're
talking about having a sequence number.  As you say: this isn't
difficult; we've been doing this for a loooong time in Kerberos land.
Note that there's no need for sequence numbers to randomized given
that we have session keys, but sequence number windows add to the
state to be kept on the server side -- can we tolerate that? Note that
while session key state might be kept on the client in an encrypted
state ticket, session number windows cannot safely be kept that way --
they must be kept locally.  I tend to think that sequencing and replay
protection are the responsibility of the application -- all it needs
to do is add a sequence number to the chunks and manage its own
[per-resource] sequence number windows.

Altogether we need: a session key identifier in the headers (this
should imply algorithm selection), a direction identifier (or separate
keys for each direction), a sequence number if we need sequencing
and/or replay detection, what content to MAC, and the MAC itself.

Regarding what to MAC: the direction flag (unless we have diff keys
for each direction), the channel binding for the TLS channel (if we
have it), the URL?, some subset of headers? and the body.  Note that
applying the MAC to any headers requires that we say something about
canonicalization (e.g., "use the headers exactly as sent") and
canonical order (if a subset of headers) (e.g., "in the relative order
of appearance").  Header and body content need to be unambiguously
separated in the MAC input.  Obviously we can't MAC all headers: some
might be added by proxies, for example.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 11 Jul 2012 19:07:40 -0400

> whence the key?  Also, what should the MAC be applied to?

The MAC should be applied to the 8-bit clean message content (i.e.
precisely that which is bounded by Content-Length)

If we are talking about Web Services then the key would be established
through some application layer key exchange (TBS).

The key requirement from a performance standpoint as I see it is that
the server has to be able to operate in a stateless fashion which
means using a ticket like approach.

> algorithm negotiation.  I'd be quite happy with this approach.

I think that there is definitely an opportunity to make use of a
ticket mode to tie the HTTP authentication to the HTTP channel.

> though we don't want it to be a cookie as such).

TLS is very large, very complex and was engineered from the assumption
that there would be public key credentials on the client. Yes, people
can train it to do other tricks, but doing that is a lot more complex
than doing what we need in HTTP.

In my particular Web Service I am using TLS but still want to have a
transport layer authentication protection because I don't want to do a
TLS public key negotiation on each transaction and I don't want to be
bound to TLS session expiry.

In a large commercial environment the TLS processing is often
completely offloaded onto an accelerator that strips out the TLS and
hands clean IP packets to the Web Service. Another frequent screw case
is TLS proxies like bluecoat devices.

But even in the simplest TLS use case, the TLS security context is
really not exposed to the Web Server or the client in the way you
would need to use it for Web Services authentication in the commonly
used APIs. The problem is that TLS is designed to conceal all the
complexity of crypto from the application. That is why it was called
SSL at the start.

> difficult; we've been doing this for a loooong time in Kerberos land.

Heh, you could use a Kerberos token in my Omnibroker protocol if you
wanted to. But since it is an opaque string of bytes as far as the
client is concerned, well there is no reason to tie it to any one
approach.

People have been using kerberized cookies for years. The problem being
that the cookie is not at all bound to the requests or responses.

> [per-resource] sequence number windows.

The way I was thinking of helping the application was to provide a
feature that allows Content-Integrity header to specify a key modifier
as well as a ticket. The key used to calculate the MAC would then be
the XOR of the modifier and the authentication key associated with the
ticket.

> and/or replay detection, what content to MAC, and the MAC itself.

Direction is already implicit in HTTP requests/responses.

I don't think we need sequencing, we do need a modifier capability though.

> might be added by proxies, for example.

I don't see the need to MAC any headers for a Web Service application.
Put all the information in the content block.

Otherwise we would have to do the sort of thing that DKIM does to sign
headers and copy them.

-- 
Website: http://hallambaker.com/

---

From: Nico Williams <[email-masked]>
Date: Wed, 11 Jul 2012 18:42:42 -0500

If you envision not using TLS but still want HTTP to provide youwith
integrity protection... why stop there?  Why not go whole hog and also
ask for confidentiality protection?

---

From: James M Snell <[email-masked]>
Date: Wed, 11 Jul 2012 16:44:23 -0700

Phillip, just want to make sure that I'm keeping up with the
conversation thus far... Because of the problems inherent in using
Content-Integrity as a Trailer, is the idea then that
Content-Integrity would be a standard Header and that a new Transfer
or Content Encoding would be defined that supports an incremental
integrity check as a component of the encoding?

  GET /some/uri HTTP/1.1
  Host: example.org
  TE: integrity

For instance... something like...

  HTTP/1.1 200 OK
  Content-Type: application/octet-stream
  Content-Integrity: SHA-256; modifier=123...; param="..."
  Transfer-Encoding: integrity

  10
  {chunk of bytes}
  {digest}
  10
  {chunk of bytes}
  {digest}
  0

- James

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 11 Jul 2012 21:51:51 -0400

What are the problems inherent in using Content-Integrity as a Trailer?

Do the legacy clients botch chunked encoding?

---

From: James M Snell <[email-masked]>
Date: Thu, 12 Jul 2012 09:19:13 -0700

> Do the legacy clients botch chunked encoding?

Well, it appears that many existing 1.1 client, server and
intermediary implementations have taken the "trailers are optional"
idea to the extreme and either (a) provide no means of appending
trailers to the output, (b) provide no means of accessing trailers
included in the input or make it quite a bit more difficult to get to
them or (c) do not properly forward trailers on as a chunked message
flows from hop-to-hop. It's not clear at this point exactly how
widespread this problem is but it appears pervasive enough to make a
trailer-based content-integrity option problematic at best.

- James

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 12 Jul 2012 12:32:16 -0400

I suspected as much, but is 're-invent chunked in HTTP/1.1' any more
practical than 'make chunked worked in HTTP/1.1'.

This might be something that we just need to make happen in HTTP/2.0
as part of multiplexing. I am willing to spend some of my personal
time on that.

>From a Web Services point of view, I think we can manage with an
integrity header for cases where we need to support intermediaries and
restrict trailers to cases where we don't. In most cases we are going
to be layering over TLS in any case.

I am not really proposing this header to compete with TLS. The reason
I want it is to enable a closer binding of the authentication layer to
the content than I can get in TLS. In particular I observe that:

HTTP + JSON + Integrity Header ~= WS-*

I can't run TLS on the type of processor typical in embedded systems
for all sorts of reasons. But I can get a JSON stack and a frame
around it.

---

From: James M Snell <[email-masked]>
Date: Thu, 12 Jul 2012 10:00:35 -0700

To provide one example... using the servlet API within Apache Tomcat
7... I can easily access trailers included in a post.. but there is no
obvious means of including a trailer in the response...

Given the input...

  POST /Testing/test HTTP/1.1
  Host: localhost
  Transfer-Encoding: chunked
  Content-Type: text/plain
  Trailer: x-foo
  TE: chunked

  4
  ABCD
  6
  EFGHIJ
  0
  x-foo: test

On the server side...

  protected void doPost(
    HttpServletRequest request,
    HttpServletResponse response)
      throws IOException {

    // outputs null since the trailer hasn't been parsed yet...
    System.out.println(request.getHeader("x-foo"));

    // consume the input
    InputStream in = request.getInputStream();
    int r = -1;
    while((r = in.read(new byte[100])) > -1) {}

    // outputs the value of the x-foo trailer... so far so good...
    System.out.println(request.getHeader("x-foo"));

    // let's do a chunked response and try that...
    response.addHeader("Trailer", "x-foo");
    OutputStream out = response.getOutputStream();
    for (int n = 0; n < 10000; n++) {
      out.write('a');
    }
    // x-foo is never included in the response...
    // there is no obvious means of including trailers
    // in chunked responses using the servlet api...
    response.addHeader("x-foo", "response");
  }

To make matters worse, whether or not the x-foo header is included in
the response is dependent entirely on the number of bytes written to
the output stream. If the number of bytes is small enough, tomcat
buffers everything and sends a non-chunked response, dropping x-foo in
as a header, rather than as a trailer, despite the explicit addition
of the "Trailer: x-foo" header. So far, I've been unable to uncover
any simple way of forcing chunked responses.

Now, to be fair, I could be overlooking something in my analysis, but
if tomcat and the servlet api do have a mechanism for sending trailers
in the response, it is, at best, quite non-obvious.

- James

---

From: Julian Reschke <[email-masked]>
Date: Thu, 12 Jul 2012 19:09:44 +0200

On 2012-07-12 19:00, James M Snell wrote:
> ...

I have to say that I'm surprised that at least it works for trailers in 
a HttpServletRequest...

Best regards, Julian

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 12 Jul 2012 13:39:29 -0400

I am very big on legacy support.

But 'I can't implement this new Web service efficiently unless I
upgrade my server' is a much lower concern for me than 'I can't
implement unless I upgrade' which in turn is much less of a concern
than 'this won't work for granny with her IE4.0'

The ability to put the integrity check at the end is nice, not being
able to do that is not a concern for most Web services where the
transactions are pretty small and easily buffered in memory.

On Thu, Jul 12, 2012 at 1:09 PM, Julian Reschke <[email-masked]> wrote:
> I have to say that I'm surprised that at least it works for trailers in a
> HttpServletRequest...
>
> Best regards, Julian

---

From: James M Snell <[email-masked]>
Date: Thu, 12 Jul 2012 10:40:00 -0700

> practical than 'make chunked worked in HTTP/1.1'.

Unfortunately no. For 1.1 it would appear that the only practical
approach would be An Ugly Hack at either the content or
content-encoding level, neither of which are great options for many
obvious reasons.

> as part of multiplexing. I am willing to spend some of my personal

+1 ... I would very much like to see integrity checking added to
http/2.0 as well as a stronger requirement to support trailers in
general.

---

From: "Ludin, Stephen" <[email-masked]>
Date: Thu, 12 Jul 2012 17:17:39 -0500

I have lost track of which tendril of this thread has gone where, but put
me down as another +1 for Phillip's suggestion.

-stephen