Subject: Content-Integrity header

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 6 Jul 2012 14:43:38 -0400

HTTP 1.0 specifies a header Content-MD5.

This is a bad header for all sorts of reasons, not least that the
header was added despite the fact that I told people that MD5 had just
been severely compromised. In particular there is only one digest
algorithm specified and no way to use others.

A better approach would be:

Content-Integrity: <base64-value> ;alg=<ID>

For many Web Services, what we would like to do is to specify a MAC
rather than a digest and to use a preshared key identified by some
form of kerberos-like ticket. We don't need to specify the internal
structure of the ticket, just accept that we will have some sort of
key exchange service interaction at the end of which the client ends
up with a shared secret, an algorithm and an identifier that can be
passed to the service where the interaction takes place:

Content-Integrity: <base64-value> ;ticket=<ticket-data>

Note that the method of establishing that ticket in the first place is
out of scope for HTTP. Just think of it as something akin to a cookie.
There are many ways that structure can be established but those are
for the Web Service to manage, not something we should bind into HTTP.

This approach is of course very similar to the www-authenticate header
but with the major distinction that WWW-Authenticate is really about
key establishment and this is a mechanism to apply the result of a key
establishment to a content payload. But I guess if people think it
better to re-use WWW-Authenticate than to introduce a new header, I
can go with that.

The way we authenticate Web Services transactions today is of course
through either TLS or by embedding the authentication data into the
content through something like JSON Signature or XML Signature or via
something like WS-*. I don't particularly like these approaches as the
content authentication is really meta-data for the content and that
means it belongs in the HTTP header and not in the transport layer or
in the content itself. That is what the HTTP headers are there for,
making meta-data statements about the content.

We seem to have got into a position where Web Services means 'just run
over HTTP as if it was TCP/IP with firewall bypass'. I think we could
make Web Services a lot simpler by using HTTP for the purpose it was
intended and not duplicating that functionality in SOAP or whatever.

I have discussed this with David Crocker and his DOSETA scheme. I
think this is a case where if HTTP has the right socket, we can both
plug into it.

-- 
Website: http://hallambaker.com/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 06 Jul 2012 19:08:12 +0000

In message <[email-masked]>

>Content-Integrity: <base64-value> ;alg=<ID>

Wouldn't you need more fields than that ?

A nonce or psk id for instance ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 6 Jul 2012 13:02:52 -0700

> HTTP 1.0 specifies a header Content-MD5.

Actually, HTTP/1.1 (draft-ietf-http-v11-spec-02.txt, April 1995).

> been severely compromised.

Compromised in 1995?  I don't think so, and certainly not in a way
that would have effected a MIC for *accidental* modification
(which is all it was intended to be).  It was never intended for
authentication because the field can be altered by an attacker
just as easily as the content.

> In particular there is only one digest

Of course there is another way -- just define another field for
that algorithm.  Defining a new content-sha256 (or whatever) field
is just as effective (and efficient) as trying to put all integrity
checks into a single field with an alg parameter.  Moreover, the
new field definition could be used without worrying about all of
the broken implementations of poorly defined integrity checks
like content-md5.

> Content-Integrity: <base64-value> ;alg=<ID>

That is another approach.  It means that a recipient that only
understands alg A has to parse each content-integrity field-value
to look for a matching A parameter, as opposed to simply checking
for the presence of a content-A field.  It has benefit if we expect
integrity checks to be performed by a multi-algo processor like
openssl, but is a nuisance if we expect only one or two algorithms
to be in common use at the same time.

> out of scope for HTTP. Just think of it as something akin to a cookie.

Why implement a MAC using the same field as a MIC?
Digital signatures are much harder to implement.

Why don't the existing digital signature mechanisms suffice?
I know they haven't been deployed well for HTTP, but I don't know
why other than the expired patents and chicken/egg problems.

Why do you think a generic integrity check processor would want
to implement anything that complex?

I agree that some common Dsig standard is needed, and that a good
way to do that is to simply assume that the shared key is already
known via other means, but I don't see any need to tie that to a
generic integrity check field when two different fields would be
easier to define, argue, and implement.

In any case, the hard part is deciding what content is included
in the hash.  Sometimes folks want to include the whole representation
(as it exists at the source), others want to include only the payload
of the current message, and still others want the payload and some
odd set of header fields.  And then we start talking about canonical
forms, and pretty soon we are just sick of arguing about it and ship
some gunk that doesn't actually solve a real problem.

Trying to do everything at once is a recipe for madness.  Just pick
a very specific problem to solve, solve it the most efficient way
for that problem, and use a field name specific to the solution.
If it turns out to be useful beyond that, that's gravy.

....Roy

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 6 Jul 2012 17:16:10 -0400

On Fri, Jul 6, 2012 at 4:02 PM, Roy T. Fielding <[email-masked]> wrote:
> (which is all it was intended to be).

Yes, Dobertin circulated his paper showing an initial collision result
in private before the header was put in the spec. I remember arguing
against the header before it was put in the spec knowing that it was
broken.

>  It was never intended for

It is not intended for authentication but the ability to create
collisions does have some very significant security consequences.

> to be in common use at the same time.

If you are using something like IIS where the header values are mapped
to class members or attributes, adding a new header is a big deal,
adding a parameter value is not.

> Digital signatures are much harder to implement.

This is not a Digital Signature. In fact repudiation is quite often an
undesirable property.

> why other than the expired patents and chicken/egg problems.

Because they are digital signature mechanisms for a start. And that
means they are tied to things like PKI that are really application
layer not platform layer.

> to implement anything that complex?

MACs are not complex, see Kerberos.

> easier to define, argue, and implement.

One of the crossover cases is where a MAC is used with a specified
key. This has some interesting properties as it allows a proof of
knowledge of a content item to be established.

For example, say we have a HEAD request that specifies a key to be
used to calculate a MAC value. We have a lightweight means of seeing
if the service actually stores the content it claims.

> some gunk that doesn't actually solve a real problem.

No, the content-integrity header would only cover the content and
nothing else. That is why it is Content-Integrity and not
'Content-and-random-headers-Integrity.

> If it turns out to be useful beyond that, that's gravy.

Then we end up with the detritus of everyone's legacy one-off
solutions like Content-MD5.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 6 Jul 2012 17:25:24 -0400

> A nonce or psk id for instance ?

The second example has a pre-shared Key ID, essentially a Kerberos
ticket in most cases, I would guess.

Nonces are important but I tend to think of them as something that
should go inside the message content rather than have the transport
binding have to engage with them. Quite often a Web Service
transaction will be split across a series of HTTP transactions and
connections and you want the nonces to carry across the Web Service
Transaction.

Another reason is that a nonce is often useful as an identifier in the
Web Service where it can double as a transaction ID.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 6 Jul 2012 18:12:59 -0400

Someone pointed me to this in a private message:
http://tools.ietf.org/html/rfc3230#section-4.3.2

This is a Proposed Standard and defines a Digest header that covers
the digest only functionality.

The digest header is not really suited to extension so that suggests
that a MAC only header such as Content-MAC or MAC would be more
appropriate.

---

From: James M Snell <[email-masked]>
Date: Fri, 6 Jul 2012 16:23:32 -0700

In general, I'm +1 on the general idea albeit with a few caveats...

1. To minimize complexity, only a single Content-Integrity header
should be used. I don't want, as Roy points out, to have to iterated
through a bunch of unsupported header values looking for the one I
want. Just as it makes very little sense for an implementor to provide
multiple Last-Modified, Etag and Content-Type headers in a single
message; there should be only a single Content-Integrity statement and
I either understand it or I don't.

2. The performance impact of calculating the digest needs to be
carefully considered. I'd rather not be required to buffer a full
representation in memory all the time just to calculate a header
value. I know it's largely unavoidable, but perhaps there's some
currently elusive solution that can be considered. For instance..
allowing Content-Integrity to appear as a trailer at the end of a
chunked response.

3. Something needs to be said about what happens if the
Content-Integrity check fails. For instance, if a request containing
Content-Integrity is sent to the server and the server detects that
the signature is invalid, what should happen? what must happen?
Likewise, how are intermediaries expected to treat the
Content-Integrity header given that any intermediary is able to modify
the payload at any time?

- James

---

From: Martin Thomson <[email-masked]>
Date: Fri, 6 Jul 2012 19:17:13 -0700

> representation in memory all the time just to calculate a header

Trailer: Digest

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 6 Jul 2012 22:37:47 -0400

I was just looking at the spec to make sure it was still in.

Specifying digest and MAC values is exactly what I intended the
trailer in chunked encoding to be used for.

       Chunked-Body   = *chunk
                        last-chunk
                        trailer
                        CRLF
       chunk          = chunk-size [ chunk-extension ] CRLF
                        chunk-data CRLF
       chunk-size     = 1*HEX
       last-chunk     = 1*("0") [ chunk-extension ] CRLF
       chunk-extension= *( ";" chunk-ext-name [ "=" chunk-ext-val ] )
       chunk-ext-name = token
       chunk-ext-val  = token | quoted-string
       chunk-data     = chunk-size(OCTET)
       trailer        = *(entity-header CRLF)

So given that we have Digest: defined as a header, should we try to
work from that or add in a separate header for MAC?

I prefer the second given where we are.

---

From: "HAYASHI, Tatsuya" <[email-masked]>
Date: Tue, 10 Jul 2012 04:08:48 +0900

+1

I know that this is demanded.
When I discussion about http-authentication and phishing,
it is requested by many people.
It is a difficult problem.
ex) proxy...

I think that it is good to do this discussion now.

---
Tatsuya

-- 
HAYASHI, Tatsuya
Lepidum Co. Ltd.

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 10 Jul 2012 11:00:49 +1200

On 10.07.2012 07:08, HAYASHI, Tatsuya wrote:
> On Sat, Jul 7, 2012 at 8:23 AM, James M Snell wrote:

Either the client advertises what it supports (opening itself to 
middleware erasing options they can't modify). Or the server uses 
multiple algorithms in hopes that the middleware cannot violate them 
all.
It makes perfect sense to have several levels of integrity check. MD5, 
SHA1, AES in one response and allow the client to validate the strongest 
it can handle.

There is also an arguable case for middleware wanting to add its own 
hash to inform the client essentially "this is what I got given". So the 
point of manipulation can be back-traced when the more secure end-to-end 
checks fail.

If you want end-to-end integrity, don't stop at half measures. 
Particularly at half measures which can be corrupted.

>> allowing Content-Integrity to appear as a trailer at the end of a

As has been said Trailers happen here.

>> Content-Integrity header given that any intermediary is able to 

This is going to be most useful on request/responses sent with 
"no-transform" of course.
  If the integrity was only a MD5 or SHA1 hash which middleware can edit 
easily there is no end-to-end integrity, just hop-by-hop integrity.

Also, there has to be a mutual secret between origin server and client. 
Without that, when integrity is compromised the transforming hop will 
simply erase or replace the Content-Integrity header value. A secret key 
unknown to that middleware is required to make the integrity hash break 
when it tries this.

AYJ

---

From: "Ludin, Stephen" <[email-masked]>
Date: Tue, 10 Jul 2012 13:33:52 -0500

I really like the idea of placing the Digest in the chunk trailers.  Being
able to calculate these digests on the fly and not buffer the entire
message is critical in my opinion.

Another concept that I have been playing with is providing digests on
individual chunks using chunk-extension.  The rational for this is for
very large objects.  With per-chunk digests the client would have the
ability to re-request a specific corrupted section of an object using a
range request rather than the entire object.  This can have enormous
perceived performance and reliability benefits for consumers of things
such as software download and large media files.

I was working on a draft to propose this, but I didn't feel it was well
baked enough to share.  If there is interest in this type of functionality
I will polish it up and post it.

One issue to point is is that for these types of "frame" based integrity
checks I generally feel like we are reinventing the content integrity
portion of SSL/TLS.  Though I see the value in begin able to do this apart
from SSL it forces the question at what point do you just switch over to
SSL to get the desired functionality?

-stephen

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 10 Jul 2012 15:00:52 -0400

Transport and application are different layers. It is not an either/or
thing. Often you want both.

What I am proposing does have an overlap with WS-* but it can be done
with one header and a five to ten page description rather than a
massive protocol stack.

TLS does have its place, but crossing credentials across from the
application layer to the transport layer is a complex problem. I see
credential management as something that should be protected in the
application even if you are going to use TLS or IPSEC. In fact my
particular application mandates HTTP over TLS in addition to service
level authentication checks.

More generally, I see this as an approach that would work very nicely
in an embedded systems context where authentication is demanded but
confidentiality may even be undesirable.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 11 Jul 2012 09:59:02 +1000

chunk-extensions aren't widely supported in APIs, and usually dropped hop-by-hop (then again, trailers can be too).

Some discussion of trailer support in Apache:
  http://apache-http-server.18135.n6.nabble.com/HTTP-trailers-td4796242.html

… and using Trailers to surface debug information in Firebug:
  https://groups.google.com/forum/?fromgroups#!topic/firebug/v5ldjoVThH8

(yes, this is a bit of a project for me)

Finally, I know some server-side implementers would would LOVE to be able to set things like ETag and Last-Modified in trailers, and have them used by caches...

Cheers,

--
Mark Nottingham
http://www.mnot.net/

---

From: James M Snell <[email-masked]>
Date: Tue, 10 Jul 2012 19:50:00 -0700

Yeah, this has been my experience as well. Unfortunately, I'm not sure
if this is something that can be effectively fixed. Off the top of my
head, I don't know of a single server or client side http stack that
is capable of easily setting Trailers in the stream. I just checked
and support for trailers is still missing in the Servlet API, for
instance.

I know this kind of thing is generally frowned upon, but one possible
approach is to make use of a special content-type... for instance...

  HTTP/1.1 200 OK
  Trailer: Content-Integrity
  Content-Type: application/http-trailers; type="text/plain"

  foobar

  Content-Integrity:
sha-256:aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f

This is just a strawman, but essentially, it would append a checksum
to the end of the content stream. It's not a trailer in the
traditional sense, it's actually encoded as part of the payload. In
lieu of proper Trailer support this is the only other way I can see it
working effectively in HTTP/1.1.

For HTTP/2.0, I suppose it would be possible for us to take a more
appropriate approach either with trailers or by building the integrity
mechanism into the framing somehow.

(Warning: thinking out loud *before* having consumed any beer... the
ideas expressed above may not make any sense. May need to apply
alcohol and try again)

- James

---

From: Brian Pane <[email-masked]>
Date: Tue, 10 Jul 2012 20:25:49 -0700

> Finally, I know some server-side implementers would would LOVE to be able to set things like ETag and Last-Modified in trailers, and have them used by caches...

The last time I looked at this issue, in the context of an optimizing
load balancer where we would have loved to set ETag in the trailers, I
found that browser support was almost nonexistent. Things haven't
improved much since then:
http://www.browserscope.org/?category=network

If there's widespread interest in trailer support, HTTP/2.0 seems like
a great opportunity to make it mandatory.

-Brian

---

From: James M Snell <[email-masked]>
Date: Tue, 10 Jul 2012 20:52:53 -0700

On Tue, Jul 10, 2012 at 7:50 PM, James M Snell <[email-masked]> wrote:
> [snip]

Oh, forgive me, I had a beer and realized I made an error in the
sample, the content-type really should be...

  application/ugly-hack-to-work-around-broken-implementations+http

Without either a hack of this nature or a new Transfer-Encoding that
we could somehow convince folks to support, there's really no other
option but to punt and make trailers required in 2.0.

- James

---

From: Yutaka OIWA <[email-masked]>
Date: Wed, 11 Jul 2012 14:04:54 +0900

Another reason to provide per-chunk integrity is to allow
clients to progressively use the content while receiving.
Progressive handling of large objects is common in
current Web systems, especially for large resources
like texts (HTMLs, PDFs), images, movies, and others.
Without HTTP-level integrity protection, it works well.
(TLS provides checking integrity of partial stream.)

However, if integrity protection is provided by
either a trailer or a header, clients must firstly store all
received data without further processing, check integrity,
and then decide whether to accept.

To this purpose, chunk-based integrity protection is
even useful for static contents for which hash values can be
pre-computed.

"Re-inventing the wheel" issue is still alive :-)

P.S.
As a personal feeling, the value of integrity protection on
this moment is more on protection against intentional content-forging
attacks, rather than unintentional communication failure
(except premature termination of streams).
To this extent, re-requesting the broken chunk is personally out of
my interest.  This may need discussion, because it may affect
the design of inter-chunk chaining of integrity signatures.

-- 
Yutaka OIWA, Ph.D.              Leader, Software Reliability Research Group
                             Research Institute for Secure Systems (RISEC)
   National Institute of Advanced Industrial Science and Technology (AIST)
                     Mail addresses: <[email-masked]>, <[email-masked]>
OpenPGP: id[440546B5] fp[7C9F 723A 7559 3246 229D  3139 8677 9BD2 4405 46B5]

---

From: Zhong Yu <[email-masked]>
Date: Wed, 11 Jul 2012 02:51:21 -0500

We can also define a new content encoding for integrity check.

It's easy to program an individual content encoding (if it focuses on
just one thing), and it's easy to compose different content encodings.
For example, in Java servlet API, integrity check can be easily
implemented as a filter, and it's easy to install the filter.

The overhead of extra framing due to new content encoding should be marginal.

On "Transfer-Encoding: chunked": Its primary function was to mark the
end of message. While it's tempting to piggyback other features on it,
the lack of support of chunk-extension and trailer suggests that the
complexity they introduce exceeds what engineers are willing to
handle. I think we should leave it at that - the sole function of
chunking is to mark end of message; don't stuff other features in it.

Zhong Yu

---

From: Wenbo Zhu <[email-masked]>
Date: Wed, 11 Jul 2012 01:01:01 -0700

> and support for trailers is still missing in the Servlet API, for
There is no new API required, necessarily, to support trailers:
1. app code needs set the header Trailer: <header_name> before the response
is committed;
2. container will ignore the header  <header_name> when the response is
committed;
3. container will output the trailer when the response is finished.

The request header "TE: trailers" may or may not be respected.

> > chunk-extensions aren't widely supported in APIs, and usually dropped
> hop-by-hop (then again, trailers can be too).
> >
> > Finally, I know some server-side implementers would would LOVE to be
> able to set things like ETag and Last-Modified in trailers, and have them
> used by caches...
> >
> > Cheers,
> >
> >

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 11 Jul 2012 09:47:28 -0400

> (TLS provides checking integrity of partial stream.)

+1 the only issue would be that it would be necessary to specify how
the integrity check carried across from chunk to chunk and there might
be a replay attack issue unless there was some way of introducing an
IV or equivalent.

Which I can add...

We have missed the -00 cut off date so I can't write this up as a
separate draft but I can add it into my omnibroker draft. I will move
the integrity header stuff out to an appendix.

> and then decide whether to accept.

That is certainly a reasonable issue for the browser display use case.

> "Re-inventing the wheel" issue is still alive :-)

I don't think so.

At the moment we have people busy re-inventing this particular wheel
as content-payload. We have XML Signature, and we have WS-*.

Those have all become big and heavy and a big part of the reason for
that is that they are re-inventing features that properly belong in
the presentation layer (i.e. http in this case).

This is not a signature header, it is an integrity check. There is a
very important distinction. The only reason to use signature over a
MAC is to get non-repudiation. But that means nothing unless you save
the bits that were signed.

> the design of inter-chunk chaining of integrity signatures.

I would see an integrity fault as being the same as a broken
connection. HTTP does have some consideration for that problem already
(range request). If any more mechanism was desired, people can ad it
separately. But it is a totally different issue.

---

From: Zhong Yu <[email-masked]>
Date: Wed, 11 Jul 2012 13:30:56 -0500

If the value of a trailer is a function of the message body, and the
function is app specific, it's a little hard for filters (who
intercept and transform responses) to correctly transform the value of
the trailer.

---

From: "Ludin, Stephen" <[email-masked]>
Date: Wed, 11 Jul 2012 16:27:05 -0500

On the "Expression of Interest" side, I would be very interested in
functionality that allows flexible piece-wise integrity checking that
enables the user-agent to detect an issue and re-request a specific part
on an object in question.  If forced into 1.1 land, Chunk trailers seem
like a logical place to implement this.  As suggested below however, a
separate transfer-encoding entirely may be a better way yo go. I'll need
to tool on that a bit.

Akamai will certainly support something like this.  To Mark's point about
intermediaries potentially removing the trailers, I think that is and
always will be true for anything under the 'Transfer-Encoding' umbrella.
This is what differentiates this feature from the Content-MD5 header.  It
does not take away from the functionality necessarily.  I expect that with
a feature like this smart surrogates may even ADD the trailers on behalf
of an origin to provide middle/last mile integrity.  Many ways to use this
one.

-stephen

On 7/11/12 11:30 AM, "Zhong Yu" <[email-masked]> wrote:

>>> > Š and using Trailers to surface debug information in Firebug:
>>> >   
>>> >> from SSL it forces the question at what point do you just switch
>>>over
>>> >> to
>>> >>>>> message; there should be only a single Content-Integrity
>>>statement
>>> >>>>> and
>>> >>> hash to inform the client essentially "this is what I got given".
>>>So
>>> >>> the
>>> >>> simply erase or replace the Content-Integrity header value. A
>>>secret
>>> >>> key

---

From: "Ludin, Stephen" <[email-masked]>
Date: Wed, 11 Jul 2012 15:36:21 -0600

>the design of inter-chunk chaining of integrity signatures.

This was where I started when I began looking into corruption issues in
downloaded objects.  What we discovered is that in a large enough sample
size, such as what we see at Akamai, random bit shifting that manages to
still get by the TCP chucksum happens regularly enough this feature is of
high interest.  I actually see this as relatively useless for content
forgery cases because, as mentioned elsewhere in this thread, the forger
would probably simply change the digest to match the forgery.  The
digesting feature in general is an act-of-nature (or act of bad hardware /
software) detection mechanism, and building in an efficient recover
mechanism seem to be a reasonable step.

-stephen

---

From: Nico Williams <[email-masked]>
Date: Wed, 11 Jul 2012 16:55:57 -0500

> mechanism seem to be a reasonable step.

We had a bad NIC on a router or fileserver (can't remember which) at
Sun once that produced lots of errors, most of which were caught by
checksums, but occasionally there were two bits flipped at just the
right distance from each other that all checksums in the stack checked
out.  This ended up manifesting itself as a corrupted SCCS file.
Scary!

Nico
--

---

From: Zhong Yu <[email-masked]>
Date: Wed, 11 Jul 2012 16:59:44 -0500

Transfer-Encoding and Content-Encoding overlap in functionality. For
example, gzip can be done in either way. But in reality, gzip is only
implemented with Content-Encoding, for whatever reason. It might be
safer to introduce a new content-coding than a new transfer-coding.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 11 Jul 2012 18:35:02 -0400

A message digest should be more than sufficient for detecting
non-malicious modification. But it turns out that we already have a
header for that.

Still, it makes good sense to look at both together as I don't think
the Integrity header has had as much play as it deserves and adding in
a MAC capability (separate or same header) will stir up much of the
same muck.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 11 Jul 2012 18:38:15 -0400

> intercept and transform responses) to correctly transform the value of

The objective of using a MAC is to detect such modification so that
the messages can be rejected.

It isn't something I would see a lot of use for in Web browsing but
there is a huge need for the capability in Web Services where people
seem to keep inventing additional packaging layers just to fit the
security data in.

For example, take a look at JSON signature where we are being asked to
BASE64 encode stuff that could and should be binary.

---

From: Nico Williams <[email-masked]>
Date: Wed, 11 Jul 2012 18:01:42 -0500

+1

(If all we want is detection of non-malicious modification then the
old MD5 digest header will do, really :)  Except that that only covers
the content, not any of the headers -- is that OK?)

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 12 Jul 2012 12:15:14 +1200

> safer to introduce a new content-coding than a new transfer-coding.

The Content-Encoding is about the entity. End-to-End.
The Transfer-Encoding is about the channel. Hop-by-Hop.

gzip is not implemented as Transfer-Encoding due to all the reasons 
Willy and I put forward against mandatory gzip compression on SPDY 
connections. It is a large performance loss to de/re-compress at every 
hop. chunked on the other hand is trivial to recode and when used right 
increases performance.

When implementing anything which intermediaries *need* visibility to 
and possibly even manipulation of, Content-Encoding is the worst way to 
do it.

AYJ

---

From: Zhong Yu <[email-masked]>
Date: Wed, 11 Jul 2012 20:43:55 -0500

Intermediaries are required to decompress a gzip transfer-coding?

Intermediaries are forbidden to decompress gzip content-coding to look
into the content? Intermediaries are forbidden to change
Accept-Encoding and Content-Encoding?

On Wed, Jul 11, 2012 at 7:15 PM, Amos Jeffries <[email-masked]> wrote:
> gzip is not implemented as Transfer-Encoding due to all the reasons Willy
> and I put forward against mandatory gzip compression on SPDY connections. It
> is a large performance loss to de/re-compress at every hop. chunked on the
> other hand is trivial to recode and when used right increases performance.
>
> When implementing anything which intermediaries *need* visibility to and
> possibly even manipulation of, Content-Encoding is the worst way to do it.
>
> AYJ
>
>

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 12 Jul 2012 14:27:08 +1200

> Intermediaries are required to decompress a gzip transfer-coding?

Yes. Transfer Encoding must be removed when caching, when transforming, 
when interpreting the body, or when transcoding to a hop with different 
negotiated Transfer-Encoding type.

There is a very limited set of middleware (essentially just SOCKS or 
VPN tunnels) which can avoid it. All the rest have to implement the 
encoding.

> Accept-Encoding and Content-Encoding?

Why do you insert this word "are forbidden to"?

It is a *performance* loss issue, not a permission one, for the large 
majority of intermediary types which do not need to touch the entity 
integrity in any way. Just look at how many of the transforming proxies 
that do content filtering find it necessary to force identity encoding 
from servers just to operate at any reasonable speed.

AYJ

---

From: Zhong Yu <[email-masked]>
Date: Wed, 11 Jul 2012 21:54:43 -0500

If there's not an official policy mandating different use of the two,
I don't see any essential technical difference between transfer coding
and content coding.

  TE: gzip
  =>
  Transfer-Encoding: gzip chunked

works equally well as

  Accept-Encoding: gzip
  =>
  Content-Encoding: gzip

for anyone on the path.

On Wed, Jul 11, 2012 at 9:27 PM, Amos Jeffries <[email-masked]> wrote:
> Yes. Transfer Encoding must be removed when caching, when transforming, when
> interpreting the body, or when transcoding to a hop with different
> There is a very limited set of middleware (essentially just SOCKS or VPN
> tunnels) which can avoid it. All the rest have to implement the encoding.
>
>
> integrity in any way. Just look at how many of the transforming proxies that
> do content filtering find it necessary to force identity encoding from
> servers just to operate at any reasonable speed.
>
> AYJ
>
>