Subject: Content-MD5 and partial responses

From: Mark Nottingham <[email-masked]>
Date: Mon, 29 Jun 2009 12:00:01 +1000

After a quick look, my reading is that a Content-MD5 header on a  
partial response reflects the bytes in that message, rather than the  
whole (non-partial) response:

> The entity-header field "Content-MD5", as defined in [RFC1864], is an
> MD5 digest of the entity-body for the purpose of providing an end-to-
> end message integrity check (MIC) of the entity-body.  (Note: a MIC
> is good for detecting accidental modification of the entity-body in
> transit, but is not proof against malicious attacks.)
>
>   Content-MD5   = "Content-MD5" ":" OWS Content-MD5-v
>   Content-MD5-v = <base64 of 128 bit MD5 digest as per [RFC1864]>
>
> The Content-MD5 header field MAY be generated by an origin server or
> client to function as an integrity check of the entity-body.  Only
> origin servers or clients MAY generate the Content-MD5 header field;
> proxies and gateways MUST NOT generate it, as this would defeat its
> value as an end-to-end integrity check.  Any recipient of the entity-
> body, including gateways and proxies, MAY check that the digest value
> in this header field matches that of the entity-body as received.
>
> The MD5 digest is computed based on the content of the entity-body,
> including any content-coding that has been applied, but not including
> any transfer-encoding applied to the message-body.  If the message is
> received with a transfer-encoding, that encoding MUST be removed
> prior to checking the Content-MD5 value against the received entity.

Also, note that a multipart message is allowed to have C-MD5 on  
individual parts;
> The entity-body for composite types MAY contain many body-parts,  
> each with its own MIME and HTTP headers (including Content-MD5,  
> Content-Transfer-Encoding, and Content-Encoding headers).

For a multipart/byteranges response, this only helps really if they  
apply to the individual parts...

However, I'm wondering what a cache should do when combining partial  
responses that include Content-MD5. This doesn't seem to be addressed  
in 2616, nor in p5 or p6.

It looks like there are two options here;

a) C-MD5 applies to the bytes in the entity-body (as above), and  
therefore we need to specify what a cache does with it when it  
combines partial responses (throw it away?).

b) C-MD5 applies to the *full* response body, avoiding the combination  
issues, and allowing clients to do a MIC of the full response  
(assuming they have it), but removing the ability to do a MIC on a  
partial response on its own.

Anybody aware of C-MD5 being used with partial responses in the wild  
(I'm looking at you, Adobe)?

Cheers,

--
Mark Nottingham     http://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 30 Jun 2009 13:55:58 +1000

RFC3230 (n.b., - standards track) seems to agree:
> HTTP/1.1 defines a Content-MD5 header that allows a server to  
> include a digest of the response body. However, this is specifically  
> defined to cover the body of the actual message, not the contents of  
> the full file (which might be quite different, if the response is a  
> Content- Range, or uses a delta encoding).

Now <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/178>.

On 29/06/2009, at 12:00 PM, Mark Nottingham wrote:

> responses that include Content-MD5. This doesn't seem to be  
> addressed in 2616, nor in p5 or p6.
>
> b) C-MD5 applies to the *full* response body, avoiding the  
> combination issues, and allowing clients to do a MIC of the full  
> response (assuming they have it), but removing the ability to do a  
> MIC on a partial response on its own.
>

---

From: Yves Lafon <[email-masked]>
Date: Tue, 30 Jun 2009 04:37:10 -0400 (EDT)

On Mon, 29 Jun 2009, Mark Nottingham wrote:

> a) C-MD5 applies to the bytes in the entity-body (as above), and therefore we 
> need to specify what a cache does with it when it combines partial responses 
> (throw it away?).
>
> issues, and allowing clients to do a MIC of the full response (assuming they 
> have it), but removing the ability to do a MIC on a partial response on its 
> own.
>
> Anybody aware of C-MD5 being used with partial responses in the wild (I'm 
> looking at you, Adobe)?

A while back, I implemented option a) as it seemed to be the most logical 
interpretation of the spec.

HEAD /Distrib/jigsaw_2.2.6.zip HTTP/1.1
Host: jigsaw.w3.org

HTTP/1.1 200 OK
Content-Length: 9331520
Content-Md5: fBhlh9ttr14YAqe45Yi+xg==

------

GET /Distrib/jigsaw_2.2.6.zip HTTP/1.1
Host: jigsaw.w3.org
Range: bytes=0-1

HTTP/1.1 206 Partial Content
Content-Length: 2
Content-Md5: 1xvdIsi7k7jSh9zm9GrtJQ==
Content-Range: bytes 0-1/9331520

Caches should "throw away" the md5 (after verification of the partial body 
received, and it is up to the cache to recompute the md5 sum of the bytes 
served.

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Yves Lafon <[email-masked]>
Date: Tue, 30 Jun 2009 10:40:47 -0400 (EDT)

> received, and it is up to the cache to recompute the md5 sum of the bytes 

(Well, no at worst it should ask it to the origin server and match it with 
the stored bytes because of the MUST NOT relative to Content-MD5 
generation by proxies).

         ~~Yves

---

From: Adrian Chadd <[email-masked]>
Date: Wed, 1 Jul 2009 10:32:20 +0800

On Tue, Jun 30, 2009, Yves Lafon wrote:

> >a) C-MD5 applies to the bytes in the entity-body (as above), and therefore 
> >we need to specify what a cache does with it when it combines partial 
> >responses (throw it away?).
> >
> >issues, and allowing clients to do a MIC of the full response (assuming 
> >they have it), but removing the ability to do a MIC on a partial response 
> >on its own.
> >

Have you tested how well this particular assumption scales for intermediary
proxies and origin servers working on large objects?

> received, and it is up to the cache to recompute the md5 sum of the bytes 

This requires the origin and cache to slurp in the required reply data off
disk before calculating the C-MD5 header. For small objects this is fine
but for large objects it could mean a -lot- of doubled up disk IO.

C-MD5 would make sense in this situation (as a message integrity check)
if it were a trailer header rather than an initial reply header.

How does this compare to the use of E-Tags, for example, with full and
range responses? Why would you do something different?

2c,

Adrian