Subject: HTTP 'HASH' Method

From: chown <[email-masked]>
Date: Fri, 25 May 2007 14:36:55 +1000

I think a 'HASH' method should be implemented into HTTP, whereby the server
responds with a hash (md5/sha) of the requested resource. This would be a
godsend for large networks which tend to use caching extensively, because
caching-proxy servers could verify the source hasn't changed while creating
a minimal amount of traffic, thereby allow the amount of time the proxy
stores cache to be be greatly increased only at the cost of hard drive
space.
Not only would this benefit caching applications, but as I'm sure you could
imagine, it could be used in may other fields, especially security.

---

From: Adrian Chadd <[email-masked]>
Date: Fri, 25 May 2007 12:50:51 +0800

> imagine, it could be used in may other fields, especially security.

.. E-Tag? If-Modified-Since timestamps? 

Adrian

---

From: Travis Snoozy <[email-masked]>
Date: Thu, 24 May 2007 21:53:35 -0700

On Fri, 25 May 2007 14:36:55 +1000, chown <[email-masked]> wrote:

> I think a 'HASH' method should be implemented into HTTP, whereby the
> server responds with a hash (md5/sha) of the requested resource. This
> would be a godsend for large networks which tend to use caching
> extensively, because caching-proxy servers could verify the source
> hasn't changed while creating a minimal amount of traffic, thereby
> allow the amount of time the proxy stores cache to be be greatly
> increased only at the cost of hard drive space.

Entity tags already fill this role, do they not? Conditional GET
requests and 304 responses? You can do this with the existing HEAD,
too, if I remember the spec properly.

> Not only would this benefit caching applications, but as I'm sure you
> could imagine, it could be used in may other fields, especially
> security.

If you could name one example, for my benefit if nothing else -- I'm
afraid I'm at a loss of imagination. :)

-- 
Travis

---

From: Eric Lawrence <[email-masked]>
Date: Thu, 24 May 2007 21:52:41 -0700

See section 14.15 of http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html.

A properly behaving HTTP proxy and server will set explicit caching headers or ETAGs, which significantly reduces the need for a Content-MD5 header.  In practice, I've never seen it used.

Headers are not protected in HTTP; hence there's no real security value in this approach.
Eric Lawrence
Program Manager
Internet Explorer
Want to view and tamper with HTTP(S) traffic?
Try http://www.fiddler2.com<http://www.fiddler2.com/>

From: [email-masked] [mailto:[email-masked]] On Behalf Of chown
Sent: Thursday, May 24, 2007 9:37 PM
To: [email-masked]
Subject: HTTP 'HASH' Method

I think a 'HASH' method should be implemented into HTTP, whereby the server responds with a hash (md5/sha) of the requested resource. This would be a godsend for large networks which tend to use caching extensively, because caching-proxy servers could verify the source hasn't changed while creating a minimal amount of traffic, thereby allow the amount of time the proxy stores cache to be be greatly increased only at the cost of hard drive space.
Not only would this benefit caching applications, but as I'm sure you could imagine, it could be used in may other fields, especially security.

---

From: Adrien de Croy <[email-masked]>
Date: Fri, 25 May 2007 17:40:10 +1200

the cost of calculating MD5 over a large resource could be a lot.

Also, it's impossible for servers to calculate the MD5 in advance with 
dynamically generated content.

the E-tag field is intended to identify a unique instance of a file, and 
should provide what you're looking for in terms of cache efficiency.  
Combine with If-Match etc headers, it allows an efficient means for a 
caching proxy to save upstream bandwidth using conditional GETs.  It 
also can be a lot cheaper to generate and validate.

MD5 won't provide security by itself, since any intermediary can change 
the content and recalculate the MD5 hash.  You'd need to salt it, or add 
some shared secret known only by the client and server, or use a method 
like DKIM for SMTP to sign the content.

Adrien

chown wrote:
> could imagine, it could be used in may other fields, especially security.
>

---

From: Jeffrey Mogul <[email-masked]>
Date: Fri, 25 May 2007 10:19:55 -0700

As people have already pointed out, if you want to avoid redundant
cache-refills, GET+If-None-Match, together with a server that properly
implements entity tags, already solves that problem.  If you want security,
you probably shouldn't be relying on caching proxies (except *perhaps* in
carefully firewalled limited-access environments).

If what you really want is a hash (and there are indeed reasons to
want this; for example, end-to-end checks against buggy implementations
of range requests/responses), then there are already standards-track
mechanisms that support this: the existing HEAD method, and the
headers defined in RFC3230.  It's pretty hard to deploy a new HTTP
method, and it doesn't seem to be necessary (or worth the fuss) in
this case.

Adrien de Croy writes,

I recently co-authored a paper, in an unrelated context, where we did
a quickie measurement to get a rough idea of the cost: "We measured
the performance of a SHA-1 HMAC over files stored on five 500GB SATA
disks with a 2-core 2GHz Intel Xeon 5130 at 362 MB/s."  (Shah et
al. "Auditing to Keep Online Storage Services Honest", Proc. HotOS-XI.)
This is probably a higher SHA-1 rate than a Web server could support;
on the other hand, relatively few server owners could afford to pay
for anything like that kind of bandwidth, so for GET requests I
doubt it would be a real problem.

But if you want the server to respond with just the hash, and not the
body (i.e., the HEAD-based approach I outline above), then there is a
risk of a DoS attack against the server CPU and file system: a malicious
client repeatedly issues these HEAD requests (against different files
or dynamic content, thus making it impossible to cache the hash
results).  So it would probably have to be an optional feature of
the server, one it could stop doing whenever it was overloaded.
RFC3230 doesn't require to send a hash, even if the client asks
for it.

-Jeff

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 27 May 2007 17:44:17 +0100

> dynamically generated content.

I'm not suggesting MD5 is useful.  But if it was:

Generally dynamic content does not have the size in advance either, so
chunked encoding may be used.  So, any calculated hash can be given in
a trailer instead of the headers at the start of the request, in the
final zero-length chunk.

I think even Etag can be given there (if it also needed to be
calculated as the content was produced), but somehow doubt any caches
would use it.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 27 May 2007 22:08:43 +0200

> calculated as the content was produced), but somehow doubt any caches

[context: there == chunked encoding trailer]

Also doubt the server would be able to process conditionals based on
ETag in such cases as you need to process the conditional before sending
the response, and if the server is unable to build an ETag before it
sends the response I don't see how it could implement the conditionals
either..

However, the day I get to care about trailers in a cache I'd probably
support ETag being there. Also I expect all caches supporting trailers
to be able to both serve the header out on hits, and to process
conditionals based on it on hits so I am not that pessimistic about the
use of ETag in trailers.

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Sun, 27 May 2007 23:40:26 +0100

> sends the response I don't see how it could implement the conditionals

Not quite the same, but related:

Some dynamic servers have server-side caches with complex
prerequisites.  For example, checking the timestamps of all files
(including code) and values of all database entries used to construct
the response - in more advanced schemes, with local invalidation
triggers so those checks are in O(1).

Those servers can efficiently validate an Etag corresponds to the
current version of a server-side cached response - without
regenerating the response.  So they can implement conditionals.

Conditionals even work _without_ a full server-side response cache -
only Etags and prerequisites need to be recorded, not the actual
content, which can be regenerated in the event of a cache miss.

However, those servers can easily generate an Etag in advance of any
calculations.  They just pick a random number for the Etag, and key
the real set of prerequisites against a local database mapping those
numbers to prerequisites.  (Much like is commonly done with opaque cookies).

Where the trailers would be useful is in specifying the Vary header.
Currently, if the server can determine which requests headers are
prerequisites for the page, before it sends anything, it can put those
in the Vary header.  If not, it has to send "Vary: *".  If it worked
to put Vary in a trailer instead, it could be more accurate in those
cases.

-- Jamie

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 28 May 2007 03:25:37 +0200

> to put Vary in a trailer instead, it could be more accurate in those

Should work, but only if the request indicates "TE: trailer" and the
response a "Trailer: Vary". If "TE: trailer" is not seen then Vary is
not allowed in the trailer. 3.6.1 Chunked transfer coding, and the fact
that Vary can not be considered "optional" or safe to ignore while
processing the response.

A cache advertising "TE: trailer" and not handling Vary in the trailer
is broken imho.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 28 May 2007 14:30:46 +1200

trailer headers in chunks leads to a number of implementation 
difficulties especially for proxies.

It seems there are a few features in HTTP that primarily consider agents 
and servers, but not proxies, or specifically administrative 
requirements for proxies.

It makes it very difficult to make decisions based on header values, 
when the header itself may arrive a long time after the main headers of 
a response in some chunk down the line.

And it added a lot of complexity to header processing.

the options available to an administrator don't lend themselves to good 
user experiences. 

For instance a proxy administrator wishing to strictly enforce actions 
based on response header values has a big problem with trailers.  
Stripping any TE header in requests, or blocking responses from servers 
that indicate there will be trailers are about the only decent options.

The overall structure of a message seems to have been muddied with the 
addition of trailers - from a clear delineation between headers and 
resource, separated by 2 CRLF, we now no longer enjoy this clarity.  
Processing of "content" now has to be able to encounter and deal with 
new "headers".  This affects all resource processing, e.g caching, 
scanning etc.

It would be nice if the concept could be deprecated or re-thought before 
someone goes ahead and implements it, but I'm dreaming there I guess.

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 29 May 2007 10:23:34 +0200

> requirements for proxies.

Well, there is considerably more focus on the endpoint, but I am not
sure I agree in this case.

> a response in some chunk down the line.

Sure, but in most cases you don't need to, and it's easy to indicate you
don't support trailers by simply not sending a TE: trailers header.

> And it added a lot of complexity to header processing.

Not really, just that you need to be able to add headers. Quite the same
requirement as the need to be able to rewrite headers on a 304..

> the options available to an administrator don't lend themselves to good 

Not sure I get the point here. The alternatives is

a) Not emit the header at all if it's value can't be realistically
determined before the body has been sent.

b) Emit the header in a trailer for those who can handle it.

and I prefer having the 'b' option available than only having 'a'.

> that indicate there will be trailers are about the only decent options.

Sure, and where is the problem? Such setups gets restricted to 'a' or
requires the server to handle the request in a more complex manner, just
as they would be if trailers did not exists.

> resource, separated by 2 CRLF, we now no longer enjoy this clarity.

Yes.

> new "headers".  This affects all resource processing, e.g caching, 

Well, it's still clearly separated at the transport level (not message
level).

> someone goes ahead and implements it, but I'm dreaming there I guess.

Don't think that's a good alternative. trailers is a good thing when
they can be used.

But there should be a note that servers should if possible try to send
the headers in the message-header when possible, and only use the
trailer if it can't compute the header value before the response.

It's also worth noting that the server MUST know it's going to emit the
header. It's not allowed to add random headers to the trailer without
advertising them in the message-header. So for a server to emit Vary in
the trailer the following conditions must all be true

a) The request had a "TE: trailer"

b) The server realized the content varies, and emits a "Trailer: Vary"
in the message-header telling the recipient that the response varies but
that it's not yet known on which details..

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 29 May 2007 22:30:40 +1200

Thanks for your points on this.  more comments below

> sure I agree in this case.
I meant many do not primarily consider admin requirements for proxies.  
Most primarily consider servers and clients.

Anyway, not much value in my original comment - apologies.  I do get 
frustrated trying to design an administrative interface for a proxy 
product where the protocol makes things difficult, and the value to the 
end-user isn't clearly understood or obviously justified.

I see HTTP getting more and more complex, and hope that it doesn't go 
the way of IMAP.  History has shown complexity and achieved 
interoperability to be conflicting / divergent paths.
> don't support trailers by simply not sending a TE: trailers header.

A proxy needs to understand the TE header to strip it.

There are plenty of good use-cases for switching policy on return 
headers.  Content-Type is a classic one, where certain types of content 
scanning for instance break streaming media that uses HTTP transport.  
Or blocking things like CGI proxies based on a signature header etc.

There's no end to some of the policies admins wish to implement, who am 
I to deny them the capability :)

> requirement as the need to be able to rewrite headers on a 304..

I meant processing of all headers (including trailers) particularly for 
policy just got split over potentially a zillion places instead of one.

> Not sure I get the point here. The alternatives is

Users don't even want to know about let alone think about trailers.  Any 
incursion of trailers onto their reality is seen by most to be an 
annoyance.  So, it's really hard to justify providing GUI in a product 
to deal with how to cope with trailers.

> and I prefer having the 'b' option available than only having 'a'.

I'd be interested if you can point me to any docs on real-life uses of 
option b.  I still haven't seen a server return any trailers, few 
clients advertise support for them.  Is there a restriction on what 
headers may appear in trailers?  A site to test against would also be 
useful.

> Sure, and where is the problem? 
the problem is that when trying to figure out what to do about trailers, 
and policy enforcement, when putting together a request for upstream, 
it's all to easy to decide to simply strip out any TE header from any 
request received before passing it on.  This has the same effect as if 
trailers didn't exist.  It's a simple way to deal with the problem.  But 
what does it break?

the arguably more correct way to deal with it is to allow the user to 
configure what to do if and when headers arrive attached to chunks.  
This has certain downsides:

1. user education. If I don't even understand the true user benefits of 
trailers, how can I educate my users.  Even if I did, it would be 
non-trivial.  Providing an understandable user interface that doesn't 
just put people off the product... etc etc.
2. performance.  Need to evaluate policy on headers possibly many more 
times - whenever a trailer arrives. 
3. A subsequent trailer can potentially invalidate previous policy 
considerations.  Policy logic may not lend itself to correct evaluation 
when  matching is piecemeal.  This is the clincher for me.

So for the moment, I have chosen to deprecate trailers, which leaves me 
uncomfortable that some day, someone will require them to work and I'll 
be faced with this problem again.

> Such setups gets restricted to 'a' or

It seems to me that implementation of trailers in a server would 
generally be thought by developers to be more trouble than they are 
worth.  If they have to support the non-trailer case anyway.  Is support 
for trailers optional?  I didn't think it was.

> that it's not yet known on which details..
hmmm, maybe I need to re-read the section on trailers.  I remember the 
bit about advertising which headers will be in trailers, but I don't 
recall it covering things like how many times a trailer may appear, 
whether there are any headers that cannot appear in a trailer (and must 
be in the response) etc.  Obviously the Trailer header must not appear 
in the trailer I guess.  But what about headers like Content-Type

there may be an opportunity for optimisation of policy where advertised 
trailer headers aren't used in the policy evaluation.

Still, I'm having fun trying to figure out how for instance a server can 
provide an API to a CGI/ISAPI/NSAPI/etc so that the module can emit 
trailers.  unless the module does the chunking itself as well.  Maybe 
that's why I've never seen one.

Speaking of which, is there a decent specification for the allowable 
output of a CGI?  Is it valid for a CGI process to emit chunked data?

Thanks

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 29 May 2007 19:22:05 +0200

> A proxy needs to understand the TE header to strip it.

Well, it's a MUST requirement in HTTP/1.1 to strip this. TE is a
hop-by-hop transport header. A HTTP/1.1 proxy not getting the
transport/message or hop-by-hop/end-to-end distinctions is not worthy to
advertise itself as a HTTP/1.1 proxy.

> to deal with how to cope with trailers.

Then don't support them. It's a thing the proxy must take an active
decision to support, not something which should get proxied by accident.

> headers may appear in trailers?  A site to test against would also be 

Of course there is restrictions. Transport headers is not allowed in the
trailer. See 14.40 Trailer.

> trailers didn't exist.  It's a simple way to deal with the problem.  But 

It shouldn't break anything, but may result in either

a) Somewhat heavier processing demands on the server to produce the
header before the content.

or

b) Some headers not available

or

c) Some header values not as accurate as they might be if the server
could place them in the trailer.

> configure what to do if and when headers arrive attached to chunks.  

chunks?

headers is only allowed in two places. The message-headers, and in the
final trailer header section in the "eof" chunk.

> just put people off the product... etc etc.

Trailers is meant to be used for things which can not be evaluated
before the response. In RFC2616 this is mainly Vary for a dynamic
varying page, and Content-MD5, and maybe ETag if the server wishes to
provide a very strong ETag. Cache-Control and Last-Modified might also
be candidates.

Note: Placing ETag and Last-Modified in the trailer isn't very wise as
it makes it harder to cache partial responses.

> times - whenever a trailer arrives. 

Yes, but you'll know beforehand if this may be required from the Trailer
message-header.

> be faced with this problem again.

For now the use of trailers is an optional extension.

> for trailers optional?  I didn't think it was.

Support for trailers is optional, only used if supported by both client
and server, and the server is having an interest in making use of it...

> in the trailer I guess.  But what about headers like Content-Type

There is only at most one trailer, in the final "eof" chunk.

       Chunked-Body   = *chunk
                        last-chunk
                        trailer
                        CRLF

       last-chunk     = 1*("0") [ chunk-extension ] CRLF

Are you maybe mixing up trailers and chunk-extensions?

> that's why I've never seen one.

The standard CGI doesn't support trailers.

Not sure about ISAPI.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 30 May 2007 09:07:42 +1200

I must have my wires crossed sorry.

Chunk extensions aren't problematic, as they aren't headers.

I must have just mis-read where trailers could occur.  It's clear from a 
closer look that they can only occur after the last-chunk.

I can still forsee some issues if headers like Content-Type are 
permitted in a trailer though.

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 29 May 2007 23:12:13 +0200

> permitted in a trailer though.

It is permitted, but no sane implementations should place it there, or
need to place it there..

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 30 May 2007 09:28:06 +1200

> It is permitted, but no sane implementations should place it there, or

That's an opportunity for another RFC perhaps - the "things should be 
sane" RFC :)

I see this more potentially as a possible exploit by rogue servers to 
bypass security etc.  e.g. porn sites or similar.

There aren't really that many response headers that it makes sense to 
base policy on, but Content-Type is a key one.

Regards

Adrien

---

From: Mark Karpelès <[email-masked]>
Date: Fri, 25 May 2007 04:51:33 +0000

A "hash" method would require the webserver to cache the md5/sha1 values in order to minimize disk access (computing cache on big files is using a lot of time). 

I'd suggest to use a kind of hash based on the inode number, the file size and modification time. This would just require a call to stat() and return the same value until the file gets changed. 

What ? It already exists and is called "E-Tag" ? Wow, I guess someone thought about it before me. 

So, your large network which tend to use caching extensively just have to send a request with HEAD query, or even better - a head with a If-Modified-Since header, and you're saved.