Subject: Use hash to reduce traffic

From: Tim Tailor <[email-masked]>
Date: 05 May 2003 17:05:40 +0200

Hi there!

I just had an idea how traffic on the web could be dramatically reduced.
May be that it is already implemented in some way and I just do not know
about it.

Now the idea:
Integrate a request command to http which will be replied by the hash of
the page only. Based on this information the webbrowser or proxy could
decide whether it needs to download the whole page or not. Moreover a
html tag could indicate pseudo-new information like timestamps and make
this hash even more effective.

That's it. Little idea, lot of work and great improvement (at least I
assume that, otherwise I would not bother anybody with it). Thanks for
your time and I would be glad to get some comments on that.

Bye Tim

---

From: Scott Lawrence <[email-masked]>
Date: 05 May 2003 12:31:17 -0400

> May be that it is already implemented in some way and I just do not know

RFC 2616, section 14.19

-- 
Scott Lawrence        
  Actively seeking work 
  http://skrb.org/scott/
  [ <[email-masked]> is deprecated ]

---

From: Jeffrey Mogul <[email-masked]>
Date: Mon, 05 May 2003 17:21:49 -0700

Tim Tailor writes:

    I just had an idea how traffic on the web could be dramatically
    reduced.  May be that it is already implemented in some way and I
    just do not know about it.

    Now the idea:  Integrate a request command to http which will be
    replied by the hash of the page only. Based on this information the
    webbrowser or proxy could decide whether it needs to download the
    whole page or not. Moreover a html tag could indicate pseudo-new
    information like timestamps and make this hash even more
    effective.

See this paper:

	Terence Kelly and Jeffrey Mogul.
	"Aliasing on the World Wide Web:
		Prevalence and Performance Implications."
	In Proc. 11th Intl. World Wide Web Conf., pages 281-292.
	Honolulu, HI, May, 2002.
	http://www2002.org/CDROM/refereed/525.pdf

especially section 7.

We've been working on a more complete paper about this idea, but
it's been a spare-time activity and neither of us has had enough
spare time.

Scott Lawrence writes:

	RFC 2616, section 14.19

Nope, this doesn't solve the aliasing problem.  And if you
reply "RFC 2616, section 14.15" then I would rebut with
"RFC 3230".

-Jeff

---

From: Tim Tailor <[email-masked]>
Date: 06 May 2003 13:16:06 +0200

Hi :-)

Thanks for your replies. It will take me some time to read and
understand all the stuff you mentioned. Up to now I only quickly went
over the TOC's and from that I am not sure if it really is what I meant.
The pdf linked by Jeffrey gets pretty close to it but as I said before,
I am not yet sure. I will get back to it at latest after finishing my
exams for this semester. 

Bye Tim

---

From: Diwakar Shetty <[email-masked]>
Date: Tue, 06 May 2003 18:46:01 +0530

Dont we have "Modified-Since" and "Etags" to do this job already ?
What will "hash" do extra which is not being done currently by the above mentioned two mechanisms ??

-- Diwakar

---

From: Jeffrey Mogul <[email-masked]>
Date: Tue, 06 May 2003 14:12:32 -0700

Diwakar Shetty <[email-masked]> writes:

    Dont we have "Modified-Since" and "Etags" to do this job already ?
    What will "hash" do extra which is not being done currently by the
    above mentioned two mechanisms ??

The existing mechanisms don't solve the problem of "aliasing" where
two different URLs point to the same content, and a related problem
where a given URL yields content in a sequence like

	A
	B
	C
	A

These two effects can cause redundant content transfer (that is,
a hypothetical perfect cache could avoid these transfers).
We found that these two effects together, in one large trace,
caused about 36% of the bytes transferred to be "redundant" in
this sense.  See the WWW 2002 paper I've already cited.

-Jeff

---

From: David Morris <[email-masked]>
Date: Tue, 6 May 2003 15:24:07 -0700 (PDT)

And how much would compression reduce the data transfered? Not mutually
exclusive, I know, but a solution to the issue you describe would be quite
complex to implement, compression would be pretty straight forward and I
suspect reduce bytes transfered by more than the 36% redundancy.

Dave Morris

---

From: Jeffrey Mogul <[email-masked]>
Date: Tue, 06 May 2003 17:07:16 -0700

Dave Morris writes:

    And how much would compression reduce the data transfered? Not
    mutually exclusive, I know, but a solution to the issue you
    describe would be quite complex to implement, compression would be
    pretty straight forward and I suspect reduce bytes transfered by
    more than the 36% redundancy.

Good question.  We couldn't answer it directly, because our
traces don't include the actual contents, and so we couldn't
compress the data to see what would happen with compression.

However, indirect evidence suggests that adding HTTP-level
compression wouldn't change the results very much, because
most of the bytes (and especially most of the redundant bytes)
were in image formats that are already compressed.

Your trace may vary ...

-Jeff