Subject: A proposal for Shared Dictionary Compression over HTTP

From: "Wei-Hsin Lee" <[email-masked]>
Date: Mon, 8 Sep 2008 13:59:47 -0700

Hi,

Over the last few weeks we've been experimenting with a way to get better
compression for HTTP streams using a dictionary-based compression scheme,
where a user agent obtains a site-specific dictionary that then allows pages
on the site that have many common elements to be transmitted much more
quickly.

We have a paper that we wrote to describe this idea, which I have put online
here:
http://groups.google.com/group/SDCH

The specifics in the paper are just one possibility, we're not attached to
the exact model described above. So far our experiments have been quite
encouraging, so we'd love to work with other browser and server vendors to
flesh this out. Is this something that anyone would be interested in? Your
advices are very welcomed.

Thank you,

SDCH team, Google Inc.

---

From: Ian Hickson <[email-masked]>
Date: Mon, 8 Sep 2008 23:06:54 +0000 (UTC)

On Mon, 8 Sep 2008, Wei-Hsin Lee wrote:
> 
> We have a paper that we wrote to describe this idea, which I have put 
> online here: http://groups.google.com/group/SDCH

For those of you not used to Google Groups, the paper is here:

   http://sdch.googlegroups.com/web/Shared_Dictionary_Compression_over_HTTP.pdf

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 8 Sep 2008 19:55:58 -0500

Ian Hickson wrote:
> > We have a paper that we wrote to describe this idea, which 
> > I have put online here: http://groups.google.com/group/SDCH
> 

It seems to me that AJAX can be used to solve this problem in a simpler
manner. Take Gmail for example--it downloads the whole UI once and then uses
AJAX to get the state-specific data. The example from the PPT showed a 40%
reduction in the number of bytes transmitted when using SDCH (beyond what
GZIP provided) for google SERPs. I bet you could do about that well just by
AJAXifying the SERPs (making them more like GMail) + using regular HTTP
cache controls + using a compact, application-specific data format for the
dynamic parts of the page + GZIP. Maybe Google's AJAX Search API already
does that? In fact, you might not even need AJAX for this; maybe IFRAMEs are
enough.

I also noticed that this proposal makes the request and response HTTP
headers larger in an effort to make entity bodies smaller. It seems over
time there is an trend of increasingly large HTTP headers as applications
stuff more and more metadata into them, where it is not all that unusual for
a GET request to require more than one packet now, especially when longish
URI-encoded IRIs are used in the message header. Firefox cut down on the
request headers it sends [2] specifically to increase the chances that GET
requests are small enough to fit in one packet. Since HTTP headers are
naturally *highly* repetitive (especially for resources from the same
server), a mechanism that could compress them would be ideal. Perhaps this
could be recast as transport-level compression so that it could be deployed
as a TLS/IPV6/IPSEC compression scheme.

Regards,
Brian

[1] http://www.whatwg.org/specs/web-apps/current-work/#offline
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=309438
    https://bugzilla.mozilla.org/show_bug.cgi?id=125682

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 9 Sep 2008 05:23:39 +0100

Brian Smith wrote:
> It seems to me that AJAX can be used to solve this problem

I agree,

> in a simpler manner.

But not necessarily with that bit!

I think you're comparing adding some protocol at the HTTP level
(e.g. something like an Apache module or a HTTP proxy, bit like
compression), with completely _reengineering_ a web site, which might
be a vast undertaking.

(That said I am vagely working on a HTTP proxy which accelerates
existing web sites by AJAXifying their updates transparently in
precisely this way, so that sites where much of the page stays the
same as you click along go that bit faster, so it's not out of the
question.)

-- Jamie

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Sep 2008 09:44:57 +0200

Hi,

quoting from page 2:

> Existing techniques compress each response in isolation, and so cannot take advantage of
> cross-payload redundancy. For example, retrieving a set of HTML pages with the same
> header, footer, inlined JavaScript and CSS requires the retransmission of the same data
> multiple times. This paper proposes a compression technique that leverages this crosspayload
> redundancy.

That is true, but isn't a *much* simpler approach not to inline these
data chunks? If they are served as separate resources, the can be
individually cached and need not to be retrieved multiple times...

Another concern is that the way it's currently specified, HTTP responses 
lose the property of being completely self-contained; one way to fix 
that would be to always return the set of applied dictionaries.

Finally, there's a concern with the IPR status of vcdiff, see 
<https://datatracker.ietf.org/ipr/search/?option=rfc_search&rfc_search=3284> 
and 
<http://lists.w3.org/Archives/Public/ietf-http-wg/2004AprJun/0086.html>, 
in which Roy said:

> Servers SHOULD NOT support the VCDIFF format until its IP restrictions
> are clarified and made available royalty-free for all uses of HTTP,
> at a minimum, and not just use within HTTP/1.1 as defined in 2001.

BR, Julian

---

From: Yves Lafon <[email-masked]>
Date: Wed, 10 Sep 2008 04:17:18 -0400 (EDT)

> on the site that have many common elements to be transmitted much more

One question, why using Accept-Encoding/Content-Encoding instead of 
TE/Transfer-Encoding ?
Cheers,

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: "Jon Butler" <[email-masked]>
Date: Wed, 10 Sep 2008 12:31:59 -0400

Yves--

You raise an interesting point.  SDCH was designed to be applied to
the message body before gzip (since cross-payload redundancy is much
harder to detect after gzipping the payloads).  One of the differences
between the two sets of headers is that transfer encodings must be
applied after and removed before content encodings, since transfer
encodings are properties of the message and content encodings are a
property of the entity inside the message.  So, we have a choice:
either we indicate both SDCH and gzip in the Content-Encodings, or
both in the Transfer-Encoding header.  Since the prior art for gzip is
to indicate it in the Content-Encoding header (a holdover from the
HTTP/1.0 standard as I understand), we proposed putting sdch there as
well.

>From my reading of the standard, it would be more in keeping with the
HTTP/1.1 standard to put both encodings (gzip and sdch) in the
TE/Transfer-Encoding headers, but it is not clear that it would be
more practical.

We'd be happy to hear others' opinions on this.

Jonathan

---

From: Yves Lafon <[email-masked]>
Date: Wed, 10 Sep 2008 12:59:55 -0400 (EDT)

> HTTP/1.0 standard as I understand), we proposed putting sdch there as

Hum, there are no reasons to keep gzip only at the Content-Encoding level. 
In fact, at least one browser uses it. It can have a huge impact in front 
of a proxy handling TE: gzip, especially for slow networks (like mobile).

The fact that gzip is used in Content-Encoding rather than 
Transfer-Encoding is mostly due to implementations, and implementation may 
evolve, especially when new products enter the market, so it's better if 
the specification allows both (and even better if implementation allows 
both).

         ~~Yves

---

From: "Jon Butler" <[email-masked]>
Date: Wed, 10 Sep 2008 13:17:37 -0400

An interesting difference between content and transfer codings is that
transfer codings are allowed to have attributes.  Were sdch to use the
Transfer-Encoding header, this might allow more flexibility in the
actual compression scheme used, e.g., Transfer-Encoding: sdch;c=vcdiff
vs Transfer-Encoding: sdch;c=gzip.  The client would indicate what
compression schemes were allowed in the TE header, e.g. TE:
sdch;c="vcdiff,gzip".  In that scenario, I expect we would need to
specify some default compression scheme to be used if none is
specified.

Jonathan

---

From: "Jon Butler" <[email-masked]>
Date: Wed, 10 Sep 2008 13:31:37 -0400

I should have perhaps used zlib or some other compression format
rather than gzip in my example below.  My intent was to address Mark's
point that a variety of compressors could underly a dictionary-based
protocol like SDCH.

Jonathan

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 12 Sep 2008 13:46:41 +1000

Indeed; e.g.,
   http://www.mnot.net/javascript/hinclude/
I use this extensively on my own site, and AFAICT several others are  
as well.

WRT doing it at the transport level -
   http://www.cs.washington.edu/homes/djw/papers/spring-sigcomm00.pdf

Schemes like this are already implemented in a number of "WAN  
optimisation" devices (e.g., Riverbed, Silver Peak, AcceleNet).

Cheers,

--
Mark Nottingham       [email-masked]