Subject: querystring part of cache key

From: Adrien de Croy <[email-masked]>
Date: Fri, 22 May 2009 14:17:30 +1200

there were discussions recently about the method being part of the cache 
key, since the response for different methods on the same URI could 
presumably be cacheable and different.

What about query string?  If the result to something that has a 
querystring is marked as cachable by the origin server, then is it 
deemed part of the URI for the cache key?  I'd presume yes.

Since URIs can be arbitrarily long, yet database fields aren't good with 
this, I'd presume it's common practise to look up based on some hash 
value.  Is this approach used?  Is there any industry-standard hashing 
method, e.g. MD5 of method+URI(normalised) + querystring ?

TIA

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 22 May 2009 16:19:30 +0100

> deemed part of the URI for the cache key?  I'd presume yes.

Of coures, the query string is just part of the URI.

> method, e.g. MD5 of method+URI(normalised) + querystring ?

I doubt it.  Why would you do that?  I don't think it's normal to use
a URI to select an application and pass the querystring verbatim to a
database, or at least it's not a good idea :-)

-- Jamie

---

From: David Morris <[email-masked]>
Date: Fri, 22 May 2009 09:39:40 -0700 (PDT)

> database, or at least it's not a good idea :-)

Why not? .. this is a caching related question where the URI is part of
the cache key ... since I've not implemented such a cache, I can't speak 
to what I have done, but a hash such as MD5 seems reasonable ... in 
particular if followed by an exact match comparison with a value stored
in a blob, etc.

---

From: Adrian Chadd <[email-masked]>
Date: Sat, 23 May 2009 00:45:38 +0800

.. Squid just caches based on the URL (with relevant rules controlling
whether the URL can be cached or not.)

I wrote some extensions to allow an external plugin to "normalise"
a URL before caching. People use this to cache things like Youtube.

2c,

Adrian

-- 
- Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support -
- $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 22 May 2009 17:48:56 +0100

> whether the URL can be cached or not.)

I think there was a time when you had to avoid '?' in a URL because
caches wouldn't cache anything with it.

-- Jamie

---

From: Adrian Chadd <[email-masked]>
Date: Sat, 23 May 2009 00:52:18 +0800

> caches wouldn't cache anything with it.

Heh. I think squid may still treat ?'s as uncachable for that reason.

In fact, plenty of origins are returning cachable info for non-normalised
query strings and this has an impact on the working size and churn of the
cache.

Adrian

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 22 May 2009 18:21:39 +0100

> particular if followed by an exact match comparison with a value stored

Ah, your question was about how to implement a cache.

There's lots of ways.  Hashing the URI is one, then that could look up
in a big hash table or a file in a directory, or multi-level directory tree.

Or it could look up in a database like DB or TDB.  There are lots of
key-value databases which are happy with arbitrary length key strings,
or which have a fairly big limit and don't pad them.

Some key-value databases hash internally, and some of them use B-trees
or other data structures.  If it's on disk, a B-tree might be good
because it'll preserve locality among similar URIs.

I've used a multi-level multi-key tree structure, in order to handle
Etags properly with different Vary on the same URI.

Using an SQL database sounds like a way to make your cache
unnecessarily slow, and not a good fit for the problem.

Be careful when normalising that you don't convert %xx of any
"sensitive" characters, as it can change the meaning of the URI.
Since any escaped characters could be meaningfully distinct from its
unescaped form to an application, that might mean don't convert any
%xx at all.

-- Jamie

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 23 May 2009 12:00:34 +1200

> Ah, your question was about how to implement a cache.
yep, sorry if that wasn't clear

> or which have a fairly big limit and don't pad them.
we are on a windows platform, and we wish the cache index to be able to 
be shared by multiple proxies.

This really directs us to a client-server SQL DBMS.  Tests so far 
haven't shown any problem with performance (depending on the DBMS).  The 
DB itself then can run on any platform.

Most DBs will allow arbitrary length text fields, but there's a wide 
variation in support for indexing on them or searching in them.  We want 
to allow the customer to choose the DBMS (using ODBC) so we need to 
cater for a wide range, which necessitates a lowest common functionality 
approach (e.g. the MS Jet engine).

We used to not cache anything with a querystring, but there are a lot of 
origin servers that mark responses to such requests as cacheable, and 
more and more sites using query strings to define the resource for 
pretty much all pages in a site via a single base URI.  So not caching 
responses to requests that had a querystring seems increasingly ineffecient.

As for normalising, the example in RFC2616 s3.2.3 on URI comparison 
implies that the URI should be normalised, and that %xx should be 
treated the same as the equivalent character.  Has this been found to be 
problematic in practise?

Also many agents have a max limit on a URI as well as has been discussed 
on this list so maybe another approach is to use a long varchar field 
and let the DB do the hashing, and just limit the URI length, and not 
cache anything with excessively long URIs... seems then a bit arbitrary 
as to where you choose the cutoff for what is deemed an over-size URI.

Thanks

Adrien