Subject: short-time client side caching and clock skew

From: "Jorrit Jongma" <[email-masked]>
Date: Thu, 27 May 2010 13:29:46 +0200

After running into some caching issues I started reading sections 13.2.3 and
13.2.4 of RFC 2616, namely "Age Calculations" and "Expiration Calculations".

I noticed how clock-skew appears to be a major issue in these calculations.
It is not 1999 anymore, and many sites now heavy monster loads including
lots of tiny AJAX calls. In many instances, these could be cached for small
amounts of time - consider for example 1 minute or 5 minutes - but generally
not for say half an hour or more. If this could be properly achieved, this
could improve site response speed and reduce server loads.

However, as I interpret it, due to how current_age is calculated in 13.2.3,
short-period client-side caching is essentially impossible unless both
server and client are NTP-synced. If either the server clock is behind
(and/)or the client clock is in front of NTP time, current_age will
generally exceed freshness lifetime due to the discrepancy. In reverse, if
the server clock is in front (and/)or the client clock is behind NTP time,
the object will be cached way beyond it's intended freshness_lifetime. The
problem does not show so much with longer freshness_lifetimes as the
discrepancy becomes less significant as freshness_lifetime goes up.

Even if a non-local proxy/cache is used that supplies an Age header that
would be correct (excluding transit time), the problem would still appear
for the case where the client clock is fast due to corrected_received_age =
max(now - date_value, age_value).

I'm not sure if it is any way possible to fix or correct this problem (or if
that is even something HTTPbis cares about), though I did have some ideas
that MIGHT be effective, though I am probably overlooking an important
factor:

(1) The Age header, if present, SHOULD override (now - date_value), instead
of taking the max of both of them in corrected_received_age. While this
ignores transit time, I would say it is safe to assume that in most cases
transit time is an order of magnitude smaller than freshness_lifetime (even
if that is only 1 minute) and would thus generally be insignificant. As
servers SHOULD be NTP-synced I would also say that in general the
reliability of the Age header sent by proxies/caches is higher than the
reliability of the client clock.

(2) Instead of using origin date to calculate the current_age, client
receive date could be used. This would result in: current_age = (now -
original_local_receive_date + age_header), and removes the dependency on a
remote clock and thus eliminates the discrepency. Again this ignores transit
time. (in case of Expires headers the freshness_lifetime would of course
still be calculated using the difference between the Date and Expires
headers)

Both would only apply to HTTP/1.1 responses, as HTTP/1.0 servers do not send
the Age header. I'm not sure if it is possible to have an HTTP/1.1 origin
server, a HTTP/1.0 proxy, then a HTTP/1.1 proxy, and then receive a HTTP/1.1
response, if so that could possibly pose a problem in this scenario.

Looking at ticket #29, this does nothing to alleviate the problem.

Just thinking out loud here, and not quite sure why this calculation was
made in this way in the first place - any comments?

Regards,
Jorrit

---

From: Henrik Nordström <[email-masked]>
Date: Fri, 28 May 2010 00:53:41 +0200

tor 2010-05-27 klockan 13:29 +0200 skrev Jorrit Jongma:

> server and client are NTP-synced.

Indeed. And this is due to legacy reasons in dealing with old HTTP/1.0
proxies not adding Age to their responses.

If you know that the response path is doing Age calculations correctly
then the parts relating to Date can be disregarded, using just

  corrected_initial_age =  age_value + (now - request_time)

and still stay within the spirit of the calculation. The authorative
calculations is the split out calculations, not the attempt in merging
these in one single formula. That single formula is very much an
approximation and do not reflect the text very well.

I think we have updated these calculations in HTTPbis. If not then that
should clearly be done. At a very minimum if needs to account for when
date_value > now (now - date_value should then result in 0, not a
negative number)

> In reverse, if
> the object will be cached way beyond it's intended freshness_lifetime.

How? I fail to see this.

I know there is broken implementations giving this result by just using
the simplified calculation, but not how you end up with this result if
actually following what is said, including the criterias that is not
reflected in the simplified calculation.

> (1) The Age header, if present, SHOULD override (now - date_value)

And you are allowed to by RFC2616 if you know that Age is correctly
sent. The combined calculation of age methods 1 & 2 is just a
suggestion.

> While this

You still need to add transit time to the age. Continue reading until
you have read the whole section.

> receive date could be used.

No, but the time the client sent the request SHOULD be used.

> response, if so that could possibly pose a problem in this scenario.

The response SHOULD be HTTP/1.1 in such case, and is why it's generally
recommended to use the conservative approach of using max(now - date,
age).

But if the proxies are HTTP/1.1 compliant then Via can be used to detect
if there is any HTTP/1.0 paths, enabling you to select between the two
methods of calculating the age.

Regards
Henrik

---

From: "Jorrit Jongma" <[email-masked]>
Date: Mon, 31 May 2010 15:04:21 +0200

Henrik,

Thank you for your insights and corrections.

If I understand you correctly, the following pseudo-code would cover all the cases, taking -09 as base, and completely bypassing date_value if possible:

if (response_version_1_1 && !response_via_1_0) {
	corrected_initial_age = age_value + (response_time - request_time);
} else {
	apparent_age = max(0, response_time - date_value);
	corrected_received_age = max(apparent_age, age_value);
	response_delay = response_time - request_time;
	corrected_initial_age = corrected_received_age + response_delay;
}
resident_time = now - response_time;
current_age = correct_initial_age + resident_time;

I have tested current operation in a few browsers, and the results are not encouraging; if the above is judged correct it is my intent to submit bug tickets to some of these browser manufacturers and see if they will update.

Regards,
Jorrit