Subject: Re: #29: correcting corrected_initial_age

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Mar 2010 15:39:59 +1100

We haven't heard back from Alex, and the other issue I mentioned didn't seem to get enough support to move on. So, I suggest we do the conservative thing:

Current text:
>   age_value     - Age header field-value received with the response
>   date_value    - Date header field-value received with the response
>   request_time  - local time when the cache made the request 
>                  resulting in the stored response
>   response_time - local time when the cache received the response
>   now           - current local time
>   
>   apparent_age = max(0, response_time - date_value);
>   corrected_received_age = max(apparent_age, age_value);
>   response_delay = response_time - request_time;
>   corrected_initial_age = corrected_received_age + response_delay;
>   resident_time = now - response_time;
>   current_age   = corrected_initial_age + resident_time;

Replacement text:
>   age_value     - Age header field-value received with the response;
>                                0 if not available.
>   date_value    - Date header field-value received with the response;
>                                see [ref] for requirements regarding responses
>                                without a date_value.
>   request_time  - local time when the cache made the request 
>                  resulting in the stored response
>   response_time - local time when the cache received the response
>   now           - current local time
>   
>   apparent_age = max(0, response_time - date_value);
>   response_delay = response_time - request_time;
>   corrected_initial_age = max(apparent_age, age_value + response_delay)
>   resident_time = now - response_time;
>   current_age   = corrected_initial_age + resident_time;

Comments?

On 14/10/2009, at 8:31 PM, Mark Nottingham wrote:

> Hi Alex,
> 
> We put this on the HTTPbis issues list a while ago <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/29>, and I've discussed it with a few folks F2F, but AFAICT it hasn't been discussed on-list.
> 
> In a nutshell, I think you're correct that there's a problem here, but your proposal:
> 
>> creation_time = min(date_value, request_time - age_value);
>> current_age = now - creation_time;
> 
> has a few (small-ish) issues.
> 
> 1) The corrected_received_age's subtraction of the date_value from now has the (intended, I assume) effect of accounting for upstream HTTP/1.0 caches that don't append an Age header. Your proposal doesn't do this.
> 
> This is already being diccussed on-list (see recent thread "cache freshness / age calcs"), and may go away anyway. Your input there would be appreciated.
> 
> 2) The behaviour when date_value isn't present isn't specified; we could address this in prose, but it would be awkward.
> 
> This could probably be worked around by either specifying a slightly more complex formula, or specifying that when the Date header isn't present, a completely separate (and presumably much simpler) formula is to be used.
> 
> Cheers,
> 
> 
> 
> On 31/08/2002, at 2:59 AM, Alex Rousskov wrote:
> 
>> 
>> Hi there,
>> 
>> 	We are testing a couple of RFC 2616 MUSTs related to
>> current_age calculation. Many proxies violate a subset of test cases
>> that includes an artificial proxy-to-server delay. Looking at the
>> results, I think that the proxies are doing the "right thing" and the
>> RFC has a problem.
>> 
>> 	I will start with a specific example when current_age formula
>> from the RFC yields a way-too-conservative and unnatural result (100%
>> error). I will then describe the problem and suggest a fix.
>> 
>> 	I understand that a way-too-conservative age does not lead to
>> stale documents being returned. However, if we want proxies to be
>> compliant, we may want to fix/mention the problem in the errata or
>> elsewhere. Otherwise, the more problems like that are left unaddressed
>> (ignored), the more difficult it would be to convince implementors to
>> pay attention to the RFC.
>> 
>> 	Perhaps I got it all wrong, please check!
>> 
>> 
>> A simple example
>> ----------------
>> 
>> Here is a real and simple example that detected the problem with the
>> original current_age formula from "13.2.3 Age Calculations". The
>> absolute values of timestamps below ("0" and "7") have no
>> significance.
>> 
>>  time event
>>  ---- ------------------------------------------------------------
>>   0.0 client request generated
>>   0.0 client request reached the proxy, it is a MISS
>>   0.0 proxy request to origin server is generated
>>   0.0 proxy request reached the origin server
>>   0.0 server response generated with Date correctly set to 0, no Age header
>>   -- a network delay of 7 seconds --
>>   7.0 server response reached the proxy
>>   7.0 proxy cached the response
>>   7.0 proxy forwarded the response
>>   7.0 the response reached the client
>>   7.0 another client request for the same URL generated
>>   7.0 client request reached the proxy, it is a HIT
>>   7.0 proxy must compute Age header value, see math below
>> 
>> Following RFC 2616:
>> 
>>   age_value = 0             (the cached response has no Age header)
>>   date_value = 0            (the cached response has Date set to 0)
>>   request_time = 0          (the proxy generated request at time 0)
>>   response_time = 7         (the proxy received response at time 7)
>>   now = 7                   (the current time is 7)
>> 
>>   apparent_age = max(0, response_time - date_value) = 7
>>   corrected_received_age = max(apparent_age, age_value) = 7
>>   response_delay = response_time - request_time = 7
>>   corrected_initial_age = corrected_received_age + response_delay = 14
>>   resident_time = now - response_time = 0
>>   current_age   = corrected_initial_age + resident_time = 14
>> 
>> The true age is, of course, 7 and not 14. The above formulas just double true
>> current age in the case of a network delay between the proxy and the origin
>> server. The fixed formula (see below for the discussion) does not:
>> 
>> current_age = now - min(date_value, request_time - age_value) =
>>            = 7 - max(0, 0 - 0) = 7
>> 
>> N.B. If the proxy computes Age header for misses and uses that as
>> age_value when serving hits, the formulas yield the same result.
>> 
>> 
>> The Problem
>> -----------
>> 
>> RFC 2616 says:
>> 
>>  Because the request that resulted in the returned Age value must have
>>  been initiated prior to that Age value's generation, we can correct
>>  for delays imposed by the network by recording the time at which the
>>  request was initiated. Then, when an Age value is received, it MUST
>>  be interpreted relative to the time the request was initiated...
>>  So, we compute:
>> 
>>     corrected_initial_age = corrected_received_age
>>                           + (now - request_time)
>> 
>> I suspect the formula does not match the true intent of the RFC
>> authors. I believe that corrected_initial_age formula counts
>> server-to-client delays twice. It does that because the
>> corrected_received_age component already accounts for one
>> server-to-client delay. Here is an annotated definition from the RFC:
>> 
>>  corrected_received_age = max(
>>    now - date_value, # trust the clock (includes server-to-client delay!)
>>    age_value)        # all-HTTP/1.1 paths (no server-to-client delay)
>> 
>> I think it is possible to fix the corrected_initial_age formula to
>> match the intent (note this is the *initial* not *received* age):
>> 
>>  corrected_initial_age = max(
>>    now - date_value,                # trust the clock (includes delays)
>>    age_value + now - request_time)  # trust Age, add network delays
>> 
>> There is no need for corrected_received_age.
>> 
>> 
>> Moreover, it looks ALL the formulas computing current_age go away with
>> the above new corrected_initial_age definition as long as "now" is
>> still defined as "the current time" (i.e., the time when current_age
>> is calculated):
>> 
>>  current_age = corrected_initial_age
>> 
>> So, we end up with a single formula for all cases and all times:
>> 
>> current_age = max(now - date_value, age_value + now - request_time) =
>>            = now - min(date_value, request_time - age_value)
>> 
>> It even has a clear physical meaning -- the min() part is the conservative
>> estimate of object creation time. We could rewrite for clarity:
>> 
>> creation_time = min(date_value, request_time - age_value);
>> current_age = now - creation_time;
>> 
>> 
>> Am I missing something important here? If I am right, and the current
>> formulas count server-to-client delays twice, is it worth mentioning
>> in the errata or elsewhere as a bug? Or should we insist that
>> implementations use current_age calculation from the RFC anyway?
>> 
>> Thank you,
>> 
>> Alex.
>> 
>> -- 
>>                           | HTTP performance - Web Polygraph benchmark
>> www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
>>                           | all of the above - PolyBox appliance
>> 
> 
> 
> --
> Mark Nottingham     http://www.mnot.net/
> 
> 

--
Mark Nottingham     http://www.mnot.net/

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 09 Mar 2010 00:50:36 +1300

I'd suggest removing the word "local" in all cases referring to time.

Since all date/times are in UTC (absolute time), the word local may 
potentially confuse implementors into using a local time (non UTC time) 
which breaks things like apparent_age.

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Julian Reschke <[email-masked]>
Date: Mon, 08 Mar 2010 16:15:14 +0100

>>    current_age   = corrected_initial_age + resident_time;

I just spent some time staring at this, and I'd *really* feel better if 
we had a test case, that is:

- values for age_value, date_value, request_time, response_time, now
- result per RFC2616
- expected result

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 08 Mar 2010 20:21:27 +0100

>>    current_age   = corrected_initial_age + resident_time;

Looks good to me. I tried both the old and the new formula with a set of 
input values, and came up with:

Request_time Age_value Date_value Response_time 
RFC2616_corrected_initial_age proposed_corrected_initial_age
1000 2 0995 1020 45 25
1000 2 1000 1020 40 22
1000 2 1005 1020 35 22
1000 2 1010 1020 30 22
1000 2 1015 1020 25 22
1000 2 1020 1020 22 22
1000 2 1025 1020 22 22
1000 0 1001 1002 3 2
1000 1 1001 1002 3 3
1000 2 1001 1002 4 4
1000 3 1001 1002 5 5
1000 4 1001 1002 6 6
1000 5 1001 1002 7 7

...so the new formula (last column) indeed is less conservative.

I agree with Adrien's comment that we should also avoid the term "local 
time".

Best regards, Julian

---

From: David Morris <[email-masked]>
Date: Mon, 8 Mar 2010 11:56:39 -0800 (PST)

On Tue, 9 Mar 2010, Adrien de Croy wrote:

> potentially confuse implementors into using a local time (non UTC time) which
> breaks things like apparent_age.

Removal doesn't seem like the right solution ... we need to clarify that
we me the time as known to the local system.

Perhaps replace: "local time" with "UTC time known to the local system"

Or some other phrasing which removes the ambiguity.

Dave Morris

> 
> Regards
> 
> Adrien
> 
> Mark Nottingham wrote:
> > We haven't heard back from Alex, and the other issue I mentioned didn't seem
> > to get enough support to move on. So, I suggest we do the conservative
> > thing:
> > 
> > Current text:
> >   
> > > We put this on the HTTPbis issues list a while ago
> > > <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/29>, and I've discussed
> > > it with a few folks F2F, but AFAICT it hasn't been discussed on-list.
> > > 
> > > 1) The corrected_received_age's subtraction of the date_value from now has
> > > the (intended, I assume) effect of accounting for upstream HTTP/1.0 caches
> > > that don't append an Age header. Your proposal doesn't do this.
> > > 
> > > This is already being diccussed on-list (see recent thread "cache
> > > freshness / age calcs"), and may go away anyway. Your input there would be
> > > appreciated.
> > > 
> > > 2) The behaviour when date_value isn't present isn't specified; we could
> > > address this in prose, but it would be awkward.
> > > 
> > > This could probably be worked around by either specifying a slightly more
> > > complex formula, or specifying that when the Date header isn't present, a
> > > completely separate (and presumably much simpler) formula is to be used.
> > > 
> > > Cheers,
> > > 
> > > 
> > >

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 9 Mar 2010 20:22:43 +1100

Agreed. Julian, since you did the patch, could you update?

Cheers,

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Mar 2010 10:30:51 +0100

On 09.03.2010 10:22, Mark Nottingham wrote:
> ...

Si.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Mar 2010 14:19:35 +0100

> Agreed. Julian, since you did the patch, could you update?

OK - it appears we want to do multiple things here:

1) clarify input fields (absence of Age header, "local time"), and

2) change the actual calculation (issue 29).

Thus I've decided to do this in two steps. For 1) I just committed 
changes <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/778> abd 
<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/778>, with the 
resulting text being:

-- snip --
2.3.2.  Calculating Age

    HTTP/1.1 uses the Age response-header to convey the estimated age of
    the response message when obtained from a cache.  The Age field value
    is the cache's estimate of the amount of time since the response was
    generated or validated by the origin server.  In essence, the Age
    value is the sum of the time that the response has been resident in
    each of the caches along the path from the origin server, plus the
    amount of time it has been in transit along network paths.

    The following data is used for the age calculation:

    age_value

       The term "age_value" denotes the value of the Age header, in a
       form appropriate for arithmetic operation; or 0, if not available.

    date_value

       HTTP/1.1 requires origin servers to send a Date header, if
       possible, with every response, giving the time at which the
       response was generated (see Section 9.3 of [Part1]).  The term
       "date_value" denotes the value of the Date header, in a form
       appropriate for arithmetic operations.  See Section 9.3.1 of
       [Part1] for requirements regarding responses without a Date
       response header.[[TODO-checkref: correct reference? --jre]]

    now

       The term "now" means "the current value of the clock at the host
       performing the calculation".  Hosts that use HTTP, but especially
       hosts running origin servers and caches, SHOULD use NTP
       ([RFC1305]) or some similar protocol to synchronize their clocks
       to a globally accurate time standard.

    request_time

       The current value of the clock at the host at the time the request
       resulting in the stored response was made.

    response_time

       The current value of the clock at the host at the time the
       response was received.

    A response's age can be calculated in two entirely independent ways:

    1.  now minus date_value, if the local clock is reasonably well
        synchronized to the origin server's clock.  If the result is
        negative, the result is replaced by zero.

    2.  age_value, if all of the caches along the response path implement
        HTTP/1.1.

    These are combined as

      corrected_received_age = max(now - date_value, age_value)

    When an Age value is received, it MUST be interpreted relative to the
    time the request was initiated, not the time that the response was
    received.

    where "request_time" is the time (according to the local clock) when
    the request that elicited this response was sent.

    The current_age of a stored response can then be calculated by adding
    the amount of time (in seconds) since the stored response was last
    validated by the origin server to the corrected_initial_age.

    With these definitions, the current_age can be calculated as per:

-- snip --

Before I go on with the actual change for issue 29, I'd like to collect 
some feedback on the rearranged text, defining the data for the age 
calculation.

Also - Mark - is...:

"See Section 9.3.1 of [Part1] for requirements regarding responses 
without a Date response header."

this the reference you had in mind for absent Date headers?

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 10 Mar 2010 09:20:16 +1100

On 10/03/2010, at 12:19 AM, Julian Reschke wrote:

[...]

> Before I go on with the actual change for issue 29, I'd like to collect some feedback on the rearranged text, defining the data for the age calculation.

I think so.

> Also - Mark - is...:
> 
> "See Section 9.3.1 of [Part1] for requirements regarding responses without a Date response header."
> 

No; it's just section 9.3; in particular this text:

>    A received message that does not have a Date header field MUST be
>    assigned one by the recipient if the message will be cached by that
>    recipient or gatewayed via a protocol which requires a Date.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 10 Mar 2010 09:50:38 +0100

>> this the reference you had in mind for absent Date headers?

Very good, that simplifies the text further. I also noted that 
request_time was explained twice, which gets us to 
(<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/784>):

-- snip --
2.3.2.  Calculating Age

    age_value

       The term "age_value" denotes the value of the Age header
       (Section 3.1), in a form appropriate for arithmetic operation; or
       0, if not available.

    date_value

       HTTP/1.1 requires origin servers to send a Date header, if
       possible, with every response, giving the time at which the
       response was generated.  The term "date_value" denotes the value
       of the Date header, in a form appropriate for arithmetic
       operations.  See Section 9.3 of [Part1] for the definition of the
       Date header, and for requirements regarding responses without a
       Date response header.

    now

    request_time

    response_time

    These are combined as

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 10 Mar 2010 16:41:10 +0100

In the meantime I realized that the current text is totally confusing 
because it first uses "now" as "time the response is received", and then 
later as "time the age is calculated". This is also reflected in pseudo 
code statements that are repeated, and in which "now" is replaced by 
"response_time".

I just fixed this by using "now", "request_time" and "response_time" 
consistently, which also allowed getting rid of duplicated pseudo-code. 
I think this improves readability a lot. See 
<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/786>.

New text:

-- snip --
2.3.2.  Calculating Age

    age_value

    date_value

    now

    request_time

    response_time

    1.  the "apparent_age": response_time minus date_value, if the local
        clock is reasonably well synchronized to the origin server's
        clock.  If the result is negative, the result is replaced by
        zero.

    These are combined as

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 11 Mar 2010 11:16:32 +1100

Looks good. The algorithm changes still needs to be applied, however.

On 11/03/2010, at 2:41 AM, Julian Reschke wrote:

> In the meantime I realized that the current text is totally confusing because it first uses "now" as "time the response is received", and then later as "time the age is calculated". This is also reflected in pseudo code statements that are repeated, and in which "now" is replaced by "response_time".
> 
> I just fixed this by using "now", "request_time" and "response_time" consistently, which also allowed getting rid of duplicated pseudo-code. I think this improves readability a lot. See <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/786>.
> 
> New text:
> 
> -- snip --
> 2.3.2.  Calculating Age
>

---

From: Julian Reschke <[email-masked]>
Date: Thu, 11 Mar 2010 20:29:07 +0100

> Looks good. The algorithm changes still needs to be applied, however.

Here we go:

-- snip --
2.3.2.  Calculating Age

    age_value

    date_value

    now

    request_time

    response_time

    2.  the "corrected_age_value", if all of the caches along the
        response path implement HTTP/1.1; note this value MUST be
        interpreted relative to the time the request was initiated, not
        the time that the response was received.

      response_delay = response_time - request_time;
      corrected_age_value = age_value + response_delay;

    These are combined as

      corrected_initial_age = max(apparent_age, corrected_age_value);

-- snip --

(see 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/29/i29.diff>)

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 17 Mar 2010 20:57:52 +1100

Looks good to me.

On 12/03/2010, at 6:29 AM, Julian Reschke wrote:

> (see <http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/29/i29.diff>)
> 
> Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 17 Mar 2010 11:13:56 +0100

> Looks good to me.

OK, applied with <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/792>.

Question: is this a Change from RFC2616 we need to mention? I'm not 
sure, as I don't think there is a normative requirement to use excatly 
this method for Age Calculation.

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 19 Mar 2010 10:55:55 +1100

I think it'd be worthwhile to point it out. 

On 17/03/2010, at 9:13 PM, Julian Reschke wrote:

> Question: is this a Change from RFC2616 we need to mention? I'm not sure, as I don't think there is a normative requirement to use excatly this method for Age Calculation.
> 
> Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 22 Mar 2010 18:53:10 +0100

> I think it'd be worthwhile to point it out.

Done with <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/795>.

Best regards, Julian