Subject: ID for Immutable

From: Patrick McManus <[email-masked]>
Date: Wed, 26 Oct 2016 17:02:35 -0400

[as individual]

FYI

A new version of I-D, draft-mcmanus-immutable-00.txt
has been successfully submitted by Patrick McManus and posted to the
IETF repository.

Name:           draft-mcmanus-immutable
Revision:       00
Title:          HTTP Immutable Responses
Document date:  2016-10-26
Group:          Individual Submission
Pages:          4
URL:            https://www.ietf.org/internet-drafts/draft-mcmanus-
immutable-00.txt
Status:         https://datatracker.ietf.org/doc/draft-mcmanus-immutable/
Htmlized:       https://tools.ietf.org/html/draft-mcmanus-immutable-00

Abstract:
   The immutable HTTP response Cache-Control extension allows servers to
   identify resources that will not be updated during their freshness
   lifetime.  This assures that a client never needs to revalidate a
   cached fresh resource to be certain it has not been modified.

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 27 Oct 2016 12:36:15 +1300

>    cached fresh resource to be certain it has not been modified.

This control seems like it will also be useful for proxy caches to
prevent relaying the same revalidations from older clients that don't
support the control.

However the draft does not mention any proxy handling.

* does it override must-revalidate etc on the stored response?
 - what about proxy-revalidate?

* does it override a client request max-age=0 and/or request no-cache?
 - the stated intention implies that it does.

* assuming immutable overrides those client reload signals; is the proxy
supposed to deliver a 200, 304 or 4xx to clients sending max-age=0 ?

* how does immutable interact with the 'must not send on re-use' headers?
 - ie. no-cache="Set-Cookie", private="Set-Cookie" and similar cases ?

Amos

---

From: Mike Bishop <[email-masked]>
Date: Wed, 26 Oct 2016 23:50:12 +0000

I'm reading between the lines here, and would like to see your questions answered in the document.  However, I would interpret this as:  The cache should behave as if it *did* revalidate the resource and received a 304.

If the client sends an unconditional request, it should get a 200 and the stored payload.  If it sends a revalidation request, it should get a 304.  Whatever you would normally do when you revalidate your stored resource and find it still good.

-----Original Message-----
From: Amos Jeffries [mailto:[email-masked]] 
Sent: Wednesday, October 26, 2016 4:36 PM
To: [email-masked]
Subject: Re: ID for Immutable

On 27/10/2016 10:02 a.m., Patrick McManus wrote:
> [as individual]
> 
> FYI
> 
> A new version of I-D, draft-mcmanus-immutable-00.txt has been 
> successfully submitted by Patrick McManus and posted to the IETF 
> repository.
> 

This control seems like it will also be useful for proxy caches to prevent relaying the same revalidations from older clients that don't support the control.

* assuming immutable overrides those client reload signals; is the proxy supposed to deliver a 200, 304 or 4xx to clients sending max-age=0 ?

Amos

---

From: Pete Wildsmith <[email-masked]>
Date: Thu, 27 Oct 2016 00:57:06 +0100

Patrick,

the relevant section of RFC7234 suggests that conditional requests before the end of the freshness lifetime are made at the discretion of the cache.

> When a response is "fresh" in the cache, it can be used to satisfy subsequent requests without contacting the origin server, thereby improving efficiency.
https://tools.ietf.org/html/rfc7234#section-4.2

My understanding of your extension is that for responses with the extension, the implicit MAY in the quoted sentence would become a SHOULD or MUST.

This seems to conflict with user-agents’ reload behaviour. If all resources that are not expected to change during the lifetime of the resource begin using this extension, will it not become necessary for ‘reload’ to invoke a process which ignores the extension for the same reason that ‘max-age=0’ is now sent? 

Pete

> On 26 Oct 2016, at 22:02, Patrick McManus <[email-masked]> wrote:
> 
> [as individual]
> 
> FYI
> 
> URL:            https://www.ietf.org/internet-drafts/draft-mcmanus-immutable-00.txt

---

From: Ben Maurer <[email-masked]>
Date: Wed, 26 Oct 2016 17:08:40 -0700

This extension is only for use by sites which actually ensure that they do
not change their resources. If a developer wishes to have the reload button
refresh subresources, they should not use immutable.

Keep in mind that all browsers have an escape hatch which lets the user do
a hard refresh. This does not invalidate any resources and requests them
from scratch.

On Wed, Oct 26, 2016 at 4:57 PM, Pete Wildsmith <[email-masked]>
wrote:

> Patrick,
>
> the relevant section of RFC7234 suggests that conditional requests before
> the end of the freshness lifetime are made at the discretion of the cache.
>
> > When a response is "fresh" in the cache, it can be used to satisfy
> subsequent requests without contacting the origin server, thereby improving
> efficiency.
> My understanding of your extension is that for responses with the
> extension, the implicit MAY in the quoted sentence would become a SHOULD or
> MUST.
>
> This seems to conflict with user-agents’ reload behaviour. If all
> resources that are not expected to change during the lifetime of the
> resource begin using this extension, will it not become necessary for
> ‘reload’ to invoke a process which ignores the extension for the same
> reason that ‘max-age=0’ is now sent?
>
> Pete
>

---

From: Patrick McManus <[email-masked]>
Date: Wed, 26 Oct 2016 20:18:31 -0400

> extension, the implicit MAY in the quoted sentence would become a SHOULD or
technically no.. immutable does not place any requirements on the client -
it remains an implicit MAY though there is no point in revalidating a fresh
response served with immutable.

> reason that ‘max-age=0’ is now sent?

I think Ben's got this one - the idea is to disambiguate reloading for
corruption-fixing (e.g. no-cache) from reloading for updates.

---

From: Patrick McManus <[email-masked]>
Date: Wed, 26 Oct 2016 21:28:44 -0400

hey amos - thanks for the thoughtful reply. Let's work through these and
I'll incorporate it into a -01

>  - what about proxy-revalidate?

my understanding is that must-revalidate and proxy-revalidate only apply to
stale responses. immutable only applies to fresh responses - therefore I
don't think it has any impact on *-revalidate handling.

>  - the stated intention implies that it does.
I think a request with no-cache means don't use the cache - so any
directives in the cache including immutable are irrelevant. (the draft
should say this about immutable). I wouldn't think requests bearing
no-cache are using revalidations anywhere along the chain - this is
normally the mechanism afaik used to correct corruption.

max-age=0 is messy, but I think OK, for a proxy cache to apply the
immutable logic to its stored fresh response. The alternative is really
just to revalidate it, but the meaning of immutable is that the
revalidation is going to return 304 during the freshness lifetime.

You would need to make sure not to send a Age > 0 in the response.

> supposed to deliver a 200, 304 or 4xx to clients sending max-age=0 ?
I don't think immutable impacts that much. The proxy has used immutable to
auto-revalidate its stored response.. whether it returns that to the
user-agent as a 200 or 304 I would assume would depend on whether the
user-agent made a conditional request.

>  - ie. no-cache="Set-Cookie", private="Set-Cookie" and similar cases ?

you're going to have to walk me through what you're thinking here because
I'm not immediately seeing the relevance. Immutable allows a client
(including a proxy) that is about to make a conditional request for a fresh
stored response to know that it is going to receive a 304 for that before
doing so and therefore avoid the transaction if it chooses to. Are you
suggesting that the resource varies by cookie or somesuch? I would think
that would be taken into account in the general logic of "do I have a fresh
resource for this request" logic.

wdyt?

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 27 Oct 2016 15:23:17 +1300

On 27/10/2016 2:28 p.m., Patrick McManus wrote:
> On Wed, Oct 26, 2016 at 7:36 PM, Amos Jeffries wrote:
> 
>>

Hmm. Does that mean we need to update the output Date header with the
amount which would otherwise be in Age?

Which makes me wonder if immutable needs to be forbidden on responses
with heuristic freshness. The current wording does not differentiate
between explicit and heuristic freshness.

If we adjust the Date in the absence of a Last-Modified we could be
wrongly extending the freshness period estimated by downstream caches.

> user-agent made a conditional request.

Okay.

> resource for this request" logic.

The main use-case for those controls using lists of header names is to
force a revalidate to get new values per-client for those headers.
Set-Cookie being the one seen most often, though others are possible.

At present we revalidate stored responses with these controls so the 304
from the server can produce any relevant new values for the new client.

Going with the style of immutable meaning as implicit 304 from upstream
we could just drop the headers as if it had produced none. But that is
quite a gotcha situation any applications using immutable will need to
be warned about clearly.

Amos

---

From: Patrick McManus <[email-masked]>
Date: Thu, 27 Oct 2016 14:08:13 -0400

> wrongly extending the freshness period estimated by downstream caches.
I think you've gotten me to change my mind on Age > 0. I wrote that
concerned that a client unaware of immutable would get cranky at seeing
"Age: 30" in response to a request with "max-age: 0". But I don't really
have any evidence of that, nor upon reflection would I really expect anyone
to care as long as the Age was correct.

The alternative, as you point out, is messing with the end-to-end headers..
like Date. And that's definitely worse.

So I'll change my position and say the proxy should just send the cached
response in the way it normally would for a full cache hit. (which could be
a 200 or 304 depending on whether the request was conditional or not). does
that make more sense?

> [...]
as I understand this, your choice to revalidate the stored responses is an
implementation decision (no doubt a wise one!).. the spec allows you to
reuse the fresh content directly with (e.g.) set-cookie redacted but your
experience tells you that the revalidation makes more sense.

I don't think immutable really addresses the concern one way or the other..
it says that the same representation will come back, but the response
headers might indeed be different just as they may be different on a 304 vs
200. If I were you I would keep doing the revalidation, but because you
aren't required to do so in the first place I'm not sure what the spec
would say about that.

and honestly, as noted in the specs, the qualified forms of no-cache and
private are often treated as unqualified forms.. so I wouldn't expect folks
to be declaring immutable content with them as it requires fresh cached
content to work. your trick to store it and revalidate it to pick up new
headers is clever and should just carry on :)

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 28 Oct 2016 21:04:51 +1300

FYI: Squid-4.0.16 beta which is due to be released in the next few days
will contain initial support for this Cache-Control:immutable extension.

This initial support consists of 'immutable' interaction with reload and
refresh requests.

Support for 'immutable' interaction with regular conditional (If-*)
requests is not yet implemented.

Amos Jeffries
Squid Project

---

From: Kari Hurtta <[email-masked]>
Date: Fri, 28 Oct 2016 17:44:05 +0300 (EEST)

> Htmlized:       https://tools.ietf.org/html/draft-mcmanus-immutable-00

|    o  User-Agents often provide two different refresh mechanisms: reload
|       and some form of force-reload.  The latter is used to rectify
|      interrupted loads and other corruption.  These reloads should
|      ignore immutable as well.

How about making it

Cache-Control: max-age=31536000, immutable=<hash-function-value>

So that immutable does not have effect if result of hash-function
does not give same value that what is value of immutable
cache control.

Several immutable cache controls are invalid if they are result
of same hash-function.

If server can't calculate hash-function over resource,
is is really static non-caching resource?

I think that this protects agaist that interrupted loads
becomes immutable.

/ Kari Hurtta

---

From: Patrick McManus <[email-masked]>
Date: Fri, 28 Oct 2016 10:50:16 -0400

the notion of integrity hashes have failed in the past (notably md5)..
separable from immutable imo and would rather not tie that anchor to its
fate.

---

From: Patrick McManus <[email-masked]>
Date: Fri, 28 Oct 2016 11:01:12 -0400

Hey Kari, In my distaste for response header hashes, I did hastily neglect
to mention that one of my implementation corner cases was to ignore
immutable in cases of weakly framed content.. we define weakly framed as
responses terminated by EOF or with Content-Lengths that don't match, or
chunked encodings without a 0 chunk at the end. Experience has shown that
we have to accept these responses for reasons of interop - but discretion
says to ignore immutable on them as they may be indications of corruption.
The ID should mention this - I'll put in -01. Thanks.

worth noting here that the refresh conditional-request path that immutable
impacts has never helped much with the corruption case.. it conditionally
verifies etags or l-m, but generally the corruption is in the message body
- most often truncation. so a 304 reply confirms to the client to keep
using the corrupted content anyhow.. that's there is often a force (or
hard) reload option that goes with no-cache and the draft tries to point
out you really shouldn't use immutable to shortcut that. We use this
force-reload path in lieu of revalidations for weekly framed things.

On Fri, Oct 28, 2016 at 10:50 AM, Patrick McManus <[email-masked]>
wrote:

> On Fri, Oct 28, 2016 at 10:44 AM, Kari Hurtta <
> [email-masked]> wrote:
>

---

From: Kari Hurtta <[email-masked]>
Date: Fri, 28 Oct 2016 18:12:15 +0300 (EEST)

Patrick McManus <[email-masked]>: (Fri Oct 28 17:50:16 2016)
> separable from immutable imo and would rather not tie that anchor to its

That why I wrote "Several immutable cache controls are invalid if they are result
of same hash-function" implying that hash-function is detectable
from result of function (Either by size or something like multihash
https://github.com/multiformats/multihash).

/ Kari Hurtta

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 28 Oct 2016 09:23:54 -0600

> is is really static non-caching resource?

There is sometimes a big difference between the _ability_ to calculate a
hash and the _expense_ of doing so. AFAICT, immutable responses may
still be dynamically generated (or otherwise costly to produce) on the
origin server.

For example, loading, encoding, and hashing a 2-hour video of the 1948
Olympics opening ceremony could be rather expensive, but will produce
the same immutable result [until the server has to inject a fresh set of
advertisements sometime tomorrow].

Alex.

---

From: Kari Hurtta <[email-masked]>
Date: Fri, 28 Oct 2016 18:41:31 +0300 (EEST)

Alex Rousskov <[email-masked]>: (Fri Oct 28 18:23:54 2016)
[ Charset windows-1252 converted... ]
> advertisements sometime tomorrow].

Yes, that is valid concern.
 
> Alex.

And calculating may be even mode expensive for client. Server
needs do it only once. Not for every request. After all supposed usage 
was that resource for same URL is not updated. 

So new advertisements produce new URL for that a 2-hour video of the 
1948 Olympics opening ceremony.

But these cases client anyway will not cache resource so
immutable does not help.

/ Kari Hurtta

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 28 Oct 2016 10:50:55 -0600

On 10/26/2016 03:02 PM, Patrick McManus wrote:

>    o  Clients should ignore immutable for resources that are not part of
>       a secure context [SECURECONTEXTS].

Please think of the children^H^H^H^H proxies. AFAICT, "secure contexts"
are currently a user agent concept. If the above "should" is meant to be
a "SHOULD", then the draft automatically disqualifies most proxies from
legally utilizing this promising "ignore reload" mechanism.

Thank you,

Alex.

---

From: Patrick McManus <[email-masked]>
Date: Fri, 28 Oct 2016 13:21:57 -0400

I do believe the lack of integrity protection in plaintext transfer is an
important security consideration for immutable that suggests they should
not be used together. I'm open to other wording on it for sure.. https://
might be sufficient here.

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 28 Oct 2016 12:12:55 -0600

On 10/28/2016 11:21 AM, Patrick McManus wrote:
> I do believe the lack of integrity protection in plaintext transfer is
> an important security consideration for immutable that suggests they
> should not be used together. I'm open to other wording on it for sure..
> https:// might be sufficient here.

Sounds good. A more general "SHOULD ignore immutable for resources
received without integrity protection" wording would allow proxies to
legally honor the immutable setting in most cases (after breaking a
hundred MUSTs to get to it inside https, naturally).

Thank you,

Alex.

> On Fri, Oct 28, 2016 at 12:50 PM, Alex Rousskov wrote:
>

---

From: Kari Hurtta <[email-masked]>
Date: Sat, 29 Oct 2016 08:30:39 +0300 (EEST)

Kari Hurtta <[email-masked]>: (Fri Oct 28 18:12:15 2016)
> > the notion of integrity hashes have failed in the past (notably md5)..

And anyway even MD5 is good enough for this purpose. Something like
crc32 is not. For avoid making truncations immutable also resource
length in bytes is good enough for this purpose.

Server still known length of resource for a 2-hour video of the 1948
Olympics opening ceremony (using example of Alex Rousskov).

Server does not know resource length in bytes when it generates
response from database dynamically (in these cases also
content-length is not used but chunked transfer-encoding)

( Assuming that you do not put Cache-Control to
  trailer of chunked transfer-encoding. In that
  case also calculating hash of response is not 
  really expensive part here.
)

My favorite test query produces about 24 gigabytes answer 
(if I remember correctly). Yes, some variations
of this may be immutable, but I'm not sure that 
"immutable" cache-control is very usefull here.

( These have nice property that any 32 bit counters
  wrap several times and loading that answer takes
  several hours. 
)

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: Sat, 29 Oct 2016 08:49:29 +0300 (EEST)

Patrick McManus <[email-masked]>: (Fri Oct 28 18:01:12 2016)
> using the corrupted content anyhow.. 

So on these case that heuristic that ignore immutable for
"weakly framed content" does not help either. 304 reply
still confirms to the client to keep using the corrupted content.

My suggestion using immutable=<property from bydy> for ignnoring
immutable was making that more explicit than heuristic. But
that does change  is that ignoring usefull or not.

/ Kari Hurtta

---

From: Kari Hurtta <[email-masked]>
Date: Sat, 29 Oct 2016 09:44:09 +0300 (EEST)

> might be sufficient here.

I suspect that corruption (truncation most likely) happens then
between TLS termination (TLS ofloading, for example load balancer)
and web server.

Yes, reloading with conditional request does not help here either.

(And if load balancer caches this then any reloading probably
 does not help here. Specially if request cache-control
 header is ignored. )

/ Kari Hurtta

---

From: Martin Thomson <[email-masked]>
Date: Sat, 29 Oct 2016 20:24:36 +1100

> that does change  is that ignoring usefull or not.

We aren't really looking for perfect here.  Corruption happens, but is
highly unlikely in the cases that we use this:

Note that Firefox also requires HTTPS (I think).  So random corruption
is astronomically unlikely and well-framed responses (Content-Length
fits, it uses h2) mean that there is enough confidence that the
response was as the server intended.  The key thing for the document
is probably not to specify precisely what conditions that it would
consider this safe to use, but to note that a degree of care might be
needed to avoid having broken responses cached.

Yes, the server might screw up, but the client won't be responsible
for that.  The server needs to create a new resource to fix it.  Or
maybe user will notice a problem and reload.

---

From: Patrick McManus <[email-masked]>
Date: Sat, 29 Oct 2016 08:43:02 -0400

> And anyway even MD5 is good enough for this purpose. Something like

rfc 2616 content-md failed before md5-the-algotrithm aged-out for unrelated
reasons.

no need to yoke that complexity to immutable - the fact that it is
trivially deployable gives the standards based approach some hope (and at
least 4 implementations already of a -00). The alternative to a too-complex
reject standard is a set of heuristics in clients that don't behave
consistently across the Internet.

---

From: Patrick McManus <[email-masked]>
Date: Sat, 29 Oct 2016 08:46:36 -0400

> still confirms to the client to keep using the corrupted content.

this was kind of my point - they are linked. so in cases of weakly framed
we eschew all conditional revalidations and that includes the special case
of immutable. The refresh happens non-conditionally.

I think the right document language is more along the lines of what martin
suggests

---

From: Patrick McManus <[email-masked]>
Date: Sat, 29 Oct 2016 08:47:41 -0400

On Sat, Oct 29, 2016 at 5:24 AM, Martin Thomson <[email-masked]>
wrote:

> The key thing for the document

+1

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 30 Oct 2016 15:55:04 +1300

On 29/10/2016 4:41 a.m., Kari Hurtta wrote:
> Alex Rousskov wrote: (Fri Oct 28 18:23:54 2016)

Verification of the response is orthoginal to being immutable.

The client application can use many independently defined approaches to
check what it gets from the server (or proxy). If it detects corruption
it can send a request containing "Cache-Control: no-cache" which should
cause replacement of the 'corrupt' object in any intermediary caches
along the way.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 30 Oct 2016 16:15:20 +1300

> might be sufficient here.

Since the behaviour which 'immutable' is defining is already optionally
followed by any cache implementation in both insecure and secure
contexts. There is no additional security considerations being added by
this option.

The security considerations around corrupt objects being considered
"fresh" applies to HTTP caching in general, not just immutable objects.

The issues that are being pointed out are real, but apply equally for
secure and for insecure contexts already today without immutable being
involved. I dont see any need to distinguish between contexts in this
draft, nor to prohibit 'insecure' contexts from having immutable objects.

To back this; Squid has for a very long time had a configuration control
which enables admin to mark specific URL as having this same immutable
behaviour. It is one of the more popular ones and used in the 'insecure
context'.

AFAIK, the only issue which has come to light in all that time that can
be attributed to this control is that objects changing faster than what
the admin was aware (or their stated expiry time) had to be manually
refreshed from the client end (if anyone actually cared). Moving the
control point from proxy config to origin server avoids that issue.

Amos