Subject: Mandating a default client timeout for HTTP 301

From: McGuireV10 <[email-masked]>
Date: Wed, 29 Nov 2017 15:03:00 +0000

I would like to suggest a mandatory default timeout for clients which cache HTTP 301 (Permanent Redirect) responses that are received without a corresponding expiration. It appears that all current major browsers take "permanent" very seriously, meaning the server potentially loses control of that URI forever. I don't live-and-breathe the HTTP spec like many of the folks on this list, but that appears to be a valid interpretation of the spec.

However, "permanent" means a simple mistake can become a very serious and irreversible problem for server owners. Sure, it's easy to say server owners should be careful, but in the real world mistakes happen, management makes decisions and changes their mind later, and so on. A quick search of stackoverflow and the web in general shows that this problem is not uncommon.

If you own the redirect target, you can issue another redirect back to the original resource, but if you don't control the clients, the safe solution is to never remove that second redirect -- which also means the second resource is permanently unavailable for other uses. If you do not own the redirect target (and the other owner can't or won't cooperate), you're simply out of luck.

A default timeout (specified as a "must" requirement) means clients would periodically re-validate cached 301s. Since 301 already requires a TTL, implementation would be trivial. (I only realized as I'm writing this that I haven't actually checked the spec to confirm expiration is supported, but the public at large seems to believe this to be the case and it seems to work with all major browsers.)

Off the top of my head, I would think the possible responses would fall into three categories: reset the timer and continue using the cached redirect (a 404 is the obvious example), update the cached redirect (for example, a different 301 response is received), or remove the redirect from the cache (the response succeeds in some "normal" fashion).

Best,
JM

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 30 Nov 2017 11:35:02 +1100

Hi,

> On 30 Nov 2017, at 2:03 am, McGuireV10 <[email-masked]> wrote:
> 

It's more than a valid interpretation -- I think it's the primary intent.

AIUI this distinction was made for the case mentioned in the RFC; bookmarks, link editors, etc. needed to know whether they should update the stored link, and 301 gives them a signal that it's OK to do so.

> Off the top of my head, I would think the possible responses would fall into three categories: reset the timer and continue using the cached redirect (a 404 is the obvious example), update the cach…

The semantics of the status code -- for better or worse -- aren't tied to caching, so this would be a pretty big change. We're generally shy of doing that unless there's a significant interoperability or security issue, and this sounds like it's more of a developer usability issue -- i.e., people should have used a temporary, rather than a permanent redirect.

All of that said, I think we *could* put a warning in the spec to the effect that "permanent means permanent," advising to use a temporary redirect unless it's really permanent.

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Barry Tunetheweb <[email-masked]>
Date: Thu, 30 Nov 2017 09:59:20 +0000

>
> First time poster, long time lurker, so be nice if I’m wrong about this :-)

The semantics of the status code -- for better or worse -- aren't tied to
> caching, so this would be a pretty big change. We're generally shy of doing
> that unless there's a significant interoperability or security issue, and
> this sounds like it's more of a developer usability issue -- i.e., people
> should have used a temporary, rather than a permanent redirect.
>

I would argue that it IS tied to caching - by most browsers making it
permanently cached as well.

While I don’t necessarily agree with the OPs suggested solution (personally
I think that perhaps it is up to the individual clients to decide on a max
setting), I do think there is a problem with permanently removing the
ability to use a URL because of a 301 and this has caused me problems in
the past.

Mistakingly using a 301 instead of a 302 is one use case and yes, arguable
this can be chalked up to user error, but I dislike there being no
forgiveness for user error in any system. There may be short term pain for
a mistake, but there should ALWAYS be a longer term solution to roll it
back IMHO. Forcing  a move to another URL (i.e. not being able to roll it
back) is therefore not a solution.

However, there are other use cases, where I have been more caught out:

Scenario 1: When trying to reuse a URL that was used in the past. For
example you have a URL called /example.html. You retire the page, and will
never be bringing it back, so put in a 301. After a while, everyone has
stopped referring to /example.html. Traffic logs show it’s not used and
maybe the redirect is even removed from the web server config (or are we
suggesting config files should just grow and grow?). Someone then decides
to use /example.html for a completely different purpose - perhaps unawares
that it was used previously. This works fine for most people, except a few
browsers that have the 301 cached and leads to weirdness (“It works for
me!”).

Should you NEVER be able to reuse a URL? Now arguably in this case, some
users may have bookmarked the original URL, be dependent on the redirect,
and therefore be surprised to see the new page if you publish something
else there, but to me that’s part and parcel of an evolving web and can be
controlled somewhat by checking web server logs before removing the
redirect.

Scenario 2: When the page it redirects to moves.

Example I have the following:

example1.html -> example2.html

Later I move example2.html so put another redirect:

example2.hrml -> example3.html

I should be able to update the original 301 to save the two step redirect.

Scenario 3: Domain is moved to a new owner with a completely new site.
After a transition period (ideally well defined) the new owner should not
be tied to the old owners decisions.

Instead of mandating a max time I would prefer if clients obeyed cage
control headers on redirects. During that time it should be treated as a
permanent redirect, and bookmarks and the likes can be updated, but if the
original is attempted again after that time, it should be able to return a
different response (either a resource or a different redirect) and know it
will be read by all clients after a set time. Maybe here there should be a
suggested max time as the OP wants for when no cache control header are
given on the 301 (my understanding is when cache control headers are not
given, it’s not clearly defined whether to cache or not, and for how long,
but I could be wrong about that).

All of that said, I think we *could* put a warning in the spec to the
> effect that "permanent means permanent," advising to use a temporary
> redirect unless it's really permanent.

I would still prefer to be able to specify a cache control header saying
it’s “permanent for the duration of that cache control” (and all the
ramifications that permanent redirects have) but allow this to be changed.
I know the apparent contradiction of a non-permanent permenant redirect
might grate with some people but nothing is really permanent on the web!

301s are heavily used for SEO purposes where, as I say above, the intention
was to make it a permanent redirect at the time (and hence want a strong
signal to search crawlers that 302 doesn’t give) but may want to change
that in future for the scenarios given above or other reasons.

The alternative is to bring in a new HTTP Status code (30X - permanent but
non-cacheable redirect) but honestly don’t see the point. I do not see the
need why 301 needs to be permanently cached - as I say nothing is permanent
on the web.

Thanks,
Barry

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 1 Dec 2017 07:14:42 +1000

On 1 Dec. 2017 03:54, "Barry Tunetheweb" <[email-masked]> wrote:

I wasn't aware that browsers cache the 301. (I mean, maybe if you send the
right combination of cache headers along with the response...) And even if
they do, doesn't "forever" in a browser cache usually mean a couple of
months (assuming no further hits)?

I always read 301 as about updating bookmarks, crawler/search indices, etc.

Cheers
-- 
Matthew Kerwin

---

From: Barry Tunetheweb <[email-masked]>
Date: Thu, 30 Nov 2017 22:52:59 +0000

>> months (assuming no further hits)?
Some of them definitely do, and often they are difficult to reset even if
you do have access to the client machines (which of course you won’t for an
Internet site).

I’m not sure how long “forever” means but a couple of months is a long time
and I personally would prefer this to be much, much smaller, or even zero.

However checking some more I’m not so sure about my post now...

https://tools.ietf.org/html/rfc7231#section-6.4.2 states:

A 301 response is cacheable by default; i.e., unless otherwise
   indicated by the method definition or explicit cache controls (see
   Section 4.2.2 of [RFC7234]
<https://tools.ietf.org/html/rfc7234#section-4.2.2>).

I was sure browsers totally ignored cache control headers from previous
investigations, but above suggests they should not, and double checking
now, it looks like I’m not setting cache control
headers on 301s when I was sure I was. Will dig some more, but it appears
that the problem is as the OP originally stated - the default value of
“forever” when no other cache control header is sent is the issue
(partially because most web servers make it easy to add 301s without cache
control headers, but less easy to add them with cache control headers so
most 301s will NOT have a cache control header).

So I would agree with the OPs request to add a max expiry time set in the
standard for when no other expiry is set. Though to be honest I’d prefer to
remove caching of 301s altogether - if the page still needs redirecting
next time, then sites can continue to send 301 and all that is lost is some
time for a redirect for those who never updated from the original URL.

Agreed that’s their primary usage. I wonder if that’s why some of the
browsers permenatly cache 301s - to effectively update the bookmark without
actually having to update it? Seems a bit fragile though if that is the
reason.

---

From: Martin J. Dürst <[email-masked]>
Date: Fri, 1 Dec 2017 16:29:57 +0900

> I always read 301 as about updating bookmarks, crawler/search indices, etc.

Me too. And my understanding was that in that case, the bookmark would 
just store the new URI, and forget the old one. Otherwise, they would 
have to store two or more URIs for each bookmark.

In that sense, a 301 would definitely be permanent (no way for the 
origin location to undo the redirect), which would be fine because 
permanent is supposed to mean permanent, or not? But it wouldn't be a 
problem for a new attempt to go to the previous URI, either.

Regards,   Martin.