Subject: The Refresh header is still with us

From: Daniel Stenberg <[email-masked]>
Date: Mon, 11 Mar 2019 10:28:07 +0100 (CET)

Hi friends!

The other day someone filed a bug[1] on curl that we don't support redirects 
with the Refresh header. This took me down a rabit hole and I figured I would 
share with you what I learned down there.

As you all know, redirects in HTTP is specified to use 3xx response codes and 
a Location: header to point out the the URL (I'll use the term URL here but 
you know what I mean). This has been the case since RFC 1945 (HTTP/1.0). 
According to an old mail[2] from Roy, Refresh "didn't make it" into that spec.

The little detail that it never made it into the 1.0 spec (nor any later one) 
doesn't seem to have affected the browsers. Still today, browsers keep 
supporting[3] the Refresh header as a sort of Location: replacement even 
though it seems to never have been present in a HTTP spec.

How frequent is the use of the Refresh header? I decided to make an attempt to 
figure out, and for this venture I used the Rapid7 data trove[4]. The method 
that data is collected with may not be the best, but it is still 52+ million 
HTTP responses from different current HTTP servers. (52254873 exactly in my 
data dump)

My counts show:

  - Location is used in 18.49% of the responses
  - Refresh is used in 0.01738% of the responses
  - Location is thus used 1064 times more often than Refresh
  - In 35% of the cases when Refresh is used, Location is *also* used
  - curl thus handles 99.9939% of the redirects in this test

Other random notes:

  - When Refresh is the only redirect header, the response code is usually 200
    (with 404 being the second most)
  - When both headers are used, the response code is almost always 30x
  - When both are used, it is common to redirect to the same target and it is
    also common for the Refresh header value to only contain a number (for
    the number of seconds until "refresh").

Contents:

Redirects can also be done by meta tags and sending the refresh that way, but 
I have not investigated how common as that isn't strictly speaking HTTP so it 
is outside of my research (and interest) here.

Conclusion:

Nah, sorry, I don't have any. Yet another undocumented quirky corner of the 
web I suppose.

[1] = https://github.com/curl/curl/issues/3657
[2] = 
https://lists.w3.org/Archives/Public/ietf-http-wg-old/1996MayAug/0594.html
[3] = http://www.otsukare.info/2015/03/26/refresh-http-header
[4] = https://opendata.rapid7.com/sonar.http/
[5] =

-- 

  / daniel.haxx.se

---

From: "Soni L." <[email-masked]>
Date: Mon, 11 Mar 2019 06:47:56 -0300

I do use Refresh and (deeply-nested) iframes for JS-less webapps. Not 
sure what the alternative is here, if any. But they're not redirects.

On 2019-03-11 6:28 a.m., Daniel Stenberg wrote:
> Hi friends!
>
> The other day someone filed a bug[1] on curl that we don't support 
> redirects with the Refresh header. This took me down a rabit hole and 
> I figured I would share with you what I learned down there.
>
> As you all know, redirects in HTTP is specified to use 3xx response 
> codes and a Location: header to point out the the URL (I'll use the 
> term URL here but you know what I mean). This has been the case since 
> RFC 1945 (HTTP/1.0). According to an old mail[2] from Roy, Refresh 
> "didn't make it" into that spec.
>
> The little detail that it never made it into the 1.0 spec (nor any 
> later one) doesn't seem to have affected the browsers. Still today, 
> browsers keep supporting[3] the Refresh header as a sort of Location: 
> replacement even though it seems to never have been present in a HTTP 
> spec.
>
> How frequent is the use of the Refresh header? I decided to make an 
> attempt to figure out, and for this venture I used the Rapid7 data 
> trove[4]. The method that data is collected with may not be the best, 
> but it is still 52+ million HTTP responses from different current HTTP 
> servers. (52254873 exactly in my data dump)
>
> My counts show:
>
> Redirects can also be done by meta tags and sending the refresh that 
> way, but I have not investigated how common as that isn't strictly 
> speaking HTTP so it is outside of my research (and interest) here.
>
> Conclusion:
>
> Nah, sorry, I don't have any. Yet another undocumented quirky corner 
> of the web I suppose.
>

---

From: Jeffrey Yasskin <[email-masked]>
Date: Mon, 11 Mar 2019 11:50:18 -0700

https://html.spec.whatwg.org/multipage/iana.html tries to register several
things, including this header, with IANA, but most of them haven't made it.
I'm not sure what that pipeline is supposed to look like or where it leaked
in this case. It's probably good to get that registration done at this
point, just to document reality, but with such low usage, someone might be
able to deprecate and remove usage of that header from browsers.

Jeffrey

On Mon, Mar 11, 2019 at 2:32 AM Daniel Stenberg <[email-masked]> wrote:

> redirects
> attempt to

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 12 Mar 2019 09:32:49 +1100

On 12 Mar 2019, at 5:50 am, Jeffrey Yasskin <[email-masked]> wrote:
> 
> https://html.spec.whatwg.org/multipage/iana.html tries to register several things, including this header, with IANA, but most of them haven't made it. I'm not sure what that pipeline is supposed to look like or where it leaked in this case. It's probably good to get that registration done at this point, just to document reality, but with such low usage, someone might be able to deprecate and remove usage of that header from browsers.

Have those actually been submitted to the relevant processes?

A number don't identify the registry being updated.

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 12 Mar 2019 10:19:26 +0100 (CET)

On Mon, 11 Mar 2019, Soni L. wrote:

> I do use Refresh and (deeply-nested) iframes for JS-less webapps. Not sure 
> what the alternative is here, if any. But they're not redirects.

So what exactly do you expect this header to do? Refresh the entire page even 
when just a sub resource response gets it? Is what what it happens? Is that 
working cross-browser?

And why use a HTTP header for this, can't you just use a meta tag since you 
are doing a web page after all? It seems like an odd layer mix to me.

-- 

  / daniel.haxx.se

---

From: "Soni L." <[email-masked]>
Date: Tue, 12 Mar 2019 06:22:59 -0300

On 2019-03-12 6:19 a.m., Daniel Stenberg wrote:
> So what exactly do you expect this header to do? Refresh the entire 
> page even when just a sub resource response gets it? Is what what it 
> happens? Is that working cross-browser?
>
> And why use a HTTP header for this, can't you just use a meta tag 
> since you are doing a web page after all? It seems like an odd layer 
> mix to me.
>
I expect it to refresh a single empty iframe. Once it's no longer empty 
it stops refreshing.

It would be nicer if I could embed HTML snippets into pages, instead of 
full documents, and still get Refresh capabilities. But anyway.

Oh, I also use Refresh for the odd RSS/Atom feed. Not sure if any feed 
readers actually care about rate-limiting themselves following the 
server's lead tho.

---

From: Sebastiaan Deckers <[email-masked]>
Date: Tue, 12 Mar 2019 17:38:44 +0800

On Tue, Mar 12, 2019 at 2:54 AM Jeffrey Yasskin <[email-masked]> wrote:

> [...] but with such low usage, someone might be able to deprecate and
> remove usage of that header from browsers.
>

Is there a popularity cutoff % at which point we can break the web? As
stated, Refresh has 23+ years of real world use. Even ~1/5000 responses
(from a tiny 52M sample) is a huge amount on the scale of the web.

The Refresh header has niche but useful applications without obvious
(simple) substitutes. E.g. briefly showing interstitial web pages as a
graceful fallback when JavaScript is unavailable.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 12 Mar 2019 09:46:14 +0000

--------
In message <[email-masked]>, Sebastiaan Deckers writes:
>--00000000000033d3d20583e27557

>(from a tiny 52M sample) is a huge amount on the scale of the web.

... or it may just be a handful of sites which need to pull their act together ?

I don't think this kind of stuff should be measured in requests, it should
be measured in origins.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 12 Mar 2019 10:53:32 +0100 (CET)

> (from a tiny 52M sample) is a huge amount on the scale of the web.

There are so many dimensions to this! =)

1. I don't think *we* (as in the IETF or HTTPbis) would break the web at all 
since this header isn't something that is standardized or documented to begin 
with! I'm sure that no matter what we say here, the browser vendors (who seem 
to be the primary implementors of this) will have their own way of making this 
decision, and it will probably be enough with one or two fairly high profile 
sites to use the header for it to remain for a long time to come. Unless 
someone works with those sites.

2. I can't find a lot of support for this header outside of the browsers, so 
this header is already to some extent "breaking the web" by being used.

3. Since it isn't documented anywhere, it's hard to implement support for it 
in a way that doesn't introduce yet another version of dealing with it. So 
it's not clear what way that breaks the web more: leaving it unimplemented or 
implementing it in a subtly different way than how it already works 
elsewhere...

BTW, my request count was done on requests done to ~52 million different 
origins out of which exactly 9080 returned a Refresh header. That of course 
doesn't exclude that the header might be used on other origins too if another 
request had been made.

-- 

  / daniel.haxx.se

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 13 Mar 2019 04:56:01 +1300

> I expect it to refresh a single empty iframe. Once it's no longer empty

That would make it only valid on 204 responses. There is no such
restriction in the WHATWG specs and usage of meta-refresh / Refresh
seems to be to *replace* iframe contents.

> full documents, and still get Refresh capabilities. But anyway.

That would be meta-refresh in HTML, XHTML or similar in JavaScript.

The use-case for Refresh seems to me to be in non-dynamic content types
(eg image objects) where the data layer does not have a way to add the
meta-refresh.

> readers actually care about rate-limiting themselves following the

IMO, that would be better done with
Cache-Control:must-revalidate,max-age=N and/or Expires header. No need
to change the feed URL, just to trigger a re-fetch every N seconds.

AYJ

---

From: "Soni L." <[email-masked]>
Date: Tue, 12 Mar 2019 15:38:53 -0300

> seems to be to *replace* iframe contents.

if it's not ready yet:

"""
GET /next.php?marker=... HTTP/1.1
Host: ...

HTTP/1.1 200 OK
Refresh: 5
Content-Type: text/html

"""

if it's ready:

"""
GET /next.php?marker=... HTTP/1.1
Host: ...

HTTP/1.1 200 OK
Content-Type: text/html

<html><head>...</head><body><iframe 
src="/next.php?marker=..."></iframe><article>post 
here</article></body></html>
"""

(look, no JS!)

> (eg image objects) where the data layer does not have a way to add the

See above. HTML requires that iframes make separate document contexts, 
whereas in my ideal world they'd be part of the same document context.

(This is definitely the wrong mailing list tho. Just trying to 
demonstrate my use-case.)

> to change the feed URL, just to trigger a re-fetch every N seconds.

Refresh instructs the reader to re-fetch after N seconds. Dynamically 
adjusting the fetch interval allows to dynamically control flood, 
assuming all clients are well-behaved.

This would do wonders for anyone using RSS feeds, as their servers would 
no longer being hammered every 30 minutes with millions of feed requests.

Cache-Control is cool and all but it simply prevents the reader from 
fetching it again if its rate is set too fast. And even then, most 
readers don't implement cache, because they expect the feeds to be mutable.

---

From: Daniel Veditz <[email-masked]>
Date: Tue, 12 Mar 2019 14:16:31 -0700

On Tue, Mar 12, 2019 at 2:22 AM Daniel Stenberg <[email-masked]> wrote:

> are doing a web page after all?
>

Ah, but the meta tag used for this since forever is
  <meta http-equiv="refresh" content="0; URL=http://www.example.com/" />

and the "http-equiv" teaches people that there MUST be an HTTP Refresh:
header. The spec must be wrong :-)
(good luck convincing web developers otherwise)

-Dan Veditz

---

From: Daniel Veditz <[email-masked]>
Date: Tue, 12 Mar 2019 14:22:28 -0700

Smilies aside, I don't see any code that supports a Refresh: header in
Gecko. What browsers does Soni L's code work on?

-Dan Veditz

---

From: Daniel Stenberg <[email-masked]>
Date: Tue, 12 Mar 2019 23:02:15 +0100 (CET)

> Gecko. What browsers does Soni L's code work on?

Karl Dubost's blog post[*] from 2015 mentions support in:

     Yes - Firefox 36.0.4
     Yes - Opera 29.0.1795.26
     Yes - Safari 8.0.4 (10600.4.10.7)
     Yes - IE11
     Yes - Chrome (something) said Hallvord ;)

I didn't check myself and that post is four years old.

[*] = http://www.otsukare.info/2015/03/26/refresh-http-header

-- 

  / daniel.haxx.se

---

From: Daniel Veditz <[email-masked]>
Date: Tue, 12 Mar 2019 15:29:42 -0700

> Karl Dubost's blog post[*] from 2015 mentions support in:

You (and Karl) are right. I had assumed it would be http networking code
and only looked there, but it's a hack in the Document code instead.
-Dan Veditz

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 13 Mar 2019 14:38:56 +1300

> GET /next.php?marker=... HTTP/1.1

... caches will still be producing the previous 200 response payload for
up to _68 years_ due to lack of expiry info.

> here</article></body></html>

Compare that to these responses using only the existing RFC723x mechanisms:

If it's not ready yet:

"
 HTTP/1.1 307 Temporary Redirect
 Date: ...
 Location: /next.php?marker=...
 Retry-After: 5
 Cache-Control: max-age=5
"

when it is ready:

"
 HTTP/1.1 200 OK
 Content-Type: text/html
 Content-Length: xxxx

 <html>...<body><iframe
"

> whereas in my ideal world they'd be part of the same document context.

AIUI, sending data in the Refresh header is supposed to be equivalent to
sending the same value in a meta-refresh of HTML. Only send the
meta-refresh in the "not ready" responses.

Since the HTML form is much more well-known and widely used it seems to
me that the better approach is from the direction of optimizing the HTML
syntax and formally removing the overlapping HTTP part.

> assuming all clients are well-behaved.

I understand. See my counter examples above which use only mechanisms
from RFC 723x.

The fetch loop is provided by the 307 status code, the delay by the
Retry-After header, and the URL by the Location header (if any).

> readers don't implement cache, because they expect the feeds to be mutable.

The middleware does though, and the reader requests go through those
just like everything else.

AYJ

---

From: "Soni L." <[email-masked]>
Date: Tue, 12 Mar 2019 23:09:24 -0300

> up to _68 years_ due to lack of expiry info.

I was just trying to keep it short for the email/because I'm lazy.

> here</article></body></html>

This doesn't even work. Refresh is much better supported in browsers. 
And technically I do need a dummy response body actually (see below, but 
this is off-topic).

> syntax and formally removing the overlapping HTTP part.

I'm talking about how you can't style an iframe from outside the iframe. 
So you need to bloat up the response with garbage, because browsers. 
Anyway, off-topic.

> Retry-After header, and the URL by the Location header (if any).

And this lets me specify the feed's refresh interval how? It is my 
understanding that this still relies heavily on the reader having a set 
refresh interval like 30 minutes or something, whereas with Refresh 
header I'm actively telling it how often I want it to check for updates, 
with the ability to adjust it as needed and basically do all the request 
scheduling on the server, because this is the next best thing beside 
just going peer-to-peer. (Note that I do always want/need to provide a 
body for the specific case of RSS feeds.)

> just like everything else.

Definitely not with properly-configured TLS. (and I have a long list of 
complaints/stuff about how we could be doing this better, but it 
involves a new protocol.)

> AYJ

PS: I strongly agree that Refresh should NEVER be used for redirects, 
ever. It's called Refresh, not Redirect.

Additionally, when I say "properly-configured TLS", I'm referring to 
something that provides end-to-end security. As an example, signed 
messages are end-to-end secured. They're not necessarily end-to-end 
encrypted, but they're still integrity protected, which is a form of 
security. This is different from how modern CDNs do it where they just 
take your signed and encrypted message, decrypt (and sometimes, but not 
always, validate) it, and then slap their own stuff on it. And sometimes 
they don't even bother encrypting your end of it!

---

From: Yoav Weiss <[email-masked]>
Date: Wed, 13 Mar 2019 05:50:37 +0100

> Gecko. What browsers does Soni L's code work on?

I see support
<https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/loader/document_loader.cc?type=cs&g=0&l=924>
for this in Chromium's code (only on Documents AFAICT).

Usage
<https://www.chromestatus.com/metrics/feature/timeline/popularity/1548> of
the meta http-equiv seems rather high. I'll add similar usecounters for the
HTTP header to see what %age of traffic actually sees these headers.