Subject: Re: Referer URI MUST NOT include a fragment

From: Julian Reschke <[email-masked]>
Date: Sun, 01 Mar 2009 12:19:57 +0100

Vincent Murphy wrote:
> During a discussion [0] about why Youtube uses ?feature=related in its 
> URIs, I observed that the Referer header URI, if it included a fragment 
> identifier, could be used identify the anchor used to initiate a GET. 
> This would be useful for
> 
> - analysing anchor popularity,
> - eliminating the need for workarounds and hacks like Youtube 
> ?feature=related
> - encourage cleaner, canonical URIs.
> 
> I did a search of discussions around the HTTP protocol, but was not able 
> to find the origin of the statement from RFC2616 Section 14.32 [1], 
> paraphrased in the subject of this message. This statement is also in 
> draft-ietf-httpbis-p2-semantics-05, section 10.6 [2].
> 
> I seek links to the discussion or rationale and origin of this 
> statement, or failing that, comments about how allowing fragment 
> identifiers in Referer URIs would enhance or violate web architecture.
> 
> Thanks,
> -Vincent Murphy

Reminder: if we *did* want to relax this in HTTPbis, we will need to 
investigate whether relaxing the value range can break existing code.

BR, Julian

---

From: Albert Lunde <[email-masked]>
Date: Sun, 1 Mar 2009 10:25:13 -0500

> investigate whether relaxing the value range can break existing code.

It's going to break existing web applications that do equality
tests on Referer for (weak) security, or to prevent deep linking
into web sites. Say, substring matches on hostname, won't be affected.
(All these things have to allow for the case that Referer
is not sent, but they can be brittle in other respects.)

So the effect will be breakage of unspecified sites by the first browser
to adopt  it.

-- 
    Albert Lunde  [email-masked]
                  [email-masked]  (new address for personal mail)
                  [email-masked] (old address)

---

From: Vincent Murphy <[email-masked]>
Date: Mon, 2 Mar 2009 20:24:33 +0000

Thank you for your comment. Do you think the benefits outlined earlier
outweigh these costs?

I think this ranks way below a data spill as discussed earlier.
Authentication checks based on string equality may fail because of a
fragment identifier but they definitely won't succeed.

It doesn't seem like a huge burden to implement string prefix checks
instead. Its common knowledge that HTTP clients drop frag ids in
requests so it wouldn't be a huge gap to expect people to understand a
similar mechanism for Referer handling.

I don't really care about making life easier for those who wish to
restrict deep linking... :)

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 03 Mar 2009 09:47:10 +1300

Is there any case at all where currently a fragment ID hits the "wire"?  
I thought it was purely an in-browser feature, not really dependent on HTTP.

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Albert Lunde <[email-masked]>
Date: Mon, 2 Mar 2009 17:49:20 -0500

> restrict deep linking... :)

The drawback of making any change here, is that the effect isn't just
on web server or web client developers, but the much broader
set of web application developers.  Dynamic tests on Referer: are often
done in things like mod_rewrite or JavaScript, making it harder
to make them perfectly general. Most uses of Referer: in real time
(as opposed to after the fact log analysis) are kind of hacky
and un-eligant.

The fact that you or I may not like the ways it is used (as a value
judgement) doesn't mean they aren't out there waiting to break.