Subject: Referer URI MUST NOT include a fragment

From: Vincent Murphy <[email-masked]>
Date: Sat, 14 Feb 2009 16:40:44 +0000

During a discussion [0] about why Youtube uses ?feature=related in its  
URIs, I observed that the Referer header URI, if it included a  
fragment identifier, could be used identify the anchor used to  
initiate a GET. This would be useful for

- analysing anchor popularity,
- eliminating the need for workarounds and hacks like Youtube ? 
feature=related
- encourage cleaner, canonical URIs.

I did a search of discussions around the HTTP protocol, but was not  
able to find the origin of the statement from RFC2616 Section 14.32  
[1], paraphrased in the subject of this message. This statement is  
also in draft-ietf-httpbis-p2-semantics-05, section 10.6 [2].

I seek links to the discussion or rationale and origin of this  
statement, or failing that, comments about how allowing fragment  
identifiers in Referer URIs would enhance or violate web architecture.

Thanks,
-Vincent Murphy

0. http://www.reddit.com/r/programming/comments/7x49v/canonical_url_tag_the_most_important_advancement/c07ne0v
1. http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.36
2. http://www.ietf.org/internet-drafts/draft-ietf-httpbis-p2-semantics-05.txt

---

From: "Hausenblas, Michael" <[email-masked]>
Date: Sat, 14 Feb 2009 17:00:56 +0000

Vincent,

We're having this discussion in the W3C Media Fragments Working Group  
as well. More details can be provided (I'm on travel and hence  
limited ;)

Cheers, Michael

Sent from my iPhone

On 14 Feb 2009, at 16:43, "Vincent Murphy" <[email-masked]> wrote:

> During a discussion [0] about why Youtube uses ?feature=related in  
> its URIs, I observed that the Referer header URI, if it included a

---

From: "Robert Brewer" <[email-masked]>
Date: Sat, 14 Feb 2009 10:58:15 -0800

> identifiers in Referer URIs would enhance or violate web architecture.

One such would be http://www.w3.org/DesignIssues/Fragment

Robert Brewer
[email-masked]

---

From: Adrien de Croy <[email-masked]>
Date: Sun, 15 Feb 2009 11:20:15 +1300

Have Youtube themselves been involved in any discussion about the use 
for ?feature=related?

Seems to me it's not anything to do with Referer headers, but occurs 
when a user actively clicks on a link to see a related video.  Such 
"related" video is returned by youtube from a previous request.  This 
allows youtube to track whether people click on links related videos or 
not - e.g. gauge effectiveness of this system.

I don't see how the referer header with a fragment would be any better.

Vincent Murphy wrote:
> - eliminating the need for workarounds and hacks like Youtube 
> ?feature=related
> 0. 
> http://www.reddit.com/r/programming/comments/7x49v/canonical_url_tag_the_most_important_advancement/c07ne0v 
>
> 2. 
> http://www.ietf.org/internet-drafts/draft-ietf-httpbis-p2-semantics-05.txt 
>
>

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Vincent Murphy <[email-masked]>
Date: Sun, 15 Feb 2009 13:03:32 +0000

2009/2/14 Adrien de Croy <[email-masked]>:
> Have Youtube themselves been involved in any discussion about the use for
> ?feature=related?

Not to my knowledge. I wouldn't expect a engineering organisation like
Youtube to engage with a standards community in this situation, but to
just engineer a solution that works for their current requirements,
even if it means deviating a little from web architecture.

The point isn't what Youtube do per-se, but the general pattern that
web authors use query parameters in their URL to track which anchor
was used to access a link. This leads to new URIs being made which
don't correspond to individual resources, and in a sense they are
'diluted'. I welcome other examples if anybody can recall them.

I'm just trying to explore whether it can be made easier to do the
Right Thing. I would be willing to work on patches for HTTPbis, Webkit
and Firefox if we can build consensus that this is a good idea.

> Seems to me it's not anything to do with Referer headers, but occurs when a
> user actively clicks on a link to see a related video.  Such "related" video
> is returned by youtube from a previous request.  This allows youtube to
> track whether people click on links related videos or not - e.g. gauge
> effectiveness of this system.

That's not my understanding. The semantics of the URI are that
?feature=related refers to which part of the linking page was clicked
on. The author wants to know which of the anchors was used to access
the link. The same link is often used in several different anchors on
the same page, and page authors want to know which anchors are most
effective.

> I don't see how the referer header with a fragment would be any better.

Does what I have said change your mind? Perhaps I am not describing
the use case adequately.

---

From: Albert Lunde <[email-masked]>
Date: Mon, 16 Feb 2009 00:07:38 -0500

> was used to access a link. This leads to new URIs being made which

It seems like one issue is that fragment identifies play a larger
role in HTML than they do in HTTP. HTTP is dealing with requests
for resources, not clicks on a page. Referer does mix the levels to some
extent, but it can apply as much to a web robot as to a browser.
-- 
    Albert Lunde  [email-masked]
                  [email-masked]  (new address for personal mail)
                  [email-masked] (old address)

---

From: Larry Masinter <[email-masked]>
Date: Wed, 25 Feb 2009 16:05:48 -0800

I think the idea of allowing fragment identifiers in
Referer is interesting, and I'm not sure what it would
break. It couldn't be mandated. I think the privacy
security concerns about Referer remain, and perhaps
the restriction was just a way of minimizing the
exposure?

The important limits on Referer in RFC 2616
are in the "Security Considerations" section
http://tools.ietf.org/html/rfc2616#section-15.1.2

At least a while ago, it was looking like the
"Origin" header proposal might instead be subsumed
by an extension to "Referer" instead, which seemed
like a positive direction. I don't think allowing
fragment identifiers in Referer for other purposes
would interfere with that.

Larry
-- 
http://larry.masinter.net

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 26 Feb 2009 16:25:01 +1100

Yes - part of that discussion needs to be exactly what is included in  
the referer URI.

Cheers,

--
Mark Nottingham     http://www.mnot.net/

---

From: Vincent Murphy <[email-masked]>
Date: Sat, 28 Feb 2009 11:39:40 +0000

The id attribute value (if one exists) of the <a> anchor element used
to initiate the HTTP request in the user agent would be used as a
fragment identifier in the referer URI. This is in exactly the same
way as fragment identifiers is used to identify a destination anchor
in current practice.

I think this mechanism is portable across media types.

2009/2/26 Mark Nottingham <[email-masked]>:
> Yes - part of that discussion needs to be exactly what is included in the
> referer URI.

---

From: Julian Reschke <[email-masked]>
Date: Sat, 28 Feb 2009 13:05:47 +0100

Vincent Murphy wrote:
> ...

Or of the closest ancestor...?

BR, Julian

---

From: Vincent Murphy <[email-masked]>
Date: Sat, 28 Feb 2009 15:01:34 +0000

> Or of the closest ancestor...?

+1

---

From: Vincent Murphy <[email-masked]>
Date: Sat, 28 Feb 2009 22:01:12 +0000

2009/2/26 Larry Masinter <[email-masked]>:
> break. It couldn't be mandated.

A SHOULD would be good enough because it would allow user agent
implementers to implement without contradicting HTTPbis. That would
make it much easier to advocate for this with incumbent browser
vendors.

> I think the privacy

If the user information in web pages is based on the cookie/session;
an agent accessing the Referer resource wouldn't see that user
information. I believe this to be the case with most web pages.

I don't think the Referer would expose much information over and above
that which is already gathered in session logs using cookies.

> http://tools.ietf.org/html/rfc2616#section-15.1.2

Thanks for the reference; I hadn't read that.

Perhaps security concerns like this could be addressed by having a
attribute on <a>, e.g. referer="none", which allows the author to
suppress the Referer header altogether. The default behaviour would be
to include a Referer header with the fragment id.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 1 Mar 2009 11:31:07 +1100

Well, not *that* portable. I don't think HTTP should be talking about  
the syntactic details of specific media types. Delegating this to  
media type definitions would be the most sensible long-term thing to  
do, although some sort of advice would be needed in the meantime...

---

From: Vincent Murphy <[email-masked]>
Date: Sun, 1 Mar 2009 01:10:57 +0000

2009/3/1 Mark Nottingham <[email-masked]>:
> Well, not *that* portable. I don't think HTTP should be talking about the
> syntactic details of specific media types. Delegating this to media type
> definitions would be the most sensible long-term thing to do, although some
> sort of advice would be needed in the meantime...

I agree; but at the moment the spec says Referer MUST NOT include a
fragment identifier, and the case that I am making is, in essence,
this should be relaxed to SHOULD or MAY. I would expect an implementor
to shoot this proposal down if it means contradicting HTTPbis as it
currently stands.

I see no reason why this proposal could not preserve the delegation of
the semantics of source fragment identifiers to media types. However,
it might be pragmatic for HTTPbis to offer non-normative guidance or
advise on HTML, which I feel would cover a large majority of use
cases. I think this could be done in a single paragraph, paraphrasing
this thread. I would also be willing to draft a seperate patch for
HTML5 if you think that's the way to go, but it would obviously help
if one could point to consensus on HTTPbis.