Subject: Sending Referer [#144]

From: Mark Nottingham <[email-masked]>
Date: Tue, 2 Jun 2009 01:50:11 +1000

<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/144>

There are some aspects of #144 that should be easy; e.g., clarifying  
that sending Referer isn't dependent of the protocol of the source.

However, in previous discussions, Adam et al indicated that it would  
be interesting to require that Referer always be sent, by minting a  
new value (e.g., 'null', although it will have to be something else,  
since "null" is a valid partial-URI) to indicate when a Referer is not  
available.

I see a few potential problems here...

1) Currently the draft (and 2616) says "The Referer field MUST NOT be  
sent if the request-target was obtained from a source that does not  
have its own URI, such as input from the user keyboard."  We'd have to  
remove this requirement and say something like "The Referer field MAY  
be sent with the "null" value if the request-target..."

SHOULD or MUST isn't possible here, because existing clients would  
become non-conformant. Technically, a server could have a problem with  
the 'null' value (as well as the fact that there isn't a referer, but  
the header is being sent), but it seems doubtful that this will cause  
an issue in practice...

2) A new requirement would need to be added to make the referer field  
compulsory, but again this would make some existing clients non- 
conformant.

Purely within HTTPbis, I think the best we could do, then, would be to  
add a null value that can be sent when appropriate, and encourage UAs  
to send referer. However, there's nothing stopping a third party  
document (e.g., a W3C browser profile) from requiring that it be sent...

Adam, what do you think? Others?

Cheers,

--
Mark Nottingham     http://www.mnot.net/

---

From: Adam Barth <[email-masked]>
Date: Mon, 1 Jun 2009 11:40:51 -0700

> Adam, what do you think? Others?

This sounds good.  I don't quite understand the constraints we're
operating under with respect to making existing clients / servers
non-conformant, but I trust you've done that analysis correctly.

With regards to the 'null' value, we should try to pick a value that's
friendly to regular expressions (i.e., avoids . / [ and similar
characters) because many web application firewalls (who would use this
value) express their rules in terms of regular expressions.

Adam

---

From: Adam Barth <[email-masked]>
Date: Mon, 1 Jun 2009 12:03:10 -0700

> value) express their rules in terms of regular expressions.

One possible value we could use is

about:blank

There is an I-D kicking around somewhere that defines the about
scheme.  The author might be amenable to allowing about:blank to be
used here (e.g., as a "placeholder" URI when no other URI is
appropriate).

Another possibility is to use something like

about:noreferrer

and define about:noreferrer to return a resource that describes what
the value means in the Referer header (e.g., the relevant requirements
from HTTPbis).

Adam

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 1 Jun 2009 14:14:26 -0500

> document (e.g., a W3C browser profile) from requiring that it be sent...

There's no way to represent the null value in a way compatible with RFC 2616
syntax. New syntax for existing headers should be done only for HTTP 1.2 or
later. Further, this would be a new feature, which would make it out of
scope for HTTPbis.

I think that the less that is said about Referer in the HTTPbis
specification, the better. Its syntax and intended purpose is enough. Any
requirements regarding Referer for UAs are pointless; they will be ignored
by most UAs, especially UAs that aren't browsers. A specification for what
web browsers should do with the Referer should go into a browser-specific
specification.

Regards,
Brian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 01 Jun 2009 21:14:42 +0200

> the value means in the Referer header (e.g., the relevant requirements

That will be tricky to use, as it would be a downref to a Proposed 
Standard (*). HTTPbis will be (at least) one step ahead on the standards 
ladder).

BR, Julian

(*) Optimistically.

---

From: Adam Barth <[email-masked]>
Date: Mon, 1 Jun 2009 14:11:40 -0700

On Mon, Jun 1, 2009 at 12:14 PM, Brian Smith <[email-masked]> wrote:
> syntax.

Why is that?  The RFC 2616 syntax appears to be happy with an
absoluteURI.  We need only represent the null value using an
absoluteURI.

> New syntax for existing headers should be done only for HTTP 1.2 or

This appears to be clarifying the semantics of the Referer header, not
adding a new feature.  As in "if you see a Referer header with this
value, then that means there wasn't a referring URI or the user agent
got confused and couldn't figure out the referring URI properly."

> web browsers should do with the Referer should go into a browser-specific

As things stand, the document forbids user agents from always sending
the Referer header, preventing a browser-specific specification from
requiring this behavior.

Adam

---

From: Adam Barth <[email-masked]>
Date: Mon, 1 Jun 2009 14:14:13 -0700

>> One possible value we could use is

[...]

>> about:noreferrer
>
> That will be tricky to use, as it would be a downref to a Proposed Standard
> (*). HTTPbis will be (at least) one step ahead on the standards ladder).

We need only suggest (require?) the literal "about:blank" without a
downref.  That is syntactically an absolute URI, and the use should
not conflict with any other use of that sequence of bytes.

Adam

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 8 Jun 2009 09:20:48 +1000

I'm not crazy about this approach, but it may be workable.

p2 section 9.6 is currently:

>    The request-header field "Referer" [sic] allows the client to
>    specify, for the server's benefit, the address (URI) of the  
> resource
>    from which the request-target was obtained (the "referrer",  
> although
>    the header field is misspelled.)  The Referer request-header  
> allows a
>    server to generate lists of back-links to resources for interest,
>    logging, optimized caching, etc.  It also allows obsolete or  
> mistyped
>    links to be traced for maintenance.  The Referer field MUST NOT be
>    have its own URI, such as input from the user keyboard.
>
>      Referer        = "Referer" ":" OWS Referer-v
>      Referer-v      = absolute-URI / partial-URI
>
>    Example:
>
>      Referer: http://www.example.org/hypertext/Overview.html
>
>    If the field value is a relative URI, it SHOULD be interpreted
>    relative to the request-target.  The URI MUST NOT include a  
> fragment.
>    See Section 11.2 for security considerations.

Based on discussion so for, here's a straw-man:

"""
The request-header field "Referer" [sic] allows the client to specify,  
for the server's benefit, the address (URI) of the resource from which  
the request-target was obtained (the "referrer", although the header  
field is misspelled.).

The Referer request-header allows servers to generate lists of back- 
links to resources for interest, logging, optimized caching, etc. It  
also allows obsolete or mistyped links to be traced for maintenance.  
Some servers use Referer as a means of controlling where they allow  
links from (so-called "deep linking"), but it should be noted that  
legitimate requests are not required to contain a Referer header field.

If the request-target was obtained from a source that does not have  
its own URI (e.g., input from the user keyboard), the Referer field  
MUST either be sent with the value "about:blank", or not be sent at  
all. Note that this requirement does not apply to sources with non- 
HTTP URIs (e.g., FTP).

   Referer        = "Referer" ":" OWS Referer-v
   Referer-v      = absolute-URI / partial-URI

Example:

   Referer: http://www.example.org/hypertext/Overview.html

If the field value is a relative URI, it SHOULD be interpreted  
relative to the request-target. The URI MUST NOT include a fragment.  
See Section 11.2 for security considerations.
"""

On 02/06/2009, at 7:14 AM, Adam Barth wrote:

>> Standard

---

From: Julian Reschke <[email-masked]>
Date: Mon, 08 Jun 2009 08:44:53 +0200

Mark Nottingham wrote:
> ...
> ...

Sounds good to me. Proposed patch: 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/144/144.diff>.

Note that I replaced "request header" by "header" in the second 
paragraph, and that I also added the Change to "A.2.  Changes from RFC 
2616" as:

"""
    Allow Referer value of "about:blank" as alternative to not specifying
    it.  (Section 9.6)
"""

BR, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 12 Jun 2009 14:28:30 +1000

Objections?

On 08/06/2009, at 4:44 PM, Julian Reschke wrote:

> Sounds good to me. Proposed patch: <http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/144/144.diff 
> >.
>
> paragraph, and that I also added the Change to "A.2.  Changes from  
> RFC 2616" as:
>
> """

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 13 Jun 2009 01:10:13 +0200

> since "null" is a valid partial-URI) to indicate when a Referer is not  

Not having that discussion in front of me, but why would one want this?

A non-existing Referer header means that the user agent either don't
have a referer URI, or do not want to tell what it was. How is sending a
"null" Referer header different from this?

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 13 Jun 2009 01:13:37 +0200

> the Referer header, preventing a browser-specific specification from

Browser specification in this case should be in the lines of "If the
request was intitiated by reference from another object with a known URI
then the Referer header SHOULD be sent indicating the URI of the
referencing resource."

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Sat, 13 Jun 2009 01:16:26 +0200

> Objections?

I do not understand the value in having two ways of representing the
exact same thing..

Regards
Henrik

---

From: Adam Barth <[email-masked]>
Date: Fri, 12 Jun 2009 16:41:18 -0700

> "null" Referer header different from this?

We've already covered this in previous discussion, but the high level
reason is so severs can distinguish the following two cases:

1) The Referer header was striped from the request in transit.
2) The User Agent did not attach a Referer because on particular URI
was appropriate.

Being unable to distinguish these cases prevent servers from being
able to use the Referer header to mitigate CSRF vulnerabilities.  For
more details, please see the archives.

Adam

---

From: Adam Barth <[email-masked]>
Date: Fri, 12 Jun 2009 16:43:05 -0700

> then the Referer header SHOULD be sent indicating the URI of the

That is insecure.  For example, if the referring URI is an HTTPS URI,
then this would ask that user agents send the path and query string of
the HTTPS URI in the clear over the network, potentially disclosing
sensitive information in those parts of the URI, such as authorization
tokens.

Adam

---

From: Adam Barth <[email-masked]>
Date: Fri, 12 Jun 2009 16:43:46 -0700

> I do not understand the value in having two ways of representing the

Please see the archives of this list for an in depth discussion of this point.

Adam

---

From: Julian Reschke <[email-masked]>
Date: Sat, 13 Jun 2009 09:14:36 +0200

> Objections?

Change applied with 
<http://trac.tools.ietf.org/wg/httpbis/trac/changeset/593>.

BR, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 25 Jun 2009 15:37:42 +1000

Closed. If we have more information about requirements, we can either  
reopen or start a new issue.

Cheers,

On 13/06/2009, at 5:14 PM, Julian Reschke wrote:

> Mark Nottingham wrote:
>> Objections?
>> ...
>
> Change applied with <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/593 
> >.
>
> BR, Julian
>