Subject: Re: Issue 43 (combining fragments)

From: Julian Reschke <[email-masked]>
Date: Wed, 07 Oct 2009 19:27:22 +0200

Julian Reschke wrote:
> Indeed.
> 
> So I just put together a set of tests that use absolute URIs, paths (see 
> <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/185>), and combined 
> them with frag ids. See <http://greenbytes.de/tech/tc2231/redirects.html>.
> 
> The UAs I tested behaved consistently, except for the Opera issue noted 
> above.
> 
> That being said, the UAs unfortunately are also consistent in ignoring 
> the fragment when it appears in a relative-ref, not a URI; this is 
> really bad as I wouldn't want to require different behaviors depending 
> on where the fragment identifier was in.
> 
> Thus, *if* we address issue 43 by mandating that the fragment from the 
> Location header overrides a previous one, that should also apply for the 
> case of relative-refs, in which case all UAs would need to be fixed 
> (sigh!).
> ...

OK, let's summarize:

1) HTTPbis, in a very early draft, has fixed the ABNF of the Location 
header to allow fragment identifiers 
(<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/6>)

2) What it did not is define what a recipient should do when the 
referring URI contained a fragment identifier, and the Location header 
does as well. This is issue 43 
(<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/43>)

3) In the meantime, people have asked for an even less strict ABNF, 
allowing relative URIs (Issue 185: 
<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/185>)

With respect to 2), we essentially have two choices:

2a) Point out that this is ambiguous, and leave it undefined.

2b) Document what many UAs do, as that happens to be sensible: in case 
both URIs contain a fragment id, the one from the Location header 
overrides the original URI, otherwise the original fragment identifier 
is appended to the new location.

This is what was tested in the first four table entries at 
<http://greenbytes.de/tech/tc2231/redirects.html>; and it shows that UAs 
seem to be consistent here with the exception of Opera.

But. For strange reasons, when seeing just an absolute path in the 
Location header, the UAs appear to ignore the fragment identifier. This 
really doesn't make any sense, because it appears that the 
implementations follow a completely different code path here.

So for 3) the choices are:

3a) Leave the behavior undefined (possible even not allowing it in the ABNF)

3b) Document what makes sense (consistent with 2b), but point out that 
current implementations fail to do this

3c) Document what they do, even though it doesn't make any sense.

I'm currently leaning to 2b/3b, but I could be convinced to do 2a/3a 
instead. However, I don't see how I could do 2b/3c.

Feedback appreciated,

Julian

3b)

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 07 Oct 2009 20:55:51 +0200

>instead. However, I don't see how I could do 2b/3c.

Well,

  % tshark -T fields -e ip.src -e http.location -R "http.location and
     not http.location contains """:"""" -r alexa50.pcap | wc -l

Gives 14 cases of relative Locations when loading the Alexa top 50 web
sites in http://cutycapt.sf.net/ at the moment. That's a lot less than
expected but then the sample is not very representative.

In the past I've found that simply choking on relative Location headers
is not really an option for non-browser applications, so I do think the
specification should say something about them, even if that's just a
note to the effect that implementers are likely to encounter them.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 07 Oct 2009 21:01:32 +0200

> note to the effect that implementers are likely to encounter them.

Relative location headers per se aren't a problem; those that also 
include a fragment id are.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 7 Oct 2009 14:43:37 -0500

Julian Reschke wrote:
> Bjoern Hoehrmann wrote:
> > In the past I've found that simply choking on relative Location 
> > headers is not really an option for non-browser applications, so I 
> > do think the specification should say something about them, even if 
> > that's just a note to the effect that implementers are likely to 
> > encounter them.
> 

You only checked a very small number of clients. That isn't sufficient to
make such a big, incompatible, change to the syntax.

It is a good idea to warn client implementers about relative links and/or
fragment ids in the Location header. However, it is a bad idea to allow
servers to send them. It should be handled instead like obsolete syntax. 

Proposal:

-------------
9.4.  Location

   The response-header field "Location" is used for the identification
   of a new resource or to redirect the recipient to a location other
   than the request-target for completion of the request.  For 201
   (Created) responses, the Location is that of the new resource which
   was created by the request.  For 3xx responses, the location SHOULD
   indicate the server's preferred URI for automatic redirection to the
   resource.  The field value consists of a single absolute URI.

     Location       = "Location" ":" OWS Location-v
-    Location-v     = absolute-URI [ "#" fragment ]
+    Location-v     = absolute-URI

   An example is:

      Location: http://www.example.org/pub/WWW/People.html

+   The Location header field value MUST be an absolute-URI;
+   In particular, it MUST NOT be a relative-ref and it
+   MUST NOT have a fragment.
     
+   For compatibility with erroneous servers, if the 
+   Location header field value is a relative-ref (with or
+   without a fragment), a client MAY resolve it relative to
+   the request target.
+
    Note: The Content-Location header field (Section 5.7 of [Part3])
    differs from Location in that the Content-Location identifies the
    original location of the entity enclosed in the response.  It is
    therefore possible for a response to contain header fields for
    both Location and Content-Location.
-
-   There are circumstances in which a fragment identifier in a Location
-   URL would not be appropriate:
-
-  o  With a 201 Created response, because in this usage the Location
-     header specifies the URL for the entire created resource.
-
-  o  With a 300 Multiple Choices, since the choice decision is intended
-     to be made on resource characteristics and not fragment
-     characteristics.
-
-  o  With 305 Use Proxy.

Regards,
Brian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 07 Oct 2009 21:56:10 +0200

> servers to send them. It should be handled instead like obsolete syntax. 

Disagreed.

The change to allow fragments has been discussed, and approved long 
before this WG has started to work. See 
<http://skrb.org/ietf/http_errata.html#location-fragments>, which, 
unfortunately, currently is broken because that page is truncated.

The change request to allow non-absolute URIs is newer, but makes a lot 
of sense to me. First of all, it's used in the wild, and it appears 
clients accept it. Second, disallowing them simply makes no sense. 
Forcing implementers to include redundant information doesn't work well, 
and if the requirement is followed, sometimes leads to broken 
information (for instance by broken reverse proxies).

BR, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 7 Oct 2009 17:52:33 -0700

On Oct 7, 2009, at 12:56 PM, Julian Reschke wrote:

>>>> headers is not really an option for non-browser applications, so  
>>>> I do think the specification should say something about them,  
>>>> even if that's just a note to the effect that implementers are  
>>>> likely to encounter them.
>>> Relative location headers per se aren't a problem; those that  
>>> also include a fragment id are.
>> servers to send them. It should be handled instead like obsolete  
>> syntax. ...
>
> Disagreed.
>
> before this WG has started to work. See <http://skrb.org/ietf/ 
> http_errata.html#location-fragments>, which, unfortunately,  
> currently is broken because that page is truncated.
>
> The change request to allow non-absolute URIs is newer, but makes a  
> lot of sense to me. First of all, it's used in the wild, and it  
> appears clients accept it. Second, disallowing them simply makes no  
> sense. Forcing implementers to include redundant information  
> doesn't work well, and if the requirement is followed, sometimes  
> leads to broken information (for instance by broken reverse proxies).

I'll add that it has also been a long-standing request
of server developers, ever since Host was introduced, to be
able to use relative URIs in Location.  Browsers have supported
that since 1997 or so, I think.

....Roy

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 7 Oct 2009 21:15:30 -0500

> that since 1997 or so, I think.

Agreed. But, as Julian showed, these features doesn't work together
in a sensible way in any browser. 

Servers really SHOULD use the absolute form of URI for
interoperability's sake. There's really no reason not to since
every server and every server framework has been designed to
facilitate this. But, clients SHOULD accept the relative form
too, like I said. Really it's just Postel's rule.

I don't think there's any sensible way of defining the processing
of fragment identifiers in the Location header, except to acknowledge
their use for HTML documents in web browsers, and recommend that
clients should try not to choke on them. But, it seems far from
obvious that every document type in every application should be
handled that same way. For example, what should a web crawler
do with Julian's test cases? I don't think you can say that it
MUST or SHOULD replace the fragment in original URI with the
Fragment in the Location header. Keep in mind that the server
doesn't even know what the original fragment was when it minted
the Location header.

There really does need to be a clear warning that clients may
not implement these features, because they were not in RFC 2616.

Regards,
Brian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 08 Oct 2009 12:19:40 +0200

> in a sensible way in any browser. 

They do, except for the case of absolute path + fragment id, which is a 
relatively new request.

> do with Julian's test cases? I don't think you can say that it

Can you elaborate? How does it affect a web crawler in any way?

> doesn't even know what the original fragment was when it minted

Well, if we add it, it would be a specification of client behavior, not 
server behavior.

> not implement these features, because they were not in RFC 2616.

With respect to URI + fragment ID, I disagree. This has been a 
well-documented erratum for something like 10 years, and is definitely 
in use in practice.

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 8 Oct 2009 07:53:41 -0500

Julian Reschke wrote:
> Well, if we add it, it would be a specification of client
> behavior, not server behavior.

What I meant is: I don't see how you can define the client behavior
Of redirects with fragments in a way that makes sense for all
clients (not just browsers). And, even in the case of browsers, I
don't see why Opera's behavior should be considered wrong. So,
even if fragments are allowed syntactically, the semantics still
will be undefined.

Every client that will conform to this specification sends a Host
header to the server. And, a relative URL in a Location header
would always be resolved relative to that Host header value.
So, the server CAN always generate a URI; it never needs to
generate a relative URI reference in the cases where it would
be allowed to do so. Accordingly, in light of your demonstration
of how poorly these two extensions work together, I think it
makes the most sense to say something like this:

* Change the syntax to:

     Location-v = URI-reference

  This would resolve issue #185.

* Don't say anything about the semantics of fragments in
  redirects. This would close issue #43 with no change.

* Note in the "Changes from RFC 2616" that the syntax of
  the Location header was changed from absolute-URI to
  URI-reference in order to allow relative URI references
  and fragments.

I noticed the wording of the language regarding fragments is poor:

  There are circumstances in which a fragment identifier in a
  Location URI would not be appropriate: 

  * With a 201 Created response, because in this usage the
    Location header specifies the URI for the entire created
    resource.

  * With 305 Use Proxy.

This should be reworded to use RFC 2119 terminology. I suggest:

  The Location header field value MUST NOT include a fragment
  If the response status code is 201 Created or 305 Use Proxy.

Regards,
Brian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 08 Oct 2009 15:08:37 +0200

> even if fragments are allowed syntactically, the semantics still

Well, it's up to us to decide that.

Yngwe has indicated that if we clarify the spec properly they are 
willing to follow it (I guess they'll make the change anyway, as their 
behavior differs from the "other" UAs).

>   This would resolve issue #185.

Yes.

>   redirects. This would close issue #43 with no change.

If we don't want to mandate that we really should state that; otherwise 
confusion will continue.

>   If the response status code is 201 Created or 305 Use Proxy.

I prefer to use RFC 2119 terminology strictly in cases where otherwise 
interoperability would suffer (see 
<http://tools.ietf.org/html/rfc2119#section-6>).

BR, Julian