Subject: Re: Issue 43 (combining fragments)

From: Julian Reschke <[email-masked]>
Date: Wed, 03 Mar 2010 17:59:06 +0100

On 07.10.2009 19:27, Julian Reschke wrote:
> Julian Reschke wrote:
>> Indeed.
>>
>> So I just put together a set of tests that use absolute URIs, paths
>> (see <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/185>), and
>> combined them with frag ids. See
>> <http://greenbytes.de/tech/tc2231/redirects.html>.
>>
>> The UAs I tested behaved consistently, except for the Opera issue
>> noted above.
>>
>> That being said, the UAs unfortunately are also consistent in ignoring
>> the fragment when it appears in a relative-ref, not a URI; this is
>> really bad as I wouldn't want to require different behaviors depending
>> on where the fragment identifier was in.
>>
>> Thus, *if* we address issue 43 by mandating that the fragment from the
>> Location header overrides a previous one, that should also apply for
>> the case of relative-refs, in which case all UAs would need to be
>> fixed (sigh!).
>> ...
>
> OK, let's summarize:
>
> 1) HTTPbis, in a very early draft, has fixed the ABNF of the Location
> header to allow fragment identifiers
> (<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/6>)
>
> 2) What it did not is define what a recipient should do when the
> referring URI contained a fragment identifier, and the Location header
> does as well. This is issue 43
> (<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/43>)
>
> 3) In the meantime, people have asked for an even less strict ABNF,
> allowing relative URIs (Issue 185:
> <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/185>)
>
> With respect to 2), we essentially have two choices:
>
> 2a) Point out that this is ambiguous, and leave it undefined.
>
> 2b) Document what many UAs do, as that happens to be sensible: in case
> both URIs contain a fragment id, the one from the Location header
> overrides the original URI, otherwise the original fragment identifier
> is appended to the new location.
>
> This is what was tested in the first four table entries at
> <http://greenbytes.de/tech/tc2231/redirects.html>; and it shows that UAs
> seem to be consistent here with the exception of Opera.
>
> But. For strange reasons, when seeing just an absolute path in the
> Location header, the UAs appear to ignore the fragment identifier. This
> really doesn't make any sense, because it appears that the
> implementations follow a completely different code path here.
>
> So for 3) the choices are:
>
> 3a) Leave the behavior undefined (possible even not allowing it in the
> ABNF)
>
> 3b) Document what makes sense (consistent with 2b), but point out that
> current implementations fail to do this
>
> 3c) Document what they do, even though it doesn't make any sense.
>
>
> I'm currently leaning to 2b/3b, but I could be convinced to do 2a/3a
> instead. However, I don't see how I could do 2b/3c.
>
> Feedback appreciated,
>
> Julian

OK,

this issue has been open since the dawn of time, or actually, since 2616 
was published.

It would be great if we finally could make progress here.

A) The conservative choice is: allow fragments & relative-refs, state 
that fragment combination is undefined, and indeed differs between clients

B) The more ambitious approach: allow fragments & relative-refs, but 
pick the behavior we consider sane (my proposal: 2b/3b), and point out 
that some aspects of this aren't currently implemented properly

Feedback appreciated,

Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 09 Mar 2010 17:54:39 +0100

> that some aspects of this aren't currently implemented properly

OK, here's a proposal 
(<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/43/i43.diff>):

- allow relative references

- state that we do not specify fragment recombination

- stay silent on invalid URI-references (I think we should discuss this 
as an orthogonal issue).

The changed text for the Location header would be:

-- snip --
9.4.  Location

    The "Location" response-header field is used to identify a newly
    created resource, or to redirect the recipient to a different
    location for completion of the request.

    For 201 (Created) responses, the Location is the URI of the new
    resource which was created by the request.  For 3xx responses, the
    location SHOULD indicate the server's preferred URI for automatic
    redirection to the resource.

    The field value consists of a single URI-reference.  When it has the
    form of a relative reference ([RFC3986], Section 4.2), the final
    value is computed by resolving it against the effective request URI
    ([RFC3986], Section 5).

      Location       = "Location" ":" OWS Location-v
      Location-v     = URI-reference

    Examples are:

      Location: http://www.example.org/pub/WWW/People.html#tim

      Location: /index.html

    There are circumstances in which a fragment identifier in a Location
    URI would not be appropriate:

    o  With a 201 Created response, because in this usage the Location
       header specifies the URI for the entire created resource.

    o  With 305 Use Proxy.

       Note: This specification does not define precedence rules for the
       case where the original URI, as navigated to be the user agent,
       and the Location header field value both contain fragment
       identifiers.

       Note: The Content-Location header field (Section 5.7 of [Part3])
       differs from Location in that the Content-Location identifies the
       original location of the entity enclosed in the response.  It is
       therefore possible for a response to contain header fields for
       both Location and Content-Location.
-- snip --

The "Changes from RFC 2616" section would be modified to say:

-- snip --
    Correct syntax of Location header to allow URI references (including
    relative references and fragments), as referred symbol "absoluteURI"
    wasn't what was expected, and add some clarifications as to when use
    of fragments would not be appropriate.  (Section 9.4)
-- snip --

Feedback appreciated,

Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 10 Mar 2010 13:19:22 +0100

> The changed text for the Location header would be:

Applied with <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/785>.

I'll send a separate summary, and cc' the W3C HTML WG, trying (yet 
again) to get feedback from browser vendors about whether they're 
willing to go further than that.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 10 Mar 2010 14:06:26 +0100

Hi,

the HTTPbis WG has two open tickets with respect to the syntax and 
semantics of the Location header (not to be confused with 
Content-Location!):

a) <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/43>: "Fragment 
combination / precedence during redirects"

This is about how to handle the case when both the original URI and the 
value of the Location header contain a fragment identifier.

b) <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/185>: "Location 
header payload handling"

This is about allowing relative references (in addition to full URIs), 
and also about treating broken values (*).

Some time ago I created a few tests, see 
<http://greenbytes.de/tech/tc2231/redirects.html>.

What I found was that

- UAs appear to treat those consistently (with the exception of Opera, 
but Yngwe already signaled that he's willing to adapt), *but*

- the behavior I found unfortunately doesn't make sense (treating the 
cases for absolute URIs and relative references differently).

Specifically, I would expect UAs to let the fragment ID found in the 
Location header (when present) override the original URI's. But we found 
this to be only the case for absolute URIs.

At this point, and with no further feedback from browser vendors about 
whether they'd consider changing the behavior for relative references, 
we changed the spec to clarify that the fragment recombination behavior 
is undefined (previously, the spec didn't say anything about this).

The new text for "Location" is:

-- snip --
9.4.  Location

    Examples are:

      Location: /index.html

    o  With 305 Use Proxy.

(see also 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p2-semantics-latest.html#header.location>).

Of course it would be nice if we *could* recommend a specific behavior 
that makes sense, so I would be very grateful to get feedback from UA 
implementers about whether they'd consider treating relative references 
and absolute URIs consistently.

Best regards, Julian

(*) We'll look at *that* issue separately.

---

From: "Toby Inkster" <[email-masked]>
Date: Wed, 10 Mar 2010 13:32:08 -0000

> but Yngwe already signaled that he's willing to adapt), *but*

Hmmm... Opera's behaviour seems nicer. It's a shame to lose that, but
pragmatically one vendor changing their behaviour is going to be simpler
than all of the others doing so.

-Toby

---

From: Jonas Sicking <[email-masked]>
Date: Wed, 10 Mar 2010 08:15:35 -0800

On Wed, Mar 10, 2010 at 5:06 AM, Julian Reschke <[email-masked]> wrote:
> Hi,
>
> the HTTPbis WG has two open tickets with respect to the syntax and semantics
> of the Location header (not to be confused with Content-Location!):
>
> b) <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/185>: "Location header
> payload handling"
>
> This is about allowing relative references (in addition to full URIs), and
> also about treating broken values (*).
>
> - UAs appear to treat those consistently (with the exception of Opera, but
> Yngwe already signaled that he's willing to adapt), *but*
>
> - the behavior I found unfortunately doesn't make sense (treating the cases
> for absolute URIs and relative references differently).
>
> whether they'd consider changing the behavior for relative references, we
> changed the spec to clarify that the fragment recombination behavior is
> undefined (previously, the spec didn't say anything about this).

Leaving things undefined seems like a very bad idea. If it really
doesn't matter today what UAs do, then my experience is that sometime
in the future sites will start depending on it one way or another.

That said. I don't see that firefox treats relative and absolute URIs
any differently. The code at [1], which handles redirects, seems to
forward the fragment ID from the original uri if the new uri doesn't
have a fragment, and otherwise use the fragment ID of the new
location. No distinction is made based on if the location header
contained a relative or absolute URI.

Do you have a testcase where we can test the behavior?

/ Jonas

---

From: Boris Zbarsky <[email-masked]>
Date: Wed, 10 Mar 2010 11:27:39 -0500

> semantics of the Location header (not to be confused with

As long as you're doing this, can you please define the behavior for 
non-ASCII values of the header?  At the moment the header only allows 
URIs, with non-ASCII bytes having undefined behavior.  UAs treat this 
inconsistently (some treat the bytes as UTF-8, some URI-escape all 
non-ASCII bytes; there may be other behaviors).  This particularly comes 
up with IDN hostnames and URL minifiers which just stick the non-ASCII 
hostname into their Location header.

-Boris

---

From: Julian Reschke <[email-masked]>
Date: Wed, 10 Mar 2010 17:32:37 +0100

> (*) We'll look at *that* issue separately.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 10 Mar 2010 18:20:14 +0100

> Do you have a testcase where we can test the behavior?

I thought I did (mentioned in my previous mail); but I just decided to 
double-check them, given your confidence (the tests were written in a 
hurry during a conference).

Turns out they relied on Apache/mod_asis to treat Location headers as 
opaque data, which apparently is not the case. So the test cases at 
<http://greenbytes.de/tech/tc2231/redirects.html> work only for those 
four tests where the redirect target is a full URI. In the other cases, 
mod_asis apparently resolves the path internally, and never sends the 
redirect.

Summary: please ignore this for now; I will have to produce test cases 
that actually test what I wanted to test.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 11 Mar 2010 16:35:57 +0100

> that actually test what I wanted to test.

In the meantime we've got fixed test cases (thanks, Mark); the results 
are attached as HTML.

The good news is that the UAs indeed treat fragments in URIs and 
relative references consistently (as Jonas reported for Firefox).

The other good news is that Firefox, IE and Chrome handle them the same 
way, and that happens to be what makes most sense (IMHO):

1) given an original URI containing a fragment identifier

2) resolve the redirect URI-reference against the original URI

3) if the resulting URI does not have a fragment identifier, use the one 
from the original URI instead

Deviations:

- Opera gives precedence to the original URI's fragment, if present

- Safari doesn't seem to implement step 3.

So compared to what I thought previously the situation is much better, 
as the most widely implemented behavior actually makes sense.

Rephrased request for feedback:

Should we recommend the behavior we see implemented (SHOULD? MUST?)? 
Note that this would make current implementations of Opera and Safari 
non-compliant.

Best regards, Julian

---

From: Jonas Sicking <[email-masked]>
Date: Thu, 11 Mar 2010 07:38:43 -0800

On Thu, Mar 11, 2010 at 7:35 AM, Julian Reschke <[email-masked]> wrote:
> Should we recommend the behavior we see implemented (SHOULD? MUST?)? Note
> that this would make current implementations of Opera and Safari
> non-compliant.

Is there a reason to use SHOULD rather than MUST? If not I'd say use MUST.

/ Jonas

---

From: "Anne van Kesteren" <[email-masked]>
Date: Thu, 11 Mar 2010 16:41:53 +0100

> Is there a reason to use SHOULD rather than MUST? If not I'd say use  

In case it matters, rendering us non-compliant is fine if fixing our "bug"  
makes us more interoperable with Fx/IE.

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: Julian Reschke <[email-masked]>
Date: Thu, 11 Mar 2010 16:48:18 +0100

On 11.03.2010 16:38, Jonas Sicking wrote:
> On Thu, Mar 11, 2010 at 7:35 AM, Julian Reschke<[email-masked]>  wrote:

Usually we don't add normative requirements on top of RFC 2616, unless 
we're clearly fixing a bug (which is not the case here), or are 
confident that we're just writing down what everybody is doing anyway.

 From that point of view, even a SHOULD may be too strong.

Best regards, Julian

---

From: Jonas Sicking <[email-masked]>
Date: Thu, 11 Mar 2010 09:22:10 -0800

On Thu, Mar 11, 2010 at 7:48 AM, Julian Reschke <[email-masked]> wrote:
> Usually we don't add normative requirements on top of RFC 2616, unless we're
> clearly fixing a bug (which is not the case here), or are confident that
> we're just writing down what everybody is doing anyway.

Why? Isn't the point of a spec to encourage interoperable behavior?

/ Jonas

---

From: Julian Reschke <[email-masked]>
Date: Thu, 11 Mar 2010 18:28:16 +0100

On 11.03.2010 18:22, Jonas Sicking wrote:
> On Thu, Mar 11, 2010 at 7:48 AM, Julian Reschke<[email-masked]>  wrote:

It depends.

If there's no interop today, and the existing implementations are 
conforming with respect to RFC 2616, we *usually* don't break them - 
there would need to be very good reasons to do so, such as security 
related ones.

Best regards, Julian

---

From: Jonas Sicking <[email-masked]>
Date: Thu, 11 Mar 2010 09:52:42 -0800

On Thu, Mar 11, 2010 at 9:28 AM, Julian Reschke <[email-masked]> wrote:
>>>>> Note
>>> we're
> If there's no interop today, and the existing implementations are conforming
> with respect to RFC 2616, we *usually* don't break them - there would need
> to be very good reasons to do so, such as security related ones.

I can't say that I agree with that reasoning. IMHO interoperability
going forward is more important than not declaring currently
conforming implementations non-conforming. If anyone gets really sad
for loosing their conforming badge, I can send them some home made
cookies ;)

/ Jonas

---

From: "Anne van Kesteren" <[email-masked]>
Date: Thu, 11 Mar 2010 20:08:59 +0100

> for loosing their conforming badge, I can send them some home made

Yes please! :-)

---

From: Julian Reschke <[email-masked]>
Date: Fri, 12 Mar 2010 12:11:47 +0100

> - Safari doesn't seem to implement step 3.

The Safari behavior already has an associated Webkit bug, see 
<https://bugs.webkit.org/show_bug.cgi?id=24175>.

The bug comments also contained a pointer to 
<http://www.w3.org/Protocols/HTTP/Fragment/draft-bos-http-redirect-00.txt>, 
which I guess is interesting historical context...

Best regards, Julian

---

From: Toby Inkster <[email-masked]>
Date: Fri, 12 Mar 2010 11:21:28 +0000

On Thu, 2010-03-11 at 16:35 +0100, Julian Reschke wrote:
> The other good news is that Firefox, IE and Chrome handle them the
> same way, and that happens to be what makes most sense (IMHO) 

It makes sense from the "let's do the least work to harmonise this"
point of view, but I don't think it makes a lot of sense from an end
user point of view.

Let's say that a long document is published as a series of pages:

	section1.html (containing #subsection1.1, #subsection1.2, etc)
	section2.html (containing #subsection2.1, #subsection2.2, etc)
	... etc

Later, it is realised that the document is not long enough to justify
the split, so they're merged into a single page:

	all.html (containing #section1, #subsection1.1, etc)

To prevent links from being broken a set of 302 Found redirects are
established:

	section1.html -> all.html#section1
	section2.html -> all.html#section2
	... etc

Now, imagine a user clicks on a link to <section1.html#subsection1.2>.
Their browser requests <section1.html> and receives a 302 redirect to
<all.html#section1>. 

Which part of the page does it make more sense to scroll the user down
to? #section1 or #subsection1.2? I would argue the latter. The server
could have redirected the browser to the latter fragment itself, if only
it had known the fragment part of the user's original request (though of
course it can't, as the fragment isn't sent over the wire in the
request).

The Firefox/IE/Chrome behaviour I would say perhaps makes sense in the
case of 303 See Other. A 303 response indicates that the resource in the
Location header is a separate resource; a 302 response indicates that
the resource in the Location header is the same resource that was
requested, just in a different place. So with 302 responses, there
should be an expectation that fragment identifiers in the originally
requested resource match any fragment identifiers in the resource from
the Location header.

-- 
Toby A Inkster
<mailto:[email-masked]>
<http://tobyinkster.co.uk>

---

From: Jonas Sicking <[email-masked]>
Date: Fri, 12 Mar 2010 14:21:35 -0800

>> for loosing their conforming badge, I can send them some home made

If updating the spec to use MUST requirements here makes you sad,
please do send me the address of the Opera HQ and I guess I'll send
the cookies. I do reserve the right to blog about an apparent change
in Operas attitude towards making specs have strict requirements
though ;)

All under the condition that Julian changes the spec of course.

/ Jonas

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 13 Mar 2010 11:13:50 +1100

FYI, tests online at:
  http://www.mnot.net:8000/
Source at:
  http://gist.github.com/330963

My take on this is that it's an area that's important for interoperability that is the responsibility of HTTP, but was omitted from the original specification. We can consider making a clarification here, and if it breaks an implementation or two, we can note that potential incompatibility (just as 2616 did in a few cases).

Cheers,

--
Mark Nottingham     http://www.mnot.net/