Subject: Re: URI handling for Location

From: Mark Nottingham <[email-masked]>
Date: Tue, 7 Apr 2009 14:18:18 +1000

Anne et al,

As discussed at the bar bof/informal liaison in SF, this seems to be  
more of a URI issue; i.e., HTML and browser vendors need a named  
construct ("dirty URI", "hypertext reference", etc.) for what goes  
into the process, but from an HTTP perspective, this isn't evident.

AIUI the follow-up will happen on the URI mailing list.

Cheers,

On 30/03/2009, at 11:25 PM, Anne van Kesteren wrote:

> Hi,
>
> The definition of the Location header differs in various ways of how  
> at least Web browsers need to deal with them to interoperate with  
> content on the Web:
>
> 1. Need to handle relative URIs.
> 2. Need to handle with spaces and other invalid URI characters in  
> the same way as done by e.g. HTML and CSS. (Percent-encode them  
> rather than treat it as error.)
>
> I suspect that any other tool that wants to deal with content on the  
> Web would have similar issues. There's some discussion here  
> including two sites this effects:
>
>  http://krijnhoetmer.nl/irc-logs/whatwg/20090323#l-152
>
> (I suspect that every feature that takes a URI is similarly  
> affected, but I have not toyed with those.)
>
> (There was also a related issue for Location regarding how to handle  
> it when there's more than one such header, but I believe that was  
> raised already.)
>
>
> -- 
> Anne van Kesteren
> http://annevankesteren.nl/
>

--
Mark Nottingham     http://www.mnot.net/

---

From: "Anne van Kesteren" <[email-masked]>
Date: Tue, 07 Apr 2009 11:16:14 +0200

On Tue, 07 Apr 2009 06:18:18 +0200, Mark Nottingham <[email-masked]> wrote:
> construct ("dirty URI", "hypertext reference", etc.) for what goes into  
> the process, but from an HTTP perspective, this isn't evident.
>

Allowing relative URIs hardly seems a "dirty URI" issue. As for the actual  
"dirty URI" issue, that isn't solved as long as HTTP keeps insisting the  
header takes an actual URI rather than such a "dirty URI".

---

From: "Anne van Kesteren" <[email-masked]>
Date: Tue, 07 Apr 2009 15:08:37 +0200

On Tue, 07 Apr 2009 11:16:14 +0200, Anne van Kesteren <[email-masked]>  
wrote:
> Allowing relative URIs hardly seems a "dirty URI" issue. As for the  
> actual "dirty URI" issue, that isn't solved as long as HTTP keeps  
> insisting the header takes an actual URI rather than such a "dirty URI".

Also, it is not just browsers, this affects all kinds of clients that try  
to do something with Web pages: curl, wget, feed finder clients, etc.

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 7 Apr 2009 22:05:42 -0500

> insisting the header takes an actual URI rather than such a

The ways in which implementers fail to the protocol don't need to be formally enumerated in the specification. An informal guide is more than sufficient. An informal guide will also be easier to update than an IETF specification when implementers find creative new ways to ignore standards. A standard is supposed to be the intersection of implementers' agreement, not the union of their mistakes.

- Brian

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 08 Apr 2009 12:05:48 +0200

On Wed, 08 Apr 2009 05:05:42 +0200, Brian Smith <[email-masked]>  
wrote:
> The ways in which implementers fail to the protocol don't need to be  
> formally enumerated in the specification. An informal guide is more than  
> sufficient. An informal guide will also be easier to update than an IETF  
> specification when implementers find creative new ways to ignore  
> standards. A standard is supposed to be the intersection of  
> implementers' agreement, not the union of their mistakes.

But there's agreement among quite a few implementors here. If the  
specification basically has to be ignored by implementors because it is no  
longer fit for deployed content it may be a standard, but also useless.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 8 Apr 2009 21:01:12 +1000

Sorry, I read that too quickly.

Regarding relative URIs -- changing their syntax to allow relative  
URIs would be a breaking change for existing implementations, so at  
first blush it's out of scope for HTTPbis (and indeed anything but  
HTTP/2.0). What's the nature of the problems you're seeing with it  
being an absolute URI?

Regarding the range of content in the Location header itself -- hm.  
The majority of Location headers are produced by HTTP implementations  
themselves (e.g., from /foo to /foo/), and if they're broken, I don't  
think there's going to be a lot of sympathy for working around these  
issues.

OTOH some Location headers are going to be generated by scripts, etc.  
However, the variation in these would likely be wide, unless UAs (not  
just browsers) already implement a predictable, well-known set of  
canonicalisations.  Do they -- for Location?

Also, how widespread is this problem; are we talking about a few  
sites, or most of the Web?

I'm happy to open an issue on this second aspect, but we need more  
information than a pointer to an IRC discussion that mentions two URIs  
that produce bad Location headers. Can you give us more context, please?

On 07/04/2009, at 7:16 PM, Anne van Kesteren wrote:

>> As discussed at the bar bof/informal liaison in SF, this seems to  
>> be more of a URI issue; i.e., HTML and browser vendors need a named

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 08 Apr 2009 13:34:51 +0200

On Wed, 08 Apr 2009 13:01:12 +0200, Mark Nottingham <[email-masked]> wrote:
> Regarding relative URIs -- changing their syntax to allow relative URIs  
> would be a breaking change for existing implementations, so at first  
> blush it's out of scope for HTTPbis (and indeed anything but HTTP/2.0).  
> What's the nature of the problems you're seeing with it being an  
> absolute URI?

In

   http://lists.w3.org/Archives/Public/ietf-http-wg/2009JanMar/0276.html

I pointed to

which says among other things that e.g.

   http://www.usafa.af.mil/
   http://www.flightsimulator.nl/

return Location with a non-absolute URI and are expected to work fine. In  
addition those Location header values also do not match the URI syntax  
because they contain a space and backslash respectively. Still, they work  
fine in Web browsers and curl/wget etc.

So I'm not quite sure what you consider "breaking" here, but it seems to  
me that the current specification requires a breaking change of existing  
implementations.

> Regarding the range of content in the Location header itself -- hm. The  
> majority of Location headers are produced by HTTP implementations  

I'm not asking for sympathy.

> canonicalisations.  Do they -- for Location?

Yes. See e.g. the URL parsing algorithm as documented by e.g. HTML5 and  
the Web Addreses draft from DanC.

> Also, how widespread is this problem; are we talking about a few sites,  
> or most of the Web?

Widespread enough that it was relatively easy to find the sites above. I'd  
expect the problem is quite widespread since relative URIs and  
syntactically invalid URIs work fine and generating them using ASP, PHP,  
etc. is trivial.

> that produce bad Location headers. Can you give us more context, please?

What exactly do you need, more examples?

---

From: Julian Reschke <[email-masked]>
Date: Wed, 08 Apr 2009 13:52:51 +0200

Mark Nottingham wrote:
> ...
 > ...

Indeed.

 > ...

BR, Julian

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 08 Apr 2009 13:55:05 +0200

>> blush it's out of scope for HTTPbis (and indeed anything but HTTP/2.0).

Deferring the issue to HTTP/2.0 does not solve the problem because  
implementations have to implement this for HTTP/1.1 (and likely earlier,  
haven't checked).

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 8 Apr 2009 13:58:35 +0200 (CEST)

> fine in Web browsers and curl/wget etc.

I must admit I was deeply disturbed that day a couple of years ago when I 
realized that our browsers handle spaces in Location: headers by replacing 
them with %20. But I still proceeded and made (lib)curl behave that way as 
well since people expect it to be able to act like the browsers.

The fact that sites all over the world uses relative URIs in Location: headers 
is hardly a secret and I would even say (without any scientific research nor 
proof) that the majority of Location: headers are.

-- 

  / daniel.haxx.se

---

From: Julian Reschke <[email-masked]>
Date: Wed, 08 Apr 2009 13:58:58 +0200

Anne van Kesteren wrote:
> On Wed, 08 Apr 2009 13:52:51 +0200, Julian Reschke 
> <[email-masked]> wrote:
>> Mark Nottingham wrote:
>>> ...
>>> HTTP/2.0).

No, implementations do not have to, according to the spec.

I believe you that browsers currently do, but I'm pretty certain that 
there's lots of code out there that doesn't accept relative URIs (or 
malformed URIs), which would become non-compliant by this change.

So, again, this would be an incompatible change.

BR, Julian

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 08 Apr 2009 14:07:21 +0200

> No, implementations do not have to, according to the spec.

The spec does not matter for this.

> malformed URIs), which would become non-compliant by this change.

Pointers?

> So, again, this would be an incompatible change.

To make HTTP compatible with deployed Web content and actually a useful  
spec to implement against.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 08 Apr 2009 14:13:06 +0200

Anne van Kesteren wrote:
> On Wed, 08 Apr 2009 13:58:58 +0200, Julian Reschke 

If the spec doesn't matter, why are you even interested in changing it?

>> malformed URIs), which would become non-compliant by this change.

None right now.

> spec to implement against.

Again, we can't step out of our charter.

What's needed is *proof* that most implementations (not only browsers!) 
already do this today because they were forced by broken content to 
support it.

BR, Julian

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 08 Apr 2009 14:21:40 +0200

> If the spec doesn't matter, why are you even interested in changing it?

I want it to matter.

> None right now.

This would  be quite useful.

>>  To make HTTP compatible with deployed Web content and actually a  
>> useful spec to implement against.
>

I already told you curl and wget do this and the author of curl just said  
on the list he does this because of broken content.