Subject: lower casing host names

From: Daniel Stenberg <[email-masked]>
Date: Fri, 30 Dec 2011 16:18:08 +0100 (CET)

Hi friends,

I've walked into an issue I just wanted to bring here for attention and 
possibly some feedback.

A curl user noticed that when he reqeusted a URI with a mixed case host name, 
the site would redirect to the same host name with all letters in lower case. 
(And since curl would re-use the same connection and the same Host: header it 
resent the same mixed case request that again gets redirected... and it turns 
into a nice loop.)

I consider treating the host name differently only based on different casing a 
protocol violation of the target server/software in question, but curious 
about this I tried out chrome and firefox to see how they handle this case.

It turns out both browsers always unconditionally lower case the host name in 
URIs so they never send HTTP requests with mixed case.

Why do they do this? Is this behavior of treating names differently based on
case common? If so, should httpbis mention it?

Unfortunately this will now also push curl towards this behavior. We haven't 
yet decided how to act, but forcibly lowercasing made it work fine against 
this particular host where this issue arose...

-- 

  / daniel.haxx.se

---

From: Dale Anderson <[email-masked]>
Date: Fri, 30 Dec 2011 09:00:58 -0800

> It turns out both browsers always unconditionally lower case the host name in URIs so they never send HTTP requests with mixed case.

I seen common browsers also "treat" the URI path (percent-encode it,
most notably). I appreciate that curl is a li'l more literal for HTTP
testing work. I hope curl and libcurl would stay that way and leave
standardizing case optional if anything.

I only used the python bindings, seems like it would fall under one of
those 'setopt' calls to change the default for whether it's
standardizing cases to lowercase before constructing the request-line
and host header, maybe independent option for request-line and
host-header case lowering.

> case common? If so, should httpbis mention it?

They being HTTP daemons and applications:
 - they are just used to being fed mechanically-softed lowercase
strings from browsers
 - they weren't tested what happens when that varies
 - it was cheap and easy to do a quick strcmp() call instead of
something case-insensitive
 - shout out to the implementation that its redirect should be
comparing hostnames on a case insensitive basis, else
application-delivery/firewall may have to step in to mitigate

There can be various exploits and bugs along these lines!

Dale Anderson

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 30 Dec 2011 10:40:20 -0800

On Dec 30, 2011, at 7:18 AM, Daniel Stenberg wrote:

> Hi friends,
> 
> I've walked into an issue I just wanted to bring here for attention and possibly some feedback.
> 
> A curl user noticed that when he reqeusted a URI with a mixed case host name, the site would redirect to the same host name with all letters in lower case. (And since curl would re-use the same connection and the same Host: header it resent the same mixed case request that again gets redirected... and it turns into a nice loop.)
> 
> I consider treating the host name differently only based on different casing a protocol violation of the target server/software in question, but curious about this I tried out chrome and firefox to see how they handle this case.
> 

That's new, I guess.  Mixed case used to be common.

> case common? If so, should httpbis mention it?

It is an attempt to reduce the number of resource aliases and thus improve
cache reuse, SEO, etc.  It is a common use case for mod_rewrite.

> Unfortunately this will now also push curl towards this behavior. We haven't yet decided how to act, but forcibly lowercasing made it work fine against this particular host where this issue arose...

Why don't you send the correct Host header field on the second request?
Host is an attribute of the request URI, not the connection, so I'd have
to say the behavior of curl in sending the same Host header field after
the redirect from the client is wrong.

I would not do the lowercasing that those browsers do, since it will run
afoul of vanity host sites that do want to insist on mixed-cased domains
and would interfere with using curl as a testing tool.

....Roy

---

From: Adrien de Croy <[email-masked]>
Date: Sat, 31 Dec 2011 07:44:53 +1300

all domain names are case insensitive according to the DNS specs RFC 
1034, 1035 etc.

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com
WinGate 7 is released! - http://www.wingate.com/getlatest/

---

From: Adam Barth <[email-masked]>
Date: Fri, 30 Dec 2011 16:14:31 -0800

On Fri, Dec 30, 2011 at 7:18 AM, Daniel Stenberg <[email-masked]> wrote:
> A curl user noticed that when he reqeusted a URI with a mixed case host
> name, the site would redirect to the same host name with all letters in
> lower case. (And since curl would re-use the same connection and the same
> Host: header it resent the same mixed case request that again gets
> redirected... and it turns into a nice loop.)
>
> I consider treating the host name differently only based on different casing
> a protocol violation of the target server/software in question, but curious
> It turns out both browsers always unconditionally lower case the host name
> in URIs so they never send HTTP requests with mixed case.
>
> Why do they do this?

IE has this behavior as well.  Safari does not, but it's definitely
the odd-browser out in this regard.  I either wrote or reviewed a
patch that canonicalized host names to lower case for Safari, but it
ran into a problem with safari-extension URLs (don't ask).

To answer your specific question, Chrome has this behavior because IE
does.  Chrome's URL process behavior is modeled as closely as possible
after IE (except in cases where IE's behavior is certifiably insane,
in which case it's modeled after Firefox).  In this case, because both
IE and Firefox agree, choosing this behavior was an easy call.  This
behavior is collectively stable and therefore unlikely to change in
the future.

> Is this behavior of treating names differently based on

It's likely to work fine for the vast majority of web sites on the
Internet (and in most intranets).  I suspect it's not really something
for HTTPbis to concern itself with, however.  The correct place for
this requirement is in the as-yet-non-existent URL specification
because host name canonicalization is visible in many places beyond
just the HTTP Host header.

Adam

---

From: Daniel Stenberg <[email-masked]>
Date: Sat, 31 Dec 2011 10:27:44 +0100 (CET)

On Fri, 30 Dec 2011, Adam Barth wrote:

> I suspect it's not really something for HTTPbis to concern itself with, 
> however.

I am concerned because when the three most used browsers do this it means lots 
of people will test their sites and servers with them and everything will be 
fine and dandy even when they lazily don't do the right thing - and this will 
force other HTTP implementors to go this way too sooner or later. Pretty much 
exactly how we ended up with how cookies work.

These three browsers then apparently do something with HTTP that isn't 
mentioned (or referred to) in httpbis. Such hidden knowledge isn't good for a 
protocol spec imho.

-- 

/ daniel.haxx.se

---

From: "Anne van Kesteren" <[email-masked]>
Date: Sat, 31 Dec 2011 10:45:56 +0100

On Sat, 31 Dec 2011 10:27:44 +0100, Daniel Stenberg <[email-masked]> wrote:
> I am concerned because when the three most used browsers do this it  
> means lots of people will test their sites and servers with them and  
> everything will be fine and dandy even when they lazily don't do the  
> right thing - and this will force other HTTP implementors to go this way  
> too sooner or later. Pretty much exactly how we ended up with how  
> cookies work.
>
> mentioned (or referred to) in httpbis. Such hidden knowledge isn't good  
> for a protocol spec imho.

You can (and I think we have) run into the same kind of issues with order  
of HTTP headers, HTTP header casing, etc. HTTP isn't very conservative in  
what clients have to do, while it probably should be given that there are  
only so many clients versus many many servers. Although not protocols, we  
learned the same lessons with e.g. CSS, the DOM, and HTML. (Also known as  
Postel's law.)

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: Adam Barth <[email-masked]>
Date: Sat, 31 Dec 2011 10:19:41 -0800

On Sat, Dec 31, 2011 at 1:45 AM, Anne van Kesteren <[email-masked]> wrote:
>> I am concerned because when the three most used browsers do this it means
>> lots of people will test their sites and servers with them and everything
>> will be fine and dandy even when they lazily don't do the right thing - and
>> this will force other HTTP implementors to go this way too sooner or later.
>> Pretty much exactly how we ended up with how cookies work.
>>
>> mentioned (or referred to) in httpbis. Such hidden knowledge isn't good for
>> a protocol spec imho.
>
> You can (and I think we have) run into the same kind of issues with order of
> HTTP headers, HTTP header casing, etc. HTTP isn't very conservative in what
> clients have to do, while it probably should be given that there are only so
> many clients versus many many servers. Although not protocols, we learned
> the same lessons with e.g. CSS, the DOM, and HTML. (Also known as Postel's
> law.)

I suspect we'll eventually want to write a document that contains all
this "hidden knowledge."  I suspect HTTPbis will call that document a
"browser profile" of HTTP, which seems fine.  Not all of these
constraints apply in every situation in which HTTP is used.

Adam

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 31 Dec 2011 11:38:24 -0800

> constraints apply in every situation in which HTTP is used.

Now that we know this isn't just a one-browser issue, we should come up
with a suitable description of it for the Host header field.  Note that
this does break some expectations by server owners (namely, that their
trademarks won't remain in the proper capitalization when used in a URI
on the Web).  I am fine with that, provided what we document is an
accurate description of what some user agents do (and preferably why).

....Roy