Subject: SNI vs Host: and a trailing dot

From: Daniel Stenberg <[email-masked]>
Date: Wed, 16 Mar 2016 12:17:29 +0100 (CET)

Heya HTTP peeps!

Input "HTTPS://example.com./". A URI using a hostname with a trailing dot. How 
to behave?

1. RFC 6066 section 3 says the hostname in SNI string should be sent "without 
a trailing dot"

2. RFC 7230 secion 5.4 says about Host: "MUST send a field-value for Host that 
is identical to that authority component" - which then would include the 
trailing dot.

Following these specs, we should send different names in SNI vs Host when a 
trailing dot is used. I don't like that as I suspect HTTPS servers will use 
the SNI field to serve contents while HTTP servers can only use Host: header 
and thus this will make them act differently. I also suspect some servers 
won't like getting different names in there, but I don't have any proof of 
that.

Right now, only curl[*] seems to have been stripping the dot from the SNI name 
according to this wget bug report:
  https://savannah.gnu.org/bugs/?47408#discussion

Qt is about to fix it: https://bugreports.qt.io/browse/QTBUG-51821

And here's the Firefox bug for trailing dots in SNI fields: 
https://bugzilla.mozilla.org/show_bug.cgi?id=1008120

... but it looks like curl is also the only one[*] that strips the dot from 
the Host: header. So curl violates RFC 7230, and most of the others seem to 
violate RFC 6066, unless I'm mistaking.

I think it would benefit us all to get a common view on how to clear this up!

Thoughts?

[*] = out of the ones I checked. I only checked a small selection of clients 
but the point is more that it isn't a general agreement.

-- 

  / daniel.haxx.se

---

From: Cory Benfield <[email-masked]>
Date: Wed, 16 Mar 2016 11:42:52 +0000

> On 16 Mar 2016, at 11:17, Daniel Stenberg <[email-masked]> wrote:
> 
> Following these specs, we should send different names in SNI vs Host when a trailing dot is used. I don't like that as I suspect HTTPS servers will use the SNI field to serve contents while HTTP servers can only use Host: header and thus this will make them act differently. I also suspect some servers won't like getting different names in there, but I don't have any proof of that.
> 
> Right now, only curl[*] seems to have been stripping the dot from the SNI name according to this wget bug report:
> And here's the Firefox bug for trailing dots in SNI fields: https://bugzilla.mozilla.org/show_bug.cgi?id=1008120
> 
> ... but it looks like curl is also the only one[*] that strips the dot from the Host: header. So curl violates RFC 7230, and most of the others seem to violate RFC 6066, unless I'm mistaking.
> 

I’d like a good answer on this. Currently requests mishandles trailing dots in URLs (I think our users have spotted this though and simply don’t use them), and if I’m going to go to fix that bug I’d like to fix it in a way that has some consensus! (Rather than doing what most of us do in this situation, which is to shrug our shoulders and do what curl does.)

FWIW, our data point is that we leave the trailing dot on the authority in all cases: SNI, Host header, and TLS hostname validation (which is clearly wrong but fails closed so could be a lot worse).

Cory

---

From: Michael Sweet <[email-masked]>
Date: Wed, 16 Mar 2016 08:09:36 -0400

FWIW, CUPS has traditionally stripped the trailing dot from both since most printers (and web sites, for that matter) have difficulty handling "example.com."

> On Mar 16, 2016, at 7:42 AM, Cory Benfield <[email-masked]> wrote:
> 
> 

_________________________________________________________
Michael Sweet, Senior Printing System Engineer

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 17 Mar 2016 01:44:24 +1300

> FWIW, CUPS has traditionally stripped the trailing dot from both since most printers (and web sites, for that matter) have difficulty handling "example.com."

FWIW; Squid likewise does that as well.

IIRC we determined that the trailing dot syntax was an outcome of people
partially understanding the DNS specifications. Those DNS specs talk
about using the trailing dot to terminate the domain labels. But on
close inspection it is only supposed to be used in the wire-format for
DNS packets. Intermediate representations like HTTP messages or TLS SNI
are expected to have no trailing dot for valid FQDN.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 16 Mar 2016 14:02:10 +0100

Hi guys,

> are expected to have no trailing dot for valid FQDN.

Not exactly because you have the problem when you need to differenciate
host names that belong to your local domain and other ones. For example
you could have a host called "www.example.com" on your local domain, and
if you want to be sure to use the public www.example.com and not
www.example.com.my.local.domain, the only way is to add the trailing dot.

I remember having been in this exact situation many years ago by which
I couldn't connect to my default gateway's web interface until I realized
that the name "gw" that I was using on my local network was first resolved
as "gw.work.local" which was my company's groupware server and it couldn't
be accessed from where I was located (hence my belief that the device did
not respond). Simply passing "http://gw./" solved the issue for me.

However I have no idea where in the chain this dot needs to be trimmed,
but it definitely has a use case for the end user and in HTTP URIs (though
not frequent).

Cheers,
Willy

---

From: Michael Sweet <[email-masked]>
Date: Wed, 16 Mar 2016 09:15:38 -0400

One of the saddest workarounds present in CUPS is a hardcoded mapping of "localhost" to 127.0.0.1 and ::1 because "localhost.<defaultdomain>" often is assigned to the AD server leading to all sorts of wonderful problems...

> On Mar 16, 2016, at 9:02 AM, Willy Tarreau <[email-masked]> wrote:
> 
> Hi guys,
>

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 17 Mar 2016 02:20:12 +1300

> www.example.com.my.local.domain, the only way is to add the trailing dot.

Currently .local exists to resolve the problem.

Previously to that the resolv.conf ndots and domain/search features were
the way to do it.

And previously (sort of in parallel as well) there was the DNS specified
recommendation/algorithm for ordering a local query before global query
(or was it vice versa?). That got applied for each of the resolv.conf
lookups. Which you seem to be describing below;

> but it definitely has a use case for the end user and in HTTP URIs (though

IIRC; the DNS specs said tolerant software could expect it occasionally
in users input and should trim. Which effectively means during input
validation for any software, via any input means.

Cheers
Amos

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 17 Mar 2016 10:02:17 +1100

> On 16 Mar 2016, at 10:17 PM, Daniel Stenberg <[email-masked]> wrote:
> 
> Heya HTTP peeps!
> 
> Input "HTTPS://example.com./". A URI using a hostname with a trailing dot. How to behave?
> 
> 1. RFC 6066 section 3 says the hostname in SNI string should be sent "without a trailing dot"
> 
> 2. RFC 7230 secion 5.4 says about Host: "MUST send a field-value for Host that is identical to that authority component" - which then would include the trailing dot.
> 
> Following these specs, we should send different names in SNI vs Host when a trailing dot is used. I don't like that as I suspect HTTPS servers will use the SNI field to serve contents

They shouldn't be doing that (if indeed they do); SNI is only for selecting the certificate, not anything to do with what happens inside HTTP.

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 16 Mar 2016 16:10:35 -0700

> On Mar 16, 2016, at 6:02 AM, Willy Tarreau <[email-masked]> wrote:
> 
> Hi guys,
> 

Yep, that's right.

> not respond). Simply passing "http://gw./" solved the issue for me.

I had the same issue at UCI when the College of Medicine decided to name their
primary gateway "com", as in "{host}.com.uci.edu".  It was a short conversation.
Now we are getting inundated by new ICANN TLDs, so expect more of that soon.
It may become common for some sites to need the trailing dot just to ensure
that HTTP routing works.

> but it definitely has a use case for the end user and in HTTP URIs (though

Personally, I never trim the dot outside of DNS itself, since it has to be
presumed that someone added it for a reason (even if that reason is
nefarious -- the distinction cannot be ignored). Since both clients and
proxies are subject to local network aliases, the dot must be retained in
HTTP's authority and Host.

....Roy

---

From: Daniel Stenberg <[email-masked]>
Date: Thu, 17 Mar 2016 00:15:35 +0100 (CET)

On Thu, 17 Mar 2016, Mark Nottingham wrote:

>> I suspect HTTPS servers will use the SNI field to serve contents
>
> They shouldn't be doing that (if indeed they do); SNI is only for selecting 
> the certificate, not anything to do with what happens inside HTTP.

Right, I wrote that part too quick without thinking properly. Sorry. Thanks 
for clearing that up!

I've since tested a bunch of random popular HTTPS sites by adding a dot to the 
host name in the Host: header (while keeping it out of the SNI field) and 
quite clearly there's a non-zero amount of servers that deliver completely 
different headers/contents than if the header is sent without the dot.

-- 

  / daniel.haxx.se

---

From: Mike Bishop <[email-masked]>
Date: Wed, 16 Mar 2016 23:27:25 +0000

In fairness, RFC 6066 also says "If the server_name is established in the TLS session handshake, the client SHOULD NOT attempt to request a different server name at the application layer." HTTP.sys prior to Windows 10/Server 2016 enforced this by refusing (400) requests for other server names.

I'm not sure what our behavior is down level if the difference between the two is only a trailing dot. I'll ask.

-----Original Message-----
From: Mark Nottingham [mailto:[email-masked]] 
Sent: Wednesday, March 16, 2016 4:02 PM
To: Daniel Stenberg <[email-masked]>
Cc: HTTP Working Group <[email-masked]>
Subject: Re: SNI vs Host: and a trailing dot

Cheers,

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 16 Mar 2016 23:29:02 +0000

I don't know if there is a good solution to this.

Maybe browsers should always resolve the difference before submitting to 
a proxy or server so that the server or proxy doesn't need to 
second-guess the client.

We had a version of WinGate that had dots on the end of all 
fully-qualified names.  In an HTTP proxy usually the vast majority of 
names are fully qualified, BUT the vast majority of anchors in web pages 
do not have a dot.

I personally would like to see the end of dot-terminated names in http.

Maybe it would be better to treat any name with more than 1 token to be 
fully qualified, and thereby force people to be reasonable about how 
they name their local servers.

Otherwise we risk breaking a lot of stuff to "fix" some minor edge 
cases.

------ Original Message ------
From: "Amos Jeffries" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 17/03/2016 2:20:12 a.m.
Subject: Re: SNI vs Host: and a trailing dot

>>>>  FWIW, CUPS has traditionally stripped the trailing dot from both 
>>>>since most printers (and web sites, for that matter) have difficulty 
>>>>handling "example.com."
>>>>
>>>
>>>

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 17 Mar 2016 05:25:39 +0100

On Thu, Mar 17, 2016 at 12:15:35AM +0100, Daniel Stenberg wrote:
> >They shouldn't be doing that (if indeed they do); SNI is only for
> >selecting the certificate, not anything to do with what happens inside
> >HTTP.
> 

No, you were right to raise this point, because I spend a lot of time
explaining people that HTTP routing has to be done on the Host header
exclusively and that they must never use SNI for this. The only example
I find to justify this is to say that older clients may not send SNI,
but as you can guess this is a weak argument. Now with this trailing
dot issue and the missing port that Xiaoyin mentionned, it's clear
that SNI and Host may differ and that's a good argument.

> I've since tested a bunch of random popular HTTPS sites by adding a dot to
> the host name in the Host: header (while keeping it out of the SNI field)
> and quite clearly there's a non-zero amount of servers that deliver
> completely different headers/contents than if the header is sent without the
> dot.

I'm not surprized, it's very common to match an exact string on the
server side. A lot of load-balancing configurations I'm seeing are
even case-sensitive just because people don't know the host part is
case-insensitive.

Willy