Subject: [TLS] Encoding of data for Server Name Indication?

From: Brad Hards <[email-masked]>
Date: Wed, 13 Jun 2007 04:41:01 +1000

I am working on some Qt code for TLS (that wraps OpenSSL and other libraries). 
We are planning to support the Server Name Indication extension (assuming 
that the underlying library also supports it), but have come up with an 
issue.

Based on 3280bis, the Certificate commonName and dNSName are supposed to be 
punycode encoded. Is SNI also supposed to be encoded with punycode?

Brad

_______________________________________________
TLS mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tls

---

From: <[email-masked]>
Date: Fri, 15 Jun 2007 13:33:37 +0300

Brad Hards wrote:

> I am working on some Qt code for TLS (that wraps OpenSSL and
> other libraries).  We are planning to support the Server Name
> Indication extension (assuming that the underlying library also
> supports it), but have come up with an issue.
> 
> Based on 3280bis, the Certificate commonName and dNSName are
> supposed to be punycode encoded. Is SNI also supposed to be
> encoded with punycode?

The text in RFC 4366 isn't exactly easy to understand, but the
answer seems to be that both work. The client can either use the
ToASCII operation (which involves more things than just Punycode
encoding, notably Nameprep), or just do Nameprep and send the
resulting string as UTF-8.

Both of these work, because the server does the ToUnicode
operation before comparing the domain name:

   "If the server needs to match the HostName against names that
   contain non-US-ASCII characters, it MUST perform the conversion
   operation described in Section 4 of [IDNA], treating the
   HostName as a "query string" (i.e., the AllowUnassigned flag
   MUST be set)."

(And ToUnicode will remove the Punycode encoding, if used.)

Best regards,
Pasi

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 15 Jun 2007 09:18:31 -0700

>resulting string as UTF-8.

I disagree with Pasi: only ToASCII makes any sense. Having the string 
as non-ASCII makes comparison to the dNSName string impossible 
without doing another conversion.

RFC 4366 could have been a bit clearer. Where it says:
    If the server needs to match the
    HostName against names that contain non-US-ASCII characters, it MUST
    perform the conversion operation described in Section 4 of [IDNA],
    treating the HostName as a "query string" (i.e., the AllowUnassigned
    flag MUST be set).
it would have been better to say at the end of the sentence "and 
processing the labels with ToASCII in step 4".

>    HostName as a "query string" (i.e., the AllowUnassigned flag

There is nothing in that sentence that supports "the server does the 
ToUnicode operation before comparing the domain name". It does not. 
All data in the certificate and in the extension are stored in the 
processed ToASCII format.

--Paul Hoffman, Director
--VPN Consortium

---

From: <[email-masked]>
Date: Mon, 18 Jun 2007 11:19:42 +0300

Paul Hoffman wrote:

> I disagree with Pasi: only ToASCII makes any sense. 

If the field can't contain anything except ASCII, then why does RFC
4366 specify UTF-8 encoding? Especially the requirement to accept
e.g. U+3002 as label separator doesn't make sense if ToASCII is
required...

> Having the string as non-ASCII makes comparison to the dNSName
> string impossible without doing another conversion.

True -- but this is not specific to TLS: if we have one IDN-unaware 
domain name slot and another IDN-aware domain name slot, comparing 
them means some kind of conversion.
 
>     If the server needs to match the HostName against names that 
>     HostName as a "query string" (i.e., the AllowUnassigned
>     flag MUST be set).

There seem to be two sources of confusion here:

First, Section 4 of [IDNA] describes two different conversion
operations (ToASCII and ToUnicode), and it's not totally obvious
which of them is meant here...

Second, "names that contain non-US-ASCII characters" can be interpreted
in two different ways. It's either "character strings that contain 
non-US-ASCII characters" or "character strings that contain only 
US-ASCII characters, but if processed via ToUnicode, would contain 
non-US-ASCII characters".

(With the former interpretation, ToUnicode makes more sense. With
the latter, ToASCII.)

It seems this issue needs to be clarified in the updated extensions
document (probably using RFC3490 terminology of "IDN-unaware"
or "IDN-aware" domain name slot)...

Best regards,
Pasi

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 18 Jun 2007 07:54:41 -0700

At 11:19 AM +0300 6/18/07, <[email-masked]> wrote:
>4366 specify UTF-8 encoding?

I can't say, I wasn't an author of that document. However, ASCII is a 
proper subset of the UTF-8 encoding. If the RFC had specfied UTF-16, 
for example, that would not be the case.

>Especially the requirement to accept

That is completely incorrect. ToASCII is applied only to *labels*, 
not to an entire domain name.

>them means some kind of conversion.

OK, so we agree here.

>which of them is meant here...

Yep. That's why I proposed the change above.

>(With the former interpretation, ToUnicode makes more sense. With

Fully disagree.

>or "IDN-aware" domain name slot)...

That would be best.

So, implementers of RFC 4366: what have you done so far? Encoded with 
ToASCII or ToUnicode?

--Paul Hoffman, Director
--VPN Consortium

---

From: <[email-masked]>
Date: Mon, 18 Jun 2007 18:29:57 +0300

Paul Hoffman wrote:

> >Especially the requirement to accept e.g. U+3002 as label 
> >separator doesn't make sense if ToASCII is required...
> 
> That is completely incorrect. ToASCII is applied only to 
> *labels*, not to an entire domain name.

Yes, but the whole process (going over all the labels) does
include this step:

   5) If ToASCII was applied in step 4 and dots are used as 
      label separators, change all the label separators 
      to U+002E (full stop).

If the client does this, the server can't receive a string
with U+3002 separating labels, so the requirement to accept
it doesn't make sense....

Best regards,
Pasi

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 18 Jun 2007 09:53:45 -0700

>it doesn't make sense....

It's called being liberal in what you accept. There is no reason to 
fail for the other full-stop characters.

--Paul Hoffman, Director
--VPN Consortium

---

From: Peter Sylvester <[email-masked]>
Date: Mon, 18 Jun 2007 18:52:51 +0200

> So, implementers of RFC 4366: what have you done so far? Encoded with 
our code in openssl leaves this to the application.
My code for apache does nothing.
_______________________________________________
TLS mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tls

---

From: Simon Josefsson <[email-masked]>
Date: Mon, 18 Jun 2007 19:05:42 +0200

Peter Sylvester <[email-masked]> writes:

>>
>> So, implementers of RFC 4366: what have you done so far? Encoded
>> with ToASCII or ToUnicode?
>>

It is the same in gnutls, the application can set any character string
and it will be sent unprocessed.

  typedef enum
  {
    GNUTLS_NAME_DNS = 1
  } gnutls_server_name_type_t;

  int gnutls_server_name_set (gnutls_session_t session,
			      gnutls_server_name_type_t type,
			      const void *name, size_t name_length);

/Simon

---

From: Nelson B Bolyard <[email-masked]>
Date: Mon, 18 Jun 2007 12:20:52 -0700

> and it will be sent unprocessed.

Likewise for NSS.  Any UTF8 string (or ASCII, since it's a subset) will
be accepted and sent as-is.

---

From: Mike <[email-masked]>
Date: Mon, 18 Jun 2007 12:38:16 -0700

> be accepted and sent as-is.

My client code also accepts any character string and sends it as
is.  The server does not perform ToASCII before comparing it to
the name in the certificate.  I have since implemented the ToASCII
and ToUnicode functions, so it wouldn't be difficult to change;
however, the server name extension has been around for 4 years, so
sending ToUnicode strings might break existing servers.

Mike

---

From: Brad Hards <[email-masked]>
Date: Tue, 19 Jun 2007 09:35:09 +1000

> sending ToUnicode strings might break existing servers.
OK, now I'm confused.

If the server is using an internationalised name and the name in the 
certificate is encoded (which I understand it needs to be), then won't this 
always fail to match?

I'm not so much concerned with what the libraries do - more in what the 
application code should do.

Brad

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 18 Jun 2007 16:57:25 -0700

>certificate is encoded (which I understand it needs to be), then won't this

It is not clear what you meant by "encoded". IDNs in PKIX 
certificates are given in their ToASCII format. See 
draft-ietf-pkix-rfc3280bis-08.txt, section 7.

--Paul Hoffman, Director
--VPN Consortium

---

From: Brad Hards <[email-masked]>
Date: Tue, 19 Jun 2007 10:27:34 +1000

On Tuesday 19 June 2007 09:57, Paul Hoffman wrote:
> >certificate is encoded (which I understand it needs to be), then won't
> > this always fail to match?
>
Sorry, I'll try to be clearer.

If I have an internationalised domain name, then the certificate for that 
domain will have the domain name in a (basically) ACE encoded format. The 
human user on the client side is presumably going to use UTF8 or perhaps 
UTF16 to specify the host they want to connect to.

Which part of the system is responsible for making sure that the host name and 
the server certificate match?

Brad

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 18 Jun 2007 17:39:20 -0700

At 10:27 AM +1000 6/19/07, Brad Hards wrote:
>domain will have the domain name in a (basically) ACE encoded format.

Correct.

>The

Correct, but unpredictably so. They type something into a text field 
in the chrome web browser. They click on a link. They run a program 
that has host names in it. And so on.

>the server certificate match?

The TLS implementation.

--Paul Hoffman, Director
--VPN Consortium

---

From: Simon Josefsson <[email-masked]>
Date: Tue, 19 Jun 2007 11:02:52 +0200

>>the server certificate match?

I'd be happy to change my implementation if there is consensus about
what RFC 4366 is intended to imply, or if an updated document is
published.  I haven't understood yet whether there is consensus about
what the text in RFC 4366 really should imply in practice?

One approach is for the server to apply ToASCII on each label of both
what's in the certificate and what's in the SNI, concatenate them using
ASCII '.' and compare the result using strcmp() (in the C locale..).

/Simon

---

From: <[email-masked]>
Date: Tue, 19 Jun 2007 12:57:20 +0300

> My code for apache does nothing.

This seems to violate a MUST in RFC 4366, no matter which 
way you interprete it.

And re-reading RFC 3490, I'm starting to (partly) agree with Paul
here: the requirement for the server does indeed refer to ToASCII,
not ToUnicode (as I thought earlier).

In other words: when the server compares server_name with something
else, it MUST process it with the "conversion operation (with
ToASCII)" from Section 4.  It might also need to process the
"something else" the same way (depending on where it came from); 
and finally it does a case-insensitive ASCII comparison.

But the result for the client seems to remain the same: it 
can either send ToASCII-encoded name, or just Nameprep+UTF-8.
Right?

(But if the client also does comparisons with the name, the
comparisons have to be done with ToASCII'd-forms.)

Best regards,
Pasi

---

From: Peter Sylvester <[email-masked]>
Date: Tue, 19 Jun 2007 13:02:33 +0200

> This seems to violate a MUST in RFC 4366, no matter which 
Probably, since I don't known whether the apache bases
code for comparisons for server names do any kind of
conversion. Probably apache needs a patch but not openssl.

---

From: Mike <[email-masked]>
Date: Tue, 19 Jun 2007 06:40:59 -0700

>> the name in the certificate.

After re-reading RFC 4366, I think that I should be using ToASCII
in the server.  The client can probably still send whatever it is
given, by requiring that the application provide a valid IDN in
UTF-8.

Mike

---

From: Martin Rex <[email-masked]>
Date: Tue, 19 Jun 2007 16:24:46 +0200 (MEST)

> >the server certificate match?

I violently disagree.

The TLS implementation has nothing to do with the concept of
Endpoint identification employed by HTTP over SSL/TLS.
This endpoint identification is the task of some other middleware
layer above TLS.

But aside from this, I wonder whether it is appropriate
to compare the name in the certificate to the hostname as
it is passed to a traditional implementation of gethostbyname().

-Martin

---

From: Peter Sylvester <[email-masked]>
Date: Tue, 19 Jun 2007 17:32:13 +0200

> This endpoint identification is the task of some other middleware
I share your disagreement.
The TLS implementation might not even know to host it is
actually talking for example if it is implemented like filter where you
just put a connected network file descriptor under it.
> it is passed to a traditional implementation of gethostbyname().
I think your question is somewhat unprecise. :-)

---

From: Nelson B Bolyard <[email-masked]>
Date: Tue, 19 Jun 2007 14:54:08 -0700

Martin Rex wrote:
>>> The human user on the client side is presumably going to use UTF8 or
>>> perhaps UTF16 to specify the host they want to connect to.
>>
>>> Which part of the system is responsible for making sure that the host
>>> name and the server certificate match?

> I violently disagree.

I also disagree.  The TLS RFC disagree, too, IMO.

> This endpoint identification is the task of some other middleware

RFC 2246 says:

   The TLS standard, however, does not specify how
   protocols add security with TLS; the decisions on [...]
   how to interpret the authentication certificates
   exchanged are left up to the judgment of the designers and
   implementors of protocols which run on top of TLS.
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RFC 2246, in section D.3, and RFC 4346, in section d.2, say

   Implementations are responsible for verifying the integrity of
   certificates and should generally support certificate revocation
   messages.

But they say nothing about how the end point decides if the certificate
identifies the desired peer end point, or not.  And clearly this depends
on the protocol running atop TLS.

The widely used technique of comparing host names with fields in the
cert is defined in section 3.1 of Eric's Informational RFC 2818, which
defines https.  But that RFC also explains that other techniques may
be appropriate in some cases, the and explicitly provides:

   If the client has external information as to the expected identity of
   the server, the hostname check MAY be omitted.

Speaking for one implementation, NSS: we provide a function that
implements the technique of RFC 2818, as a convenience to higher layer
applications, such as https, to decide if a cert is or is not the
intended one, after it has been verified.  Applications are free to use
it, or not, as they see fit.

/Nelson

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 19 Jun 2007 15:23:48 -0700

At Tue, 19 Jun 2007 14:54:08 -0700,
> I also disagree.  The TLS RFC disagree, too, IMO.

I disagree as well. 

-Ekr

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 19 Jun 2007 15:37:07 -0700

>I disagree as well.

OK, I see a pattern. :-)

That still doesn't answer what should be done and how. Given the 
earlier confusion, I think an update of at least SNI is in order.

--Paul Hoffman, Director
--VPN Consortium