Subject: Re: [TLS] ALPN concerns

From: Peter Gutmann <[email-masked]>
Date: Wed, 6 Nov 2013 06:52:42 +0000

Brian Smith <[email-masked]> writes:

>I am very concerned about the issues that they've run into where many web
>servers are failing to handshake when the ClientHello message is larger than
>255 bytes.

I'm curious as to how something like this could come about, is anyone familiar
with the code base for something that does this?  Is there actually code out
there that explicitly checks:

  if( sizeof( client_handshake ) > 255 )
    return( -1 );

and if so, why?

Peter.

---

From: John Mattsson <[email-masked]>
Date: Wed, 6 Nov 2013 16:06:03 +0000

Isn't the most probable explanation that they use an unsigned 8-bit int to store the length of the ClientHello message?

Then they might check that the length actually fits (as your example code), get some overflow exception, or just succeeding in discarding the most significant bits in the length.

/John

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Peter Gutmann
Sent: den 6 november 2013 07:53
To: <[email-masked]>
Subject: Re: [TLS] ALPN concerns

Brian Smith <[email-masked]> writes:

>I am very concerned about the issues that they've run into where many 
>web servers are failing to handshake when the ClientHello message is 
>larger than
>255 bytes.

I'm curious as to how something like this could come about, is anyone familiar with the code base for something that does this?  Is there actually code out there that explicitly checks:

and if so, why?

Peter.
_______________________________________________
TLS mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tls

---

From: Yoav Nir <[email-masked]>
Date: Wed, 6 Nov 2013 16:10:00 +0000

>  if( sizeof( client_handshake ) > 255 )

Based on no actual knowledge, maybe it's part of some anti-DoS scheme.

Big ClientHello --> Weird --> Attack

---

From: Xiaoyong Wu <[email-masked]>
Date: Wed, 6 Nov 2013 18:00:05 +0000

It is a little bit more calculation than that and related to some historic reasons, aka SSLv2.

For SSL records, the SSLv3 and TLS ClientHello headers are as follows:

| 22 | version major | version minor | length high bits | length low bits |

If this is interpreted as an SSLv2 header, it will be considered as a 3 byte header:
| v2 header b0 | v2 header b1 | v2 header b2 | message type | 

The value for Client Hello message type is SSLV2_MT_CLIENTHELLO which is 1.
When there is an SSLv3/TLS client-hello of length 256 - 511 bytes, this is ambiguous as "message type" is 1 or it is the "length high bits" to be 1.

Out implementation before the patch was to prefer SSLv2 and thus the issue.

As I am explaining this in detail, I would say that another work around on this would be making a client hello that exceeds 512 in length.

-Xiaoyong

> Sent: Tuesday, November 05, 2013 10:53 PM
> To: <[email-masked]>

---

From: Adam Langley <[email-masked]>
Date: Wed, 6 Nov 2013 13:03:24 -0500

> As I am explaining this in detail, I would say that another work around on this would be making a client hello that exceeds 512 in length.

^^^ Holy crap. I wish I had known that sooner. That might solve the issue.

Cheers

AGL

---

From: Yoav Nir <[email-masked]>
Date: Wed, 6 Nov 2013 18:10:35 +0000

> ^^^ Holy crap. I wish I had known that sooner. That might solve the issue.

Time to standardize the "jpeg-of-cat" extension for TLS.

---

From: Dr Stephen Henson <[email-masked]>
Date: Wed, 06 Nov 2013 18:12:55 +0000

> ^^^ Holy crap. I wish I had known that sooner. That might solve the issue.

Just did a quick test with OpenSSL on a couple of known "hang" machines. Seems
to work.

Steve.
-- 
Dr Stephen N. Henson.
Core developer of the   OpenSSL project: http://www.openssl.org/
Freelance consultant see: http://www.drh-consultancy.co.uk/
Email: [email-masked], PGP key: via homepage.

---

From: Nico Williams <[email-masked]>
Date: Wed, 6 Nov 2013 14:18:56 -0600

> Just did a quick test with OpenSSL on a couple of known "hang" machines. Seems

I'm not adding much, but: that sound we hear is the collective sigh of
relief of the TLS WG list!

This will also help with TLSv1.3, assuming we get there.

---

From: Yutaka OIWA <[email-masked]>
Date: Thu, 7 Nov 2013 05:50:21 +0900

> | v2 header b0 | v2 header b1 | v2 header b2 | message type |

Hmm, if the message is 256-byte TLS 1.2 Handshake message,
the first five bytes will be 0x 22.03.03.01.00.
In SSLv2 context, it means a beginning of a handshake message
containing 0x2203 (8707) bytes with 0x03 (3) bytes of padding.

The V2-aware servers are just waiting for 8000+ more
bytes to arrive, describing why they seems to be silence...

-- 
Yutaka OIWA, Ph.D.                 Leader, System Life-cycle Research Group
                               Research Institute for Secure Systems (RISEC)
     National Institute of Advanced Industrial Science and Technology (AIST)
                       Mail addresses: <[email-masked]>, <[email-masked]>
OpenPGP: id[440546B5] fp[7C9F 723A 7559 3246 229D  3139 8677 9BD2 4405 46B5]

---

From: Dr Stephen Henson <[email-masked]>
Date: Wed, 06 Nov 2013 20:56:12 +0000

> Just did a quick test with OpenSSL on a couple of known "hang" machines. Seems

I added this to the OpenSSL master branch in case anyone wants to test it.

If enabled it uses a dummy extension[1] to pad the ClientHello with enough
zeroes to make it long enough.

Steve.
1. Provisional name Workaround TLS Filler aka WTF ;-)
-- 
Dr Stephen N. Henson.
Core developer of the   OpenSSL project: http://www.openssl.org/
Freelance consultant see: http://www.drh-consultancy.co.uk/
Email: [email-masked], PGP key: via homepage.

---

From: Adam Langley <[email-masked]>
Date: Wed, 6 Nov 2013 16:00:44 -0500

> zeroes to make it long enough.

Likewise for Chrome: https://codereview.chromium.org/62103003

Cheers

AGL

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 6 Nov 2013 13:23:27 -0800

For whatever it’s worth — thank you (personally and F5) for being open about it. You may have just saved us a LOT of pain / gnashing of teeth.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Wan-Teh Chang <[email-masked]>
Date: Wed, 6 Nov 2013 23:27:26 -0800

> As I am explaining this in detail, I would say that another work around on this would be making a client hello that exceeds 512 in length.

Hi Xiaoyong,

Thank you very much for explaining the bug and suggesting a workaround.

Could you also explain a related bug in F5 BIG-IP? This bug is
described in the OpenSSL ChangeLog as follows:

   *) Workarounds for some broken servers that "hang" if a client hello
     record length exceeds 255 bytes.

     1. Do not use record version number > TLS 1.0 in initial client
        hello: some (but not all) hanging servers will now work.
     ...
     [Steve Henson]

To work around this F5 BIG-IP bug, the third byte in a TLS ClientHello
record (which is the protocol minor version) needs to be 0 or 1 and
cannot be 2 or 3. Is this bug also caused by interpreting the record
as an SSLv2 ClientHello? In SSLv2 if the first byte of a record is 22
(0x16), then the third byte is PADDING.

Thanks,
Wan-Teh

---

From: Wan-Teh Chang <[email-masked]>
Date: Wed, 6 Nov 2013 23:30:13 -0800

> Likewise for Chrome: https://codereview.chromium.org/62103003

It would be nice if we could all use the same extension number for
this dummy extension.

Wan-Teh

---

From: Xiaoyong Wu <[email-masked]>
Date: Thu, 7 Nov 2013 08:18:54 +0000

The work around I proposed should work in either cases. 

Some of our earlier releases do not support TLS 1.1 and 1.2. During the development of the patch, we mistakenly only fixed the issue explicitly for SSL 3.0/TLS 1.0 (aka, preferring these 2 versions instead of SSLv2 and not the others) on these releases. This is the reason why using SSL 3.0/TLS 1.0 helped some times.

-Xiaoyong

> From: Wan-Teh Chang [mailto:[email-masked]]
> Sent: Wednesday, November 06, 2013 11:27 PM
> To: Xiaoyong Wu
> Cc: Peter Gutmann; <[email-masked]>
> > | 22 | version major | version minor | length high bits | length low
> > | bits |
> >
> > When there is an SSLv3/TLS client-hello of length 256 - 511 bytes, this is
> ambiguous as "message type" is 1 or it is the "length high bits" to be 1.
> >
> > As I am explaining this in detail, I would say that another work around on
> this would be making a client hello that exceeds 512 in length.
> 
> Hi Xiaoyong,
> 
> Could you also explain a related bug in F5 BIG-IP? This bug is described in
> the OpenSSL ChangeLog as follows:
> 
> To work around this F5 BIG-IP bug, the third byte in a TLS ClientHello record
> (which is the protocol minor version) needs to be 0 or 1 and cannot be 2 or 3.
> Is this bug also caused by interpreting the record as an SSLv2 ClientHello? In
> SSLv2 if the first byte of a record is 22 (0x16), then the third byte is
> PADDING.
> 
> Thanks,
> Wan-Teh

---

From: Brian Smith <[email-masked]>
Date: Mon, 18 Nov 2013 21:58:56 -0800

> As I am explaining this in detail, I would say that another work around on this would be making a client hello that exceeds 512 in length.

Xiaoyong, thank you very much for sharing that with us! That is
incredibly useful.

In the thread "Final nail in the coffin for cleartext SNI/ALPN in TLS
1.3," Yoav Nir <[email-masked]> wrote:
> I think it would actually be worse than that. Browsers don't want
> reconnect fallback. So whatever TLS 1.3 looks like, the ClientHello
> that browsers send will be such that it can be accepted by at least
> TLS 1.0 servers (and there are plenty of those around). And as
> others have mentioned, SNI is needed in many cases. Without it,
> https://www.gmail.com will give you a certificate error.
>
> So you might have a clientHello that leads to super-duper TLS 1.3,
> but to avoid the need for fallback, you include the SNI (and
> presumably ALPN) in the ClientHello.

This is exactly right for SNI. Even if we define a new replacement for
SNI, browsers will still be sending SNI in the clear for the next 10
years. We should still try to work toward a long-term solution to that
problem, but it isn't something that can be solved in the short term.

During the TLS WG session at IETF88, somebody commented that TLS 1.3
is supposed to solve the problem of encrypting information in the
ClientHello, so the plaintext vs. cleartext issue shouldn't be a
concern for ALPN. But, for the same reason we can't have any near-term
solution for encrypting SNI in the ClientHello, we won't have any
near-term solution for encrypting ALPN in the ClientHello that would
meet browsers needs.

Again, I want to emphasize that Firefox ALREADY solves this problem,
even without TLS 1.3, by implementing NPN and avoiding ALPN. If other
browsers just turned off ALPN then the problem of how to hide the
protocol negotiation information would be solved. As far as I've been
led to believe, ALPN exists primarily to allow MitM to more easily
inspect what protocol the application is speaking and possibly block
the application, with as low of a cost to the MitM as possible--i.e.
exactly the opposite goal of everything we discussed at IETF88 in
perpass and in other discussions. I still have a lot of trouble
understanding how we can agree in the IETF as a whole that we should
be trying to make attacks more difficult, even if we can't make them
impossible, but here in the TLS WG, we are literally going out of our
way to leak information in the clear unnecessarily.

I think that in many cases, traffic analysis that relies on timing
information will be able to tell with a high degree of accuracy which
protocol has been selected for use on the connection. However, traffic
analysis cannot tell us what protocols were offered but *not*
selected. Further, traffic analysis based on timing is more expensive
and less reliable than having the information given on a silver
platter. In some cases, traffic analysis vs. explicit plaintext
leakage might be the difference between reasonable doubt or not. In
other cases, the difficulty of discerning two protocols--whether you
are sending email (SMTP) or pinging the server to retrieve email
(IMAP) vs loading the login page (HTTP)--may increase the storage and
processing cost by an order of magnitude or more, if we can get it to
work. Personally, I'm disappointed to hear the arguments that it is
not worthwhile to consider trying to solve that problem in the context
of the design of ALPN, because we worry it is unsolvable. I feel like
we haven't even started trying to solve it yet, so I don't understand
why we're giving up so easily.

Further, I don't see a need to rush to a conclusion on ALPN for HTTP/2
given this chart:
https://github.com/http2/http2-spec/wiki/Implementations

At the time I write this, 11/13 support NPN. 6/13 support ALPN, and
there are plenty of implementations that can interoperate. Literally,
a higher number of products would have to do extra work to reduce the
privacy of their protocol and settings negotiation than the number of
products that would need to flip a single bit (or less) to have a
higher level of privacy protection for the protocol and settings
negotiation. (Note that it isn't even clear what settings negotiation
HTTP/2 may add to the protocol negotiation step, so we cannot even
debate how sensitive the information in that settings negotiation is
when trying to determine if leaking application data in the ALPN
extension is acceptable.)

Also note that the Mozilla line in the chart is a little misleading.
Patrick added ALPN support to a private copy of Firefox for the
purpose of the interop testing, which is I support. However, Firefox
doesn't actually support ALPN in our shipping product, only in
Patrick's special build. And, I would very much like to keep it that
way.

FWIW, I am working on an experiment for adding NPN support to
Thunderbird (an email client based on Gecko, like Firefox), so that
Thunderbird can speak SMTP, IMAP, POP, LDAP, and HTTP to a single
server on a single port, https://example.org:443/. One purpose of this
experiment is to demonstrate how we can get through networks that
block, or charge extra money for, email protocols while still allowing
HTTPS, without needing separate hostnames for each server. A second
purpose is to evaluate whether there are effective ways of making side
channel analysis for distinguishing the user's activity--sending email
vs retrieving email, for example--or even what product the user is
using--Thunderbird vs. Firefox--more difficult than we might first
suspect. I would appreciate it if the working group could give me some
more time to complete this experiment and/or help me with it, before
completely giving up on trying to protect the protocol negotiation
information in the handshake.

Also, if we can build a reliable 1-RTT handshake in TLS 1.3, then
there may be a design that allows us to remove protocol negotiation
from TLS completely, putting it back in the application layer. I will
be sharing a write-up of how this would work shortly. IMO, it is much
better to have the long-term solution for protocol and settings
negotiation be in the application layer, if it is reasonable to do so.
I am not one of those people that yell "layer violation" at
everything, but if we can avoid later violations in the long term,
then we should do so.

All that said, I am very grateful to the F5 team for sharing the
information to work around the interop issue that I originally
complained about in my ALPN Concerns email during IETF88. Xiaoyong set
a great example for us all and I hope that that kind of openness in
sharing information continues here on this list.

Cheers,
Brian
-- 
Mozilla Networking/Crypto/Security (Necko/NSS/PSM)

---

From: Andrei Popov <[email-masked]>
Date: Tue, 19 Nov 2013 19:42:08 +0000

Hi Brian,

I totally agree that we need to encrypt more information in the TLS handshake, including the client cert and the contents of various extensions, such as SNI and ALPN. Unfortunately, the only way to make this information confidential in the existing versions of TLS is to incur additional round-trips (by using renegotiation, or otherwise exchanging messages after the TLS handshake is complete). This is one of the reasons why the TLS WG is pushing hard to make progress on TLS1.3.

> As far as I've been led to believe, ALPN exists primarily to allow MitM to more easily inspect what protocol the application is speaking and possibly block the application, with as low of a cost to the MitM as possible--i.e.
> exactly the opposite goal of everything we discussed at IETF88 in perpass and in other discussions.

No, ALPN exists primarily to negotiate an application protocol without incurring extra round-trips, to allow the server to pick the application protocol and present the corresponding identity, and to align with the existing TLS extensions.

>I think that in many cases, traffic analysis that relies on timing information will be able to tell with a high degree of accuracy which protocol has been selected for use on the connection. However, traffic analysis cannot tell us what protocols were offered but *not* selected.

Correct, and NPN offers the list of the application protocols in clear-text, so it does not solve this problem.

> I will be sharing a write-up of how this would work shortly. IMO, it is much better to have the long-term solution for protocol and settings negotiation be in the application layer, if it is reasonable to do so.
> I am not one of those people that yell "layer violation" at everything, but if we can avoid later violations in the long term, then we should do so.

If we can avoid application protocol negotiation at the TLS layer without incurring additional round-trips, I am in full support. Looking forward to the write-up.

Thanks,

Andrei

-----Original Message-----
From: TLS [mailto:[email-masked]] On Behalf Of Brian Smith
Sent: Monday, November 18, 2013 9:59 PM
To: Xiaoyong Wu
Cc: <[email-masked]>; Peter Gutmann
Subject: Re: [TLS] ALPN concerns

> As I am explaining this in detail, I would say that another work around on this would be making a client hello that exceeds 512 in length.

Xiaoyong, thank you very much for sharing that with us! That is incredibly useful.

In the thread "Final nail in the coffin for cleartext SNI/ALPN in TLS 1.3," Yoav Nir <[email-masked]> wrote:
> TLS 1.0 servers (and there are plenty of those around). And as others 
> have mentioned, SNI is needed in many cases. Without it, 
> So you might have a clientHello that leads to super-duper TLS 1.3, but 
> to avoid the need for fallback, you include the SNI (and presumably 
> ALPN) in the ClientHello.

This is exactly right for SNI. Even if we define a new replacement for SNI, browsers will still be sending SNI in the clear for the next 10 years. We should still try to work toward a long-term solution to that problem, but it isn't something that can be solved in the short term.

During the TLS WG session at IETF88, somebody commented that TLS 1.3 is supposed to solve the problem of encrypting information in the ClientHello, so the plaintext vs. cleartext issue shouldn't be a concern for ALPN. But, for the same reason we can't have any near-term solution for encrypting SNI in the ClientHello, we won't have any near-term solution for encrypting ALPN in the ClientHello that would meet browsers needs.

Again, I want to emphasize that Firefox ALREADY solves this problem, even without TLS 1.3, by implementing NPN and avoiding ALPN. If other browsers just turned off ALPN then the problem of how to hide the protocol negotiation information would be solved. As far as I've been led to believe, ALPN exists primarily to allow MitM to more easily inspect what protocol the application is speaking and possibly block the application, with as low of a cost to the MitM as possible--i.e.
exactly the opposite goal of everything we discussed at IETF88 in perpass and in other discussions. I still have a lot of trouble understanding how we can agree in the IETF as a whole that we should be trying to make attacks more difficult, even if we can't make them impossible, but here in the TLS WG, we are literally going out of our way to leak information in the clear unnecessarily.

I think that in many cases, traffic analysis that relies on timing information will be able to tell with a high degree of accuracy which protocol has been selected for use on the connection. However, traffic analysis cannot tell us what protocols were offered but *not* selected. Further, traffic analysis based on timing is more expensive and less reliable than having the information given on a silver platter. In some cases, traffic analysis vs. explicit plaintext leakage might be the difference between reasonable doubt or not. In other cases, the difficulty of discerning two protocols--whether you are sending email (SMTP) or pinging the server to retrieve email
(IMAP) vs loading the login page (HTTP)--may increase the storage and processing cost by an order of magnitude or more, if we can get it to work. Personally, I'm disappointed to hear the arguments that it is not worthwhile to consider trying to solve that problem in the context of the design of ALPN, because we worry it is unsolvable. I feel like we haven't even started trying to solve it yet, so I don't understand why we're giving up so easily.

Further, I don't see a need to rush to a conclusion on ALPN for HTTP/2 given this chart:
https://github.com/http2/http2-spec/wiki/Implementations

At the time I write this, 11/13 support NPN. 6/13 support ALPN, and there are plenty of implementations that can interoperate. Literally, a higher number of products would have to do extra work to reduce the privacy of their protocol and settings negotiation than the number of products that would need to flip a single bit (or less) to have a higher level of privacy protection for the protocol and settings negotiation. (Note that it isn't even clear what settings negotiation
HTTP/2 may add to the protocol negotiation step, so we cannot even debate how sensitive the information in that settings negotiation is when trying to determine if leaking application data in the ALPN extension is acceptable.)

Also note that the Mozilla line in the chart is a little misleading.
Patrick added ALPN support to a private copy of Firefox for the purpose of the interop testing, which is I support. However, Firefox doesn't actually support ALPN in our shipping product, only in Patrick's special build. And, I would very much like to keep it that way.

FWIW, I am working on an experiment for adding NPN support to Thunderbird (an email client based on Gecko, like Firefox), so that Thunderbird can speak SMTP, IMAP, POP, LDAP, and HTTP to a single server on a single port, https://example.org:443/. One purpose of this experiment is to demonstrate how we can get through networks that block, or charge extra money for, email protocols while still allowing HTTPS, without needing separate hostnames for each server. A second purpose is to evaluate whether there are effective ways of making side channel analysis for distinguishing the user's activity--sending email vs retrieving email, for example--or even what product the user is using--Thunderbird vs. Firefox--more difficult than we might first suspect. I would appreciate it if the working group could give me some more time to complete this experiment and/or help me with it, before completely giving up on trying to protect the protocol negotiation information in the handshake.

Also, if we can build a reliable 1-RTT handshake in TLS 1.3, then there may be a design that allows us to remove protocol negotiation from TLS completely, putting it back in the application layer. I will be sharing a write-up of how this would work shortly. IMO, it is much better to have the long-term solution for protocol and settings negotiation be in the application layer, if it is reasonable to do so.
I am not one of those people that yell "layer violation" at everything, but if we can avoid later violations in the long term, then we should do so.

All that said, I am very grateful to the F5 team for sharing the information to work around the interop issue that I originally complained about in my ALPN Concerns email during IETF88. Xiaoyong set a great example for us all and I hope that that kind of openness in sharing information continues here on this list.

Cheers,
Brian
--
Mozilla Networking/Crypto/Security (Necko/NSS/PSM) _______________________________________________
TLS mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tls