Subject: [TLS] weird ECDSA interop problem with cloudflare/nginx

From: [email-masked] (Martin Rex)
Date: Mon, 25 Jul 2016 21:08:49 +0200 (CEST)

I've just run into a weird interoperability problem with an (alleged)
cloudflare/nginx TLS server and my personal Firefox settings.

https://regmedia.co.uk/2015/07/14/giant_weta_mike_locke_flicker_cc_20.jpg

Traditionally I have all TLS ciphersuites with ECDSA disabled through
about:config, but it seems that recently two new TLS ciphersuites were
added to FF, which caused complete loss of interop with regmedia.co.uk
for me with my existing configuration. (Loss of pictures&media on the
www.theregister.co.uk news site).

specifically, after the FF update, this new TLS ciphersuite:

   security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256  (0xcc, 0xa9)

was the only ECDSA cipher suite enabled in my Firefox 47.0.1, and this
kills connectivity (TLS handshake_failure alert) with regmedia.co.uk.

It looks like a bug in the cloudflare/nginx cipher suites selection
algorithm, which appears to blindly go for ECDSA, even though there is
no actual ECDSA cipher suite available which the server supports.

-Martin

---

From: Ilari Liusvaara <[email-masked]>
Date: Mon, 25 Jul 2016 22:37:08 +0300

> no actual ECDSA cipher suite available which the server supports.

I recently (tried to) implement(ed) TLS 1.2 ciphersuite negotiation
in a way that always negotiates something if at least one valid
configuration exists, and respects TLS 1.2 rules.

The resulting code was totally insane, and I am very much not surprised
to see buggy implementations. Nor can I say my implementation is not
buggy, as testing that mess is just about impossible (and it it will
very much do GIGO in order to maximize interop).

-Ilari

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 25 Jul 2016 16:15:28 -0400

> On Jul 25, 2016, at 3:08 PM, Martin Rex <[email-masked]> wrote:
> 

FWIW, OpenSSL interoperates with this server:

Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4169 bytes and written 310 bytes
---
New, TLSv1.2, Cipher is ECDHE-ECDSA-AES128-GCM-SHA256
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES128-GCM-SHA256
    Session-ID: 740096F3B8F9BD08D40028405CA2631E0348BE65BE7453635E1A970081CAE3CE
    Session-ID-ctx:
    Master-Key: 17FDD891614358874800B6F152D4297E715FE20C74205A43BD803DA91CD891287677672AE4E4F7FBCD989FFE675BF22F
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64800 (seconds)

The server then responds to an HTTP "HEAD" request:

HTTP/1.1 404 Not Found
Date: Mon, 25 Jul 2016 20:09:52 GMT
Content-Type: text/html
Connection: keep-alive
Set-Cookie: __cfduid=dc03aaa6d430491fcd2a736eaa790f5e11469477392; expires=Tue, 25-Jul-17 20:09:52 GMT; path=/; domain=.regmedia.co.uk; HttpOnly
Cache-Control: private, no-cache, no-store, max-age=0
Accept-Ranges: bytes
Server: cloudflare-nginx
CF-RAY: 2c8240858296187c-EWR

-- 
	Viktor.

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 25 Jul 2016 16:36:27 -0400

> kills connectivity (TLS handshake_failure alert) with regmedia.co.uk.

OpenSSL lists "CC, A9" as:

0xCC,0xA9 - ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD

Which is not AES_128_GCM.  The IANA registry seems to agree:

https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-4

  0xCC,0xA9	TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256	Y	[RFC7905]

-- 
	Viktor.

---

From: Ilari Liusvaara <[email-masked]>
Date: Tue, 26 Jul 2016 00:00:21 +0300

> 0xCC,0xA9 - ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD

I wonder what is happening is that having ECDSA ciphersuite enabled
causes Firefox to include the ECDSA signatures in signature_algorithms
and the server end sees ECDSA supported, tries to choose that, but
then blows up when it does not find any enabled ciphersuites it knows
that support ECDSA...

-Ilari

---

From: Benjamin Kaduk <[email-masked]>
Date: Mon, 25 Jul 2016 16:19:37 -0500

If I remember/understand correctly, the cloudflare patch for chacha/poly
would (when server preference is in use) only attempt to use it when it
appeared first in the client's preference list, and would ignore it
elsewhere.  This could potentially lead to negotiation failures if,
e.g., the server only supported chacha/poly but the client did not put
it first in the list.  That doesn't seem to match up with what is being
described here, but could provide some insight into where to look.

-Ben

On 07/25/2016 04:00 PM, Ilari Liusvaara wrote:
> _______________________________________________
> TLS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tls

---

From: [email-masked] (Martin Rex)
Date: Tue, 26 Jul 2016 09:19:38 +0200 (CEST)

Correction--

I'm sorry, I mistyped the firefox config, this should have said
the chacha20_poly1305 (0xcc 0xa9) cipher suite was the only one enabled.

>    security.ssl3.ecdhe_ecdsa_aes_128_gcm_sha256  (0xcc, 0xa9)

    security.ssl3.ecdhe_ecdsa_chacha20_poly1305_sha256  (0xcc, 0xa9)

---

From: Peter Gutmann <[email-masked]>
Date: Tue, 26 Jul 2016 07:48:05 +0000

Ilari Liusvaara <[email-masked]> writes:

>I recently (tried to) implement(ed) TLS 1.2 ciphersuite negotiation in a way
>that always negotiates something if at least one valid configuration exists,
>and respects TLS 1.2 rules.
>
>The resulting code was totally insane, and I am very much not surprised to
>see buggy implementations. Nor can I say my implementation is not buggy, as
>testing that mess is just about impossible (and it it will very much do GIGO
>in order to maximize interop).

Do you have any more details on some of the issues you ran into?  It'd be good
to know for anyone else in this situation.

(I've had to do the same thing, with awkward logic that backs off and retries
different options if an earlier attempt fails.  This was one of the motivators
for the Grigg's Law cipher-suite handling in TLS-LTS).

Peter.

---

From: [email-masked] (Martin Rex)
Date: Tue, 26 Jul 2016 11:52:25 +0200 (CEST)

>   0xCC,0xA9	TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256	Y	[RFC7905]

Sorry for the confusion about the cipher suite.

The issue seems a little weirder than what I thought, because the
failure seems to happen only for a particular cipher suite combo
(which happens to be the combo produced by my own Firefox config):

I can repro the handshake failure with openssl-1.1.0-pre5 with this
command line:

Failure:
openssl s_client -connect regmedia.co.uk:443 -cipher ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305

Success:
openssl s_client -connect regmedia.co.uk:443 -cipher ECDHE-RSA-AES128-GCM-SHA256

Success:
openssl s_client -connect regmedia.co.uk:443 -cipher ECDHE-ECDSA-CHACHA20-POLY1305

-Martin

---

From: Ilari Liusvaara <[email-masked]>
Date: Tue, 26 Jul 2016 13:09:04 +0300

> openssl s_client -connect regmedia.co.uk:443 -cipher ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305

If you swap the order of these two ciphersuites, does it suceed or fail?

I.e.

openssl s_client -connect regmedia.co.uk:443 -cipher ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256

Well, your test results certainly blow basic "negotiation accidentially
blows off all valid candidates and then fails" hypothesis out of the
water. So it has to be soemthing more complicated.

Succeeding with the ciphersuites swapped would suggest (as somebody
else in this thread already said) that it only considers Chacha in
the first place, not noticing that it may be the only choice after
certificate selection.

-Ilari

---

From: Ilari Liusvaara <[email-masked]>
Date: Tue, 26 Jul 2016 13:32:57 +0300

> for the Grigg's Law cipher-suite handling in TLS-LTS).

The basic problem (let's ignore non-cert modes for a bit):

When choosing the certificate, you need to consider if you have a
ciphersuite that can use some supported group and protection/prf-hash
available.

Similarly, when choosing a group, you need to consider that you have
a ciphersuite that is consistent in other (possible) choices.

And protection/prf-hash also has to be consistent with other
choices.

The reason these choices couple is because the client can send
pretty much arbitrary combination of oters that depends on each
considered factor.

And the selections have to be consistent: The current leading
hypothesis about this interop issue is that certificate selection
consisers ciphersuite valid candidate when protection selection does
not.

TLS 1.3 negotiation revamp would solve this by ensuring that
whichever you choose first, you don't restrict choice on others
(PSK modes are special here, so need to be chosen first[1]), so
whatever you have chosen so far, there either is next valid
choice, or the whole problem has no solution.

[1] There are some couplings here too, but one only needs to check
group overlap for ke and signature/cert overlap for auth (and these
overlap decisions can be used for normal dhe/cert anyway). And one
does not need to consider interaction of those two.

-Ilari

---

From: Peter Gutmann <[email-masked]>
Date: Tue, 26 Jul 2016 11:15:09 +0000

Ilari Liusvaara <[email-masked]> writes:

>When choosing the certificate, you need to consider if you have a ciphersuite
>that can use some supported group and protection/prf-hash available.
>
>Similarly, when choosing a group, you need to consider that you have a
>ciphersuite that is consistent in other (possible) choices.
>
>And protection/prf-hash also has to be consistent with other choices.
>
>The reason these choices couple is because the client can send pretty much
>arbitrary combination of oters that depends on each considered factor.

OK, so that's pretty much the same stuff I ran into.  What makes it really
ugly is that you can no longer decide, based on a cipher suite offered,
whether you can actually continue with that, but then have to tunnel down into
the extensions that follow the suites to see what's there.  If there's nothing
appropriate you go back to the cipher suites and find the next-best match, try
that against the extensions, and so on.  Mostly things appear to work now
because virtually everything supports the de facto standard of P256 + SHA-256
with ECDSA and ECDH (which is why TLS-LTS specifies it as one of its two suite
choices), but once you start veering away from that de facto universal
standard you run into problems.  Rather than spend forever trying various
poke-and-hope strategies, my code looks for the default form (P/SHA-256) and
if that's not found falls back to DHE + RSA.  That seems to work well enough,
and I don't have to debug weird failures with oddball combinations.

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Tue, 26 Jul 2016 11:28:05 +0000

Since I've referred to TLS-LTS a couple of times now I should mention that
I've just posted an update, with the following changes:

- Clarified what happens during a session resumption.

- Fixed the ServerKeyExchange text to indicate what happens when the hash
  isn't the default SHA-256.  Is the resulting text comprehensible?  That is,
  does it make clear what's signed, and with what hash?

- Added an alternative, quicker way to verify domain parameters that doesn't
  require the full FIPS 186 checks.

- Reworked the text about the handling of extensions yet again.  I'm still not
  happy with this, or certain that it's sufficiently unambiguous, can people
  see if they can pick holes in it?

- Reworked the rationale.

Peter.

---

From: Hubert Kario <[email-masked]>
Date: Tue, 26 Jul 2016 13:29:05 +0200

> no actual ECDSA cipher suite available which the server supports.

could you post packet capture (pcap format, if possible) that shows that 
problem?

-- 
Regards,
Hubert Kario
Senior Quality Engineer, QE BaseOS Security team
Web: www.cz.redhat.com
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 26 Jul 2016 12:08:33 +0000

> openssl s_client -connect regmedia.co.uk:443 -cipher ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256

I can reproduce the reported failure in the original order, and at
least for me the swapped variant succeeds.

> the first place, not noticing that it may be the only choice after

Perhaps that's the issue.

-- 
	Viktor.

---

From: Hubert Kario <[email-masked]>
Date: Tue, 26 Jul 2016 16:01:15 +0200

On Tuesday, 26 July 2016 12:08:33 CEST Viktor Dukhovni wrote:
> > > openssl s_client -connect regmedia.co.uk:443 -cipher
> > > ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305> 
> > openssl s_client -connect regmedia.co.uk:443 -cipher
> > ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256

if you send the AES-GCM then Chacha but do NOT include signature algorithms, 
it allows you to connect (other extensions are irrelevant).

If you send AES-GCM then Chacha in ciphers but in signatures algorithms don't 
include the SHA256/ecdsa pair, it works too.

in other words, this connects:
openssl s_client -connect regmedia.co.uk:443 -cipher ECDHE-RSA-AES128-GCM-
SHA256:ECDHE-ECDSA-CHACHA20-POLY1305 -sigalgs ECDSA+SHA512:RSA+SHA256

and this connects:
openssl s_client -connect regmedia.co.uk:443 -cipher ECDHE-RSA-AES128-GCM-
SHA256:ECDHE-ECDSA-CHACHA20-POLY1305 -sigalgs ECDSA+SHA512:RSA+SHA512

but this doesn't:
openssl s_client -connect regmedia.co.uk:443 -cipher ECDHE-RSA-AES128-GCM-
SHA256:ECDHE-ECDSA-CHACHA20-POLY1305 -sigalgs ECDSA+SHA256:RSA+SHA256

(using openssl-1.1.0-pre5)
-- 
Regards,
Hubert Kario
Senior Quality Engineer, QE BaseOS Security team
Web: www.cz.redhat.com
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic

---

From: Ilari Liusvaara <[email-masked]>
Date: Tue, 26 Jul 2016 17:09:06 +0300

> least for me the swapped variant succeeds.

Thank you.
 
> Perhaps that's the issue.

Oops, that should have been:

Chacha ciphersuites are considered for certificate selection in any
position, but only in first position for protection selection.

-Ilari