Subject: [TLS] Preliminary data on Firefox TLS 1.3 Middlebox experiment

From: Eric Rescorla <[email-masked]>
Date: Tue, 5 Dec 2017 13:35:42 -0800

Hi folks,

I now have some preliminary numbers to share with the group based on
our Firefox experiments. The executive summary is that our data
confirms Google's results. More detail below.

EXPERIMENTAL DESIGN
This is a forced experiment in which each client tries all the
variants. The experiment is deployed via a system add-on (a remotely
deployable, centrally managed piece of JavaScript code), and then
takes measurements by trying to do an XHR to a given URL
(https://mail.google.com/robots.txt) with a specific set of flags. We
do the following three measurements:

- TLS 1.2
- TLS 1.3 draft-18
- TLS 1.3 draft-18 with (approximately) PR#1092 ("7e02")

We take five trials for each measurement, randomly shuffling the
measurement order and then repeating the shuffled pattern five
times. Each trial is done with a different connection and we declare
"success" when any of the five trials succeeds.

RESULTS
This experiment was run on a 2% sample of the Firefox Beta population
who have locale set to en-US, which we selected because of very
high GMail blocking rates in some locales, which is a potential
confounding factor. The experimen started 11/27 and has been running
through today.

This gave us an initial population of 161578, of whom 160809 (99.5%
completed the experiment and reported results). This produced the
following results:

                     Success      Failure      Fail Rate
--------------------------------------------------------
TLS 1.2               158260         2549          .0158
TLS 1.3-18            158194         4743          .0291
TLS 1.3-Experiment    158194         2615          .0163

For the statistics minded, the difference between -18 and 1.2 is
significant at p < .001 and the 95% confidence interval of the failure
rate difference is .0122-.0143 (using R's prop.test). There is no
significant difference between 1.2 and 1.3-experiment (p = .36).

We've got a -22 experiment in flight now, but it will only be on
Nightly, so this is probably the strongest data we will have for
a while.

-Ekr

ADDITIONAL DETAILS
The relevant NSS version:
https://dxr.mozilla.org/mozilla-beta/source/security/nss/lib/ssl
Experimental code:
https://github.com/mozilla/one-off-system-add-ons/tree/master/addons/tls13-middlebox-ghack
iPython Notebook with analysis:
https://gist.github.com/ekr/598208b5399faf303453b10cb11647bf

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 5 Dec 2017 13:53:52 -0800

>                      Success      Failure      Fail Rate

Oops. This first number should be 156066. This is what happens when you cut
and paste from your notebook.

-Ekr

> The relevant NSS version: https://dxr.mozilla.org/
> mozilla-beta/source/security/nss/lib/ssl
> Experimental code: https://github.com/mozilla/one-off-system-add-ons/tree/
> master/addons/tls13-middlebox-ghack
> iPython Notebook with analysis: https://gist.github.com/ekr/
> 598208b5399faf303453b10cb11647bf
>
>
>
>
>
>
>