Subject: [TLS] More compatibility measurement results

From: Eric Rescorla <[email-masked]>
Date: Fri, 22 Dec 2017 12:00:12 -0800

Hi folks,

Here are the results of our experiment with Firefox Nightly (draft-22)
against Facebook.

EXPERIMENTAL DESIGN
This is a forced experiment in which each client tries all the
variants. The experiment is deployed via a system add-on (a remotely
deployable, centrally managed piece of JavaScript code), and then
takes measurements by trying to do an XHR to a given URL
(https://www.tls13.facebook.com/) with a specific set of flags. We
do the following three measurements:

- TLS 1.2
- TLS 1.3 draft-22
- TLS 1.3 draft-22 in compat mode.

We take five trials for each measurement, randomly shuffling the
measurement order and then repeating the shuffled pattern five
times. Each trial is done with a different connection and we declare
"success" when any of the five trials succeeds.

RESULTS
This experiment was run on a 40% sample of the Firefox Nightly population
who have locale set to en-US. The data below is taken from the period
20171216 to 20171222. There's a bit of contamination in the targeting
because we temporarily failed to filter on "en-US", but that should
mostly only affect the first day.

37716 clients started the experiment and 37430 completed it (99.2%).

The results are:

                                    Success         Fail         Rate
                      fb-tls12        35615         1815     0.048491
             fb-tls13-draft-22        35552         1878     0.050174
       fb-tls13-draft22-compat        35630         1800     0.048090

The overall failure numbers here are a lot higher than with our Beta
experiment, which may be a result of different targeting on Nightly
versus Beta. In particular, I'm still seeing a lot of data from China
and Vietnam, which seem to have high blocking rates in general (i.e.,
not just for TLS 1.3). If I restrict to non China and non-Vietnam, we
get:

                                    Success         Fail         Rate
                      fb-tls12        35034         1176     0.032477
             fb-tls13-draft-22        34960         1250     0.034521
       fb-tls13-draft22-compat        35037         1173     0.032394

None of these differences are statistically significant (in the second
data set, the p value for 1.2 versus -22 is .13), but this all seems
consistent with saying that that -22 compat mode isn't significantly
worse than TLS 1.2 and that normal -22 may be somewhat worse
(unfortunately, we don't have -18 in this experiment).

Taken together with the results David has reported and our previously
reported Beta results, this seems fairly encouraging. We'll probably
let the Nightly experiment run a little longer to see if we hit
significance,
but after that will start looking at a rollout of -22 to Release.

-Ekr

ADDITIONAL DETAILS
Experimental code:
https://github.com/mozilla/one-off-system-add-ons/tree/master/addons/tls13-middlebox-draft22

---

From: Ilari Liusvaara <[email-masked]>
Date: Sat, 23 Dec 2017 16:07:15 +0200

> but after that will start looking at a rollout of -22 to Release.

~3.25% baseline failure rate? That sounds quite high. ~0.2% above-
baseline failure rate for non-compat? That sound fairly low, but
there have been improvements here that could have caused substantial
decrease.

I wonder if the high baseline failure rate is due to high amount of
blocking of the test server. And unfortunately, the places that
blocked the test server are some of the most interesting when it comes
to the compatibility.

However, the results do establish that the incremential failure rates
in open environments (anything that blocks the testserver very probably
is not open environment) are low enough to proceed with.

-Ilari

---

From: Sean Turner <[email-masked]>
Date: Sat, 23 Dec 2017 08:30:43 -0600

Thanks for these!

spt

Sent from my iPhone

> On Dec 22, 2017, at 14:00, Eric Rescorla <[email-masked]> wrote:
> 
> Hi folks,
> 
> let the Nightly experiment run a little longer to see if we hit significance,
> Experimental code: https://github.com/mozilla/one-off-system-add-ons/tree/master/addons/tls13-middlebox-draft22
> 
> _______________________________________________
> TLS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tls