Subject: [DNSOP] Do53 vs DoT vs DoH Page Load Performance Study at ANRW

From: Kevin Borgolte <[email-masked]>
Date: Fri, 19 Jul 2019 12:26:40 +0800

Hi all,

we recently did a study on Do53/DoT/DoH performance on webpage load times in Firefox, which we wanted to share with you. Austin Hounsel will give a talk about our work at ANRW on Monday. Paul Schmitt and I will also be around (myself only Monday afternoon and Tuesday) if you’d like to chat about it in person. The extended pre-print of our paper is available here: https://arxiv.org/abs/1907.08089

We looked at the effect of Do53/DoT/DoH on page load times because we figured they will be a more interesting measure than simple DNS resolution times. We also looked at simulated mobile connections (4G, lossy 4G, and 3G). We found that DoH is on par with Do53/DoT if your Internet connection is good, and worse if it is not. The potential for DoH seems to be largely push. ADD can also make sense, or at least an extended/updated OS API would.

We think there are two opportunities to improve Do53 and DoT: better wire format caching and partial responses combined with multiple questions. Part of the improvements and issues we discovered we later found out (by talking to some of you) were mentioned in early working group drafts, but they never made it into published RFCs or mailing list posts (which is where we looked initially). We hope you can help and explain the thoughts that went into these decisions 15 to 20 years ago. Maybe it even makes sense to revisit the ideas from back then combined with our ideas, considering how the Internet has evolved?

We‘d appreciate any feedback on our work. Please also feel free to reach out to us directly (in person or by email) if you have any insight or feedback you’d prefer not to post to the list.

Best,
Kevin

P.S. Please excuse the posting to multiple lists, but all seem relevant.

---

From: Rob Sayre <[email-masked]>
Date: Thu, 18 Jul 2019 22:10:34 -0700

On Thu, Jul 18, 2019 at 9:27 PM Kevin Borgolte <[email-masked]> wrote:

> We‘d appreciate any feedback on our work. Please also feel free to reach
> out to us directly (in person or by email) if you have any insight or
> feedback you’d prefer not to post to the list.
>

This paper looks interesting. Is the software used in the paper published?

Or, at least, is the test page set published? I haven't read the whole
thing yet, but it seems like the page set would be relevant if the paper
tests page load time.

thanks,
Rob

---

From: Kevin Borgolte <[email-masked]>
Date: Fri, 19 Jul 2019 13:42:26 +0800

> This paper looks interesting. Is the software used in the paper published?

Thanks! The code isn’t open source yet, but we will make it public alongside the Docker setup we used for running it. Not sure when that is going to happen exactly though.

> Or, at least, is the test page set published? I haven't read the whole thing yet, but it seems like the page set would be relevant if the paper tests page load time.

The list of websites is attached. It is extracted from the top 1,000 and 99,000 to 100,000 of a Tranco list.

Best,
Kevin

---

From: Rob Sayre <[email-masked]>
Date: Thu, 18 Jul 2019 22:56:19 -0700

> 99,000 to 100,000 of a Tranco list.

Thanks for attaching the list. Having seen a fair a number of these, I
think it looks reasonable.

But, I think you should add the list and the reason for the range choice to
the paper. For example, I can't tell what range you actually used from your
description (although that might just be due to a hurried reply).

Another issue is that, while your paper might accurately capture the
network conditions on your local network, it's probably doesn't capture
network variation as well as a large scale test along the lines of what
Mozilla did. For example, if the university used a single router brand,
this could skew the test. As one data point, I've never seen the various
network-throttling apps match a real-user-metrics test very well, although
they do catch really problematic situations.

This test is a welcome contribution, but given the data in the paper, it
would be difficult to reproduce.

thanks,
Rob

---

From: Kevin Borgolte <[email-masked]>
Date: Fri, 19 Jul 2019 18:10:34 +0800

> But, I think you should add the list and the reason for the range choice to the paper. For example, I can't tell what range you actually used from your description (although that might just be due to a hurried reply).

Section 3.2.4 talks about the selection of websites:

We collect HARs (and resulting DNS lookups) for the top 1,000 websites on the Tranco top-list to understand browser performance for the average user visiting popular sites. Furthermore, we measure the bottom 1,000 of the top 100,000 websites (ranked 99,000 to 100,000) to understand browser performance for websites that are less popular. We chose to measure the tail of the top 100,000 instead of the tail of the top 1 million because we found through experimentation that many of the websites in the tail of the top 1 million were offline at the time of our measurements. Furthermore, there is significant churn in the tail of top 1 million, which means that we would not be accurately measuring browser performance for the tail across the duration of our experiment.

We didn't include the full list in the paper itself for space reasons and because extracting the list from the paper would be cumbersome. It will be part of our future open source release though.

> Another issue is that, while your paper might accurately capture the network conditions on your local network, it's probably doesn't capture network variation as well as a large scale test along the lines of what Mozilla did. For example, if the university used a single router brand, this could skew the test. As one data point, I've never seen the various network-throttling apps match a real-user-metrics test very well, although they do catch really problematic situations.

It is true that the an experiment from more diverse vantage points could be more useful to capture network variations, however, we already see that the protocols perform differently to the degree that a human could notice (that is, we already show that selecting your DNS protocol based on your network characteristics can have a non-negligible impact). Running a large scale test for page load times like Mozilla did for resolution times is also quite difficult: Simple telemetry data won't answer the questions we want to ask, as websites need to be fetched multiple times (per protocol/recursor combination), and browser caches need to be flushed. These are not things you want to do with your users. Using existing measurement platforms also was not an option, as they are limited in what you can access or do not support the necessary protocols. Meaning, you would need to set up your own measurement clients, which brings questions like "How do you select the vantage points?" (you can't use hosting/cloud providers, as this would lead to little network variation). If you have an idea on how to easily run this on a larger scale, we'd love to look into it.

We set link capacity and latency based on best case values from real-world measurements (Section 3.2.3) from OpenSignal, and we use iproute2 for traffic control (tc), which we verified to be accurate through latency (Table 1) and multiple "speedtest" (via Ookla). By tuning the network setup in this way (instead of using a 4G/3G modem for connectivity), we eliminated potential differences in routing that could have otherwise influenced our results and impact comparability.

-- Kevin

---

From: Rob Sayre <[email-masked]>
Date: Fri, 19 Jul 2019 14:13:59 -0700

On Fri, Jul 19, 2019 at 3:10 AM Kevin Borgolte <[email-masked]> wrote:

>
> > But, I think you should add the list and the reason for the range choice
> to the paper. For example, I can't tell what range you actually used from
> your description (although that might just be due to a hurried reply).
>
> We collect HARs (and resulting DNS lookups) for the top 1,000 websites on
> the Tranco top-list to understand browser performance for the average user
> visiting popular sites. Furthermore, we measure the bottom 1,000 of the top
> 100,000 websites (ranked 99,000 to 100,000) to understand browser
> performance for websites that are less popular.
>

I see, that makes sense. I'm still having trouble interpreting the
measurements, given that pageload can be blocked by additional DNS queries
and HTTP requests. For example, could an analytics provider interfere with
performance evaluation here, if it were present in a lot of pages? Or, what
if the results indicate a performance problem in one common hosting vendor
(anything from AWS to Wordpress)? As I said, the paper is a welcome
contribution, but I find the results surprising and want to look more. I
would have expected DoH and DoT to be about the same in practice, but your
paper does not show this.

> We didn't include the full list in the paper itself for space reasons and
> because extracting the list from the paper would be cumbersome. It will be
> part of our future open source release though.
>

Understood--I meant that I looked for a separate download and didn't see
one (thank you for sending it along).

thanks,
Rob

---

From: Puneet Sood <[email-masked]>
Date: Sun, 21 Jul 2019 16:26:24 -0400

Thanks for sharing the results of your work. It will be great to have
the software available so others can run the experiments from other
locations.

When looking at the page load results the CDF graphs comparing the
various services are very useful to see the relative performance of
different services. However I could not find the range of time values
for the page loads in the experiment. Basically what percentage of the
page load time variation was related to DNS?

Note: For Google DoH, we will be reviewing our implementation for
latency. BTW we have launched our production RFC 8484 DoH service
recently at https://dns.google/dns-query
(https://security.googleblog.com/2019/06/google-public-dns-over-https-doh.html).
It will be great if you can update your software to use this endpoint.

* The experiment was run from Princeton, New Jersey in Northeast US.
The location is in a very well connected part of the world between
network peering points in NYC and Washington DC. You will not see much
difference (due to network latency) between the cloud providers and
the default (local) Do53. Running the experiment from locations which
are further away from cloud providers would provide another
interesting set of data.

* Conclusion on benefit (or lack) of ECS.
Did the page load measurements include content that would benefit from
proximity to the end user, e.g. streaming videos or large downloads?
This kind of content benefits from ECS when the resolver is further
away from the client.

Thanks,
Puneet

On Fri, Jul 19, 2019 at 1:42 AM Kevin Borgolte <[email-masked]> wrote:
> _______________________________________________
> dns-privacy mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dns-privacy

---

From: "Livingood, Jason" <[email-masked]>
Date: Mon, 22 Jul 2019 16:41:57 +0000

On 7/21/19, 4:27 PM, "DNSOP on behalf of Puneet Sood":
> Running the experiment from locations which are further away from cloud providers would provide another interesting set of data.

[JL] +1. I understand the US FCC is considering running DoH tests from their Measuring Broadband America end points around the US. Those devices are managed by SamKnows, which just announced they added a DoH test to their suite of measurements. 

> Conclusion on benefit (or lack) of ECS. Did the page load measurements include content that would benefit from proximity to the end user, e.g. streaming videos or large downloads? This kind of content benefits from ECS when the resolver is further away from the client.

[JL] I wonder if it might be useful for CDNs to publish some measurement test URLs that are highly localized (e.g. distributed widely & as close as possible to users). That might enable better testing of performance for accessing content that benefits from proximity to users.

---

From: Andrew Campling <[email-masked]>
Date: Mon, 22 Jul 2019 20:45:43 +0000

> [JL] I wonder if it might be useful for CDNs to publish some measurement test URLs that are highly localized (e.g. distributed widely & as close as possible to users). That might enable better test…

[AC] This would be helpful given it appears some (all?) DoH resolvers have indicated that they will not pass sufficient information to (rival) CDN vendors to allow geographic routing.  Clearly if this is commonplace then performance gains from DoH, if any, could be lost from an end-user perspective due to inefficient routing of content.  

Andrew

---

From: Paul Vixie <[email-masked]>
Date: Mon, 22 Jul 2019 20:52:39 +0000

On Monday, 22 July 2019 20:45:43 UTC Andrew Campling wrote:
> ...
> 
> [AC] This would be helpful given it appears some (all?) DoH resolvers have
> indicated that they will not pass sufficient information to (rival) CDN
> vendors to allow geographic routing.  Clearly if this is commonplace then
> performance gains from DoH, if any, could be lost from an end-user
> perspective due to inefficient routing of content.  

several people have disagreed with my public statements that "everyone should 
run their own rdns, don't use quad-anything" on the basis of the blender 
effect. when i tell them about ECS they stop talking, which is too bad, 
because they then don't get to hear the stories of how /32 and /128 are the 
default ECS subnet size for many ISP resolvers.

apparently ECS creates problems for privacy, but _how could we have 
suspected_?

-- 
Paul

---

From: Jim Reid <[email-masked]>
Date: Mon, 22 Jul 2019 22:19:47 +0100

> On 22 Jul 2019, at 21:52, Paul Vixie <[email-masked]> wrote:
> 
> apparently ECS creates problems for privacy, but _how could we have suspected_?

IIRC the ECS privacy issues were recognised at the time. They lost out to the argument that CDNs were already doing (or about to do) ECS and it would be better (for some definition of better) if this was done in a way that had an RFC behind it.

---

From: Paul Vixie <[email-masked]>
Date: Mon, 22 Jul 2019 21:27:34 +0000

> > apparently ECS creates problems for privacy, but _how could we have

(that, by the way, was sarcasm.)

> IIRC the ECS privacy issues were recognised at the time.

raised, yes. recognized, no.

> They lost out to
> the argument that CDNs were already doing (or about to do) ECS and it would
> be better (for some definition of better) if this was done in a way that
> had an RFC behind it.

this was said, and was false, at the time. it's more obviously false today, i 
think.

-- 
Paul

---

From: Willem Toorop <[email-masked]>
Date: Wed, 24 Jul 2019 17:18:03 -0400

> are further away from cloud providers would provide another

Thanks for this Puneet and for stating this at the microphone during the
presentation.  I have some numbers that support this.

At the hackathon at the African Internet Summit in Kampala Uganda one
month ago, I participated with a team using RIPE Atlas to compare
response time of the locally configured resolvers of RIPE Atlas probes
in the Africa region, to the cloud resolvers 1.1.1.1, 8.8.8.8 and 9.9.9.9.

On UDP, on the median, local resolvers returned responses 27 times
quicker than the fastest cloud resolver: 2ms RTT for the local resolver
versus 55ms for the fastest cloud resolver (which was 9.9.9.9). On TCP,
local resolvers provide responses at least 6 times quicker. 14ms versus
94ms (again Quad9).

We also measured DoT which had +-3 times longer response time than TCP
on the cloud DNS providers.  There were only very few probes with
resolvers other than the quads that supported DoT, but those were also
remote given their long round trip times.

See also our post/report on:

https://labs.ripe.net/Members/willem_toorop/hackathon-africa-internet-summit-2019