How Ookla Ensures Accurate, Reliable Data: A Guide to Our Metrics and Methodology (Updated for 2020)
At Ookla® we care deeply about providing data and analyses that are accurate and statistically sound so that consumers and businesses can trust the information they’re receiving. We’ve been doing this for over 14 years.
Speedtest®, our flagship product, is the most reliable tool for measuring internet performance and providing network diagnostics. Every day, millions of people use Speedtest to better understand the performance and quality of their internet connections. Read more about why consumer-initiated testing is more accurate than other methods.
Ookla’s Speedtest Server Network™ is comprised of high-performance servers in every country and major population center. Read more about the Speedtest Server Network and how Speedtest is uniquely able to measure high-speed connections.
We subject this vast amount of data to rigorous aggregation and analysis, making Ookla the preferred data provider for telecommunications operators, regulatory bodies, trade bodies, analysts, journalists and nonprofits worldwide. Read more about how our aggregation methodology sets Speedtest apart.
This article also shares the details of Ookla’s operational definitions and metrics. Read more about the language we use to describe our analyses.
Speedtest is the most accurate measure of real-world network performance and coverage
Each time a user takes a Speedtest, a snapshot of the internet is captured for that specific time, place, device and network. Because these tests are initiated by consumers when and where they need performance data, Speedtest gives users accurate information about internet speeds at the times and locations that are important to them. When aggregated, these measurements describe the network’s real-world performance.
Speedtest measures the full capacity of an internet connection
Each Speedtest runs as a dedicated foreground service. This allows the device to use enough data to flood the internet connection and measure the full capability of both the network connection and the device. Only a dedicated foreground service can accurately assess network performance and quality metrics such as: download speed, upload speed, latency, packet loss, jitter and other indicators of network conditions.
Companies that attempt to measure speed using background tests hidden inside of other apps only send small amounts of data back and forth and cannot accurately measure performance, particularly at high speeds.
Because Speedtest operates in the foreground and measures the full throughput capacity of a connection, we can properly assess the performance capability of even the fastest connection. This is why speeds measured with Speedtest are often higher than those measured with other methods. This difference can be substantial, especially when testing the performance of newer technologies like 5G and multi-gigabit fiber.
Mobile Speedtest users on Android devices can also opt in to submit data from background coverage scans. Our users contribute billions of measurements each day on the quality and conditions of mobile networks in their area from over 300 million scans. These coverage scans provide real-time insights into signal conditions, spectrum usage and network equipment at a fine level of geographic detail.
The hidden downsides of background testing from other providers
Other internet testing solutions run in the background of third-party applications that users are often unaware of, such as messengers, call recorders, dating apps and media converters.
Due to protections Apple has in place for user privacy, iOS doesn’t surface information like connection type to apps that run in the background. It is vital that a testing solution run in the foreground on iOS to accurately test the network and collect information about the active SIM and access technology. Aggregate speeds you might see from other network testing providers do not adequately represent the results of iOS users, which make up a large and important segment of the market.
As Google continues to update their data privacy policies for Android, having a dedicated app like Speedtest with a transparent consumer experience and clear user permissions around location data collection is also increasingly important.
Ookla provides a reliable, consistent test experience across devices
Speedtest provides an accurate, consistent test experience that consumers trust across the many device types available on today’s market. Our rigorous methodology applies to all of our applications.
Testing to the right server eliminates latency and bottlenecks that can skew performance metrics.
Each link and node through which data is transferred can affect the final measurements as the link with the most constraining characteristics (highest latency, lowest bandwidth or highest packet loss) will typically limit the final measurements. Therefore, the fewer links between a device and a server, the more relevant the measurement is to quantifying and understanding the networking capability of a particular device.
Each Speedtest connects to a nearby server in Ookla’s global network of over 10,000 servers in more than 190 countries. This local connection allows us to ensure an accurate view of network performance that isn’t tarnished by external factors. Ookla takes comprehensive steps to ensure our traffic is indistinguishable from other applications or browser traffic to the server.
Testing to a CDN alone does not provide an accurate picture of network performance
Although CDNs serve a large portion of internet content, platforms that test to a content delivery network (CDN) only test to a single provider, often in a distant location, which only measures the connection to that specific CDN.
Speedtest measures the last-mile service provided to the end user by their ISPs and mobile operators. The last mile is the part of a user’s internet experience that a provider has the most control over — and responsibility for. Consumers can more accurately measure and troubleshoot the network connection they’re actually paying for, based on the location from which they are testing, because Speedtest connects to dedicated local servers. This allows us to provide the most accurate quality of service (QoS) measurement possible and uniquely positions us to evaluate the service provided by every ISP and mobile operator in the world — including removing the variability that comes with measuring CDN performance.
We also recognize the value in measuring a user’s performance beyond the last mile, so Speedtest allows testing to various providers in diverse locations. In this way, users can assess various connection scenarios to understand the full potential of a connection instead of being limited to a single provider. Only Ookla allows you to detect points of network congestion, and gain a better understanding of internet performance as a whole.
Speedtest is uniquely capable of measuring the full throughput capacity of 5G and super-fast fiber connections
Modern network speeds are increasing rapidly with technologies like 4G, 5G and fiber broadband being deployed across the globe. This makes the ability to measure a network’s full capacity more essential than ever. Speedtest uses a client and server testing engine that dynamically scales the number of connections to the server in order to saturate and accurately measure client-side connections up to 10 Gbps. This enables Ookla to overcome the effects of network bottlenecks and measure the full extent of a network’s performance.
Accurate 5G connection detection and identification
Not all 5G-capable devices natively identify the 5G connection type when reporting to applications. That’s why Ookla has directly partnered with device manufacturers worldwide to implement accurate in-app 5G detection in Speedtest. Consumers can reliably see when they’re testing a 5G connection in the Speedtest app, and providers can reliably measure their customers’ 5G network performance and quality.
Testing coverage, availability and quality of service
Fast speeds only matter on mobile when you actually have coverage. Consumer-initiated testing is the gold standard for speed and other performance metrics, but to measure signal and coverage Speedtest also collects over 300 million daily scans of coverage data in the background, submitted by Speedtest Android app users. These coverage scans capture where service is offered, what the quality of service is at each location, and information about a mobile user’s “radio environment,” including: the technology used (e.g., 5G, 4G LTE, etc.), the cellular infrastructure to which they are connected, and the accompanying strength and quality of signal.
Combined, consumer-initiated testing and coverage scans provide an unparalleled amount of data on performance and coverage that fully describes the quality of a user’s network connection. Here is an example of that complementary relationship in Brazil:
The largest volume of consumer-initiated tests
Numbers matter in data collection, and it’s important to look deeply at what the numbers represent. If a testing provider were to run 100 background tests per day on 100 phones for 100 days that record 100 values each, they’d have 1 million measurements. That overall number might sound impressive — until you realize that 100 phones do not give you a very wide (or interesting) distribution of phone types, locations or even experiences. The numbers are even less impressive when you realize that the 100 data points they’re collecting include less significant details like the device’s screen width and battery type that are only interesting to tiny cross sections of our industry.
Each and every day, over 10 million tests are actively initiated by pressing the “Go” button on Speedtest, and we receive additional data from over 300 million coverage scans. We see daily results on almost every mobile and fixed broadband network in the world which provides us with a real-time view of how the internet is performing at a global scale. This constant flow of immense amounts of data allows us to precisely track how networks respond to events like large crowds, the capabilities of new devices, the impact of network upgrades and the rollout of new technologies like 5G.
Unbiased data and a statistical sampling methodology
We use a rigorous statistical sampling methodology to combat sampling bias and ensure data accuracy. Through consumer-initiated testing, Speedtest gives every user a voice to describe what the internet connection is like on their device at the times and places that matter most to them. Whether a user takes a Speedtest once per month or once per hour, our sampling methodology makes sure that each user’s voice is heard and is not drowned out by high-volume testers.
When we aggregate data, each unique Speedtest user’s results are averaged to create a single sample that summarizes their internet experience for that time period and geographic area. We then evaluate each service provider based equally on the samples provided by each of its users. This removes the potential for results to be skewed by individual outliers or short-term fluctuations in service or user behaviors.
To ensure that our results represent the true commercially-available user experience, measurements from some tests are excluded from aggregation and published results, including tests performed in controlled environments by network engineers, tests taken from our CLI tool and results from the following platforms: mobile web tests taken on Speedtest.net, tests on Windows phones and tests using the Chrome app and our embedded solutions. Our data scientists and analysts employ a host of tools that allow us to identify and remove any tests that could intentionally or unintentionally bias our results.
For definitions of basic internet measures like download, upload, latency and jitter, visit our glossary.
Ookla uses additional performance metrics to describe real-world internet performance and coverage. Our metrics help consumers to understand their network performance and service providers to understand their customer experience, gain competitive insights and identify how to improve their networks.
An operator or ISP must account for 3% or more of total test samples in the market for the period to be designated as a top provider. We use this designation to ensure that most consumers in the area actually have access to the provider to qualify it as the fastest, most consistent or having the best coverage.
Ookla closely monitors the launch and widespread release of new device models built on chipsets capable of utilizing the latest network technologies. When calculating Speed Score™ and determining the winners of Speedtest Awards, we use results from devices built on modern chipsets so that an operator’s score is not negatively impacted if a portion of its subscriber base continues to use older technology.
We created Speed Score to fully account for the range of speeds a provider offers in a single metric. This makes it easier to compare mobile operators and ISPs on several measures of speed using one metric.
Speed Score incorporates a measure of each provider’s download and upload speed to rank network speed performance (90% of the final Speed Score is attributed to download speed and the remaining 10% to upload speed because online experiences are typically more affected by download speed). Speed Score uses a modified trimean to combine speeds from the 10th percentile, 50th percentile (also known as the median) and 90th percentile in a weighted average using a 1:8:1 ratio, respectively. We place the most emphasis on the median speeds as those represent what most network providers’ customers will experience on a day-to-day basis.
Not all providers serve the same geographic area. Some focus their efforts more in urban areas, where economies of scale make fast speeds easier and cheaper to provide. Others also serve vast rural areas, where it’s more difficult and expensive to provide fast speeds. Ookla’s Competitive Geography filter ensures a balanced comparison of U.S. mobile operators by eliminating geographic outliers.
To meet the definition of “competitive,” a ZIP code must contain samples from at least three top national competitors (those who have at least 3% of market share at a national level), but no competitor can have more than 2/3 of the samples in that ZIP code. Operators are considered present in a ZIP code if they have at least 3% of the samples in the area and show samples on multiple devices. Limiting any operator from having more than 2/3 of samples ensures actual competition in a ZIP code rather than including areas where one competitor dominates the market.
While fast speeds are paramount, a consistent experience is also a worthy measure of a network’s quality. Consistency Score is the metric we use to identify fixed broadband or mobile networks that provide a consistent quality of service. It reflects the percentage of a provider’s data samples that meet minimum thresholds for download and upload speeds, depending on the type of network. Consistency Score for fixed broadband uses thresholds of 25 Mbps minimum download speed (the recommended speed for streaming in 4K) and 3 Mbps minimum upload speed. Consistency Score for mobile uses thresholds of 5 Mbps minimum download speed (the recommended speed for streaming in HD) and 1 Mbps minimum upload speed. The higher a provider’s Consistency Score, the more likely a consumer will enjoy acceptable internet performance and quality.
We use “mean” and “average” interchangeably unless specifically stated otherwise.
Coverage metrics for mobile broadband
An operator’s geographic coverage is determined using a sample of scans received from devices on that operator’s network in each 100 m2 area. Because coverage is a spatially-focused metric, only scans with precise and legitimate location information are included as we build samples that normalize data by user, operator, location and timeframe. Coverage and availability metrics require that scans have been received from multiple devices in each area represented.
We created Coverage Score to account for both the quality and coverage of service for mobile operators. Coverage Score multiplies the proportion of locations in which an operator was seen with service (its footprint) against the average tile quality score (based on availability metrics) among all locations in which that operator is present. Coverage Score has a range of 0 to 1000 to avoid any potential for confusion that the Coverage Score represents a percentage of an area or population with coverage.
Coverage Score is not comparable across different countries because it is strongly tailored to the unique geography of each market.
To calculate Coverage Score, we use Footprint, which is the fraction of locations within a given market, across all operators in that market, where a device has access to service. Operators with a relatively small footprint will have a lower Coverage Score than competitors with equal availability and a larger footprint. we require a footprint of at least 30% in an operator’s market before we will calculate a Coverage Score to ensure that an operator is generally available to the public.
We divide our data on availability into three separate views: General Availability, 4G Availability, and On-Network Availability. These metrics indicate how likely a user, on average, is to have service available in the places they go. They are individually defined as:
- General Availability
The percentage of an operator’s known locations where a device has access to any kind of service (including roaming).
- 4G Availability
The percentage of an operator’s known locations where a device has access to 4G LTE service (including roaming).
- On-Network Availability
The percentage of an operator’s known locations where a device has access to service from that operator.
We use Time Spent to give mobile operators insight into the percent of time that an average user spends on a given cellular technology — both on and off of their subscriber (SIM) network. Time Spent uses coverage scans to construct daily timelines for each device. Then, daily per-device durations on subscriber network, active network and cellular technology are calculated for a given area of interest. Next, the percent time spent on cellular technology per-device, based on the area and time period of interest, are determined. Finally, we aggregate the average percent time over devices for a subscriber network to give the final metrics seen in Speedtest Intelligence™.
We hope this reference guide will give you deeper insight into the metrics we present on Insights. You can also download an extended version here. We also have further information on 5G and a glossary that contains some general terms used when discussing internet performance across the industry. For specific information about what our data has to say about your market, learn more about Speedtest Intelligence.