HOW OOKLA ENSURES ACCURATE, RELIABLE DATA

Conflicting news reports have been flying around trumpeting the speeds of one operator over another. The differences come down to different test methods, different data sets and different methodologies for analyzing that data. At Ookla we care deeply about providing data and analyses that are accurate and statistically sound so that consumers and businesses can trust the information they’re receiving. We’ve been doing this for over thirteen years.

While we continually refine our methodology and explore new ways of testing all aspects of internet performance, we stand by consumer-initiated testing as the best measure of network speeds. Equally important, the people who build the internet (including regulators, providers and nonprofit monitoring organizations) continue to trust Ookla’s data and methodology to make decisions that impact internet performance around the globe.

Consumer-initiated testing is the best measure of internet speed

Some people question why we favor consumer-initiated testing, claiming that other testing methods produce a higher volume of data or that consumer-initiated testing produces biased results. We’ll dive into our data volume and sampling methods below, but first it’s important to understand that a dedicated test initiated by a consumer and run in the foreground is the best measure of their internet performance capability. This is true for measures of download speed, upload speed, latency, packet loss, jitter, errors and other indicators of network conditions.

Testing the full capacity of an internet connection

Companies that rely on background tests hidden inside of other apps to measure speed only send small amounts of data back and forth. This is not a sound method for accurately measuring speeds. While useful for assessing metrics like signal and coverage, background tests are not able to flood an internet connection with a high enough volume of data to measure the full throughput capability.

Because Speedtest measures the full capacity of your connection, we can properly show off its full performance capability. This is why speeds measured with Speedtest are often higher than other methods and why users sometimes note that Speedtest uses a lot of data.

periodic-vs-dedicated-2

Testing when and where it’s relevant

We all have individual internet usage patterns, whether you need your connection on the way to and from the office, during the day at a specific location, or at the end of the day when you’re winding down at home. Background tests are run by the test provider on a schedule that suits their needs. In fact, these tests are often done in the moments you are least likely to be using the internet, such as on roads while driving, in locations like an elevator where your signal is diminished, and when you’re asleep. A consumer-initiated test like Speedtest gives users accurate information about internet speeds at the times and in the locations that are important to them.

Thoughtful data collection and analysis makes for accurate results

Now that you understand our testing methodology for speed measurements, let’s look at questions of data volume and how we counter bias in results.

The largest volume of consumer-initiated tests

Numbers matter in data collection, and it’s important to look deeply at what the numbers represent. If a testing provider were to run 100 background tests per day on 100 phones for 100 days that record 100 values each, they’d have 1 million measurements. That overall number might sound impressive until you realize that 100 phones do not give you a very wide (or interesting) distribution of phone types, locations or even experiences. The numbers are even less impressive when you realize that the 100 data points they’re collecting include less significant details like the device’s screen width and operating system version. Those might be interesting data points for tiny cross sections of our industry, but they contribute little valuable information about internet speed or performance at a global scale.

Each and every day, the “Go” button on Speedtest is actively pressed over 10 million times. We could delve into how many data points each of those tests returns, but we don’t need to because what matters most is the breadth and volume of users. We see daily Speedtest results on just about every mobile and fixed broadband network in the world, which provides us with a real-time view of how the internet is performing around the globe. This constant flow of immense amounts of data allows us to track how networks respond to large crowds, the capabilities of new devices, the impact of network upgrades, and the rollout of new technologies like 5G.

A reliable, consistent test experience across devices

Part of having a test that speaks to your individual needs is making sure that it works on your device. Speedtest provides an accurate, consistent test experience on all your devices. We have mobile apps for both Android and iOS that are optimized to measure the specific intricacies of a mobile connection. There are dedicated apps for computers (Mac and Windows), a browser extension for Chrome, integrations on consumer routers and even an AppleTV app. Our original web test at Speedtest.net covers most other platforms.

Speedtest-Platforms

Internet tests that don’t operate in the foreground are limited in what data they can collect on platforms like iOS. That means that aggregate speeds you might see from those entities are not adequately representing important segments of the market.

Nearby servers provide accurate results

Other apps have consumers test to a small group of servers that may or may not be close to them. Even if those apps received millions of tests from millions of different users during a quarter, the fact that they’re testing to limited locations would skew their results. This is critical, because without the ability to test to a nearby server, a person could be measuring internet connection speed to a server thousands of miles across the country or in another country entirely. As the distance between the user and server increases, the test is more likely to be affected by network conditions along that specific path that would otherwise have no impact on the rest of that user’s internet activity.

server-distance-1

Ookla’s vast network of over 8,000 global servers (1,200 in the U.S. alone) allows consumers to always test to one that’s physically near them, reducing travel time of the data as it communicates between the device and server.

server-network-flat-3

Unbiased data and a statistical sampling methodology

We use a rigorous sampling methodology to combat sampling bias and ensure data accuracy. Through consumer-initiated testing, Speedtest gives every user a voice to describe what the internet connection is like on their device at the times and places that matter most to them. Whether a user takes a Speedtest once per month or once per hour, our sampling methodology makes sure that each user’s voice is heard and is not drowned out by high-volume testers. When we aggregate data from all of the tests we’ve received, each user’s Speedtest results are averaged to create a single sample that summarizes his or her internet experience for that time period and geographic area. We then evaluate each service provider based equally on the samples provided by each of its users. This removes the potential for results to be skewed by individual outliers or short-term fluctuations in service or user behaviors.

For example, we recently saw a large spike in testing among AT&T users following a software update that introduced the 5Ge icon to a subset of iPhones. Although this change inspired users to take more tests than normal in order to determine whether the performance of their service had improved, these users still received one vote over the entire quarter in determining AT&T’s overall performance. So while these users may have had an oversized influence in the week or two after the update appeared, their impact on the larger time frame was muted by our sampling methodology.

Network engineers often use Speedtest while developing the latest technologies. To ensure that our results represent the true commercially-available user experience, these tests are not included in our published results. This is because these tests are often performed in controlled environments that do not necessarily reflect what their customers see (just yet, at least). Similarly, our data scientists and analysts employ a host of tools that allow us to identify and remove tests they suspect may be intended to bias our results.

How Ookla supplements our consumer-initiated speed data with coverage scans

As discussed above, consumer-initiated testing is the best measure for speed and several other metrics. Consumer-initiated testing is not the best tool, however, for measuring things like signal and coverage (which describes both where service is offered and what the quality of service is at each location). This is why Ookla allows Android users to opt in to an additional layer of measurement: coverage scans.

A user’s mobile internet experience is greatly impacted by coverage. With coverage scans, we also examine information about a mobile user’s “radio environment,” including the technology used (e.g., 4G LTE, 3G, etc.), the cellular infrastructure to which they are connected, and the accompanying strength and quality of signal. This allows us to get a better understanding of coverage in that area and to provide more information about why particular Speedtest results were seen. We use the hundreds of millions of coverage scans that we receive each day to provide an accurate and up-to-date understanding of the coverage offered by mobile operators. When a user opts in to coverage scans, we provide them with a useful personal coverage map like the one modeled here:

coverage-map-1

Consumer-initiated testing and coverage scans complement one another, and together they provide an unparalleled amount of data on performance and coverage. This is what that looks like in Brazil:

brazil-map

Ookla is trusted by regulators, providers, nonprofits and consumers across the globe

If the ins and outs of Ookla’s testing and sampling methodology are too much to absorb, it’s enough to know that our data is trusted by the people building, regulating, monitoring and using the internet.

When you encounter conflicting reports in the marketplace, investigate the sources with a critical eye. Our mission has always been to put accurate information about internet performance in the hands of consumers and service providers which has made Ookla the go-to company for data on internet performance for over thirteen years. By offering an independent, unbiased and scientific view we will continue to provide the kind of quality data and analysis that got us there while continuing to innovate along the way.

Disclaimer: The views and opinions expressed in this article/press release are those of the authors and do not necessarily reflect the approved policy or position of the GSMA or its subsidiaries.

Contact the GSMA

Please get in touch if you need more information or have any queries about anything you see on our website.

Contact us