Internet Speed Test results are at odds with other speed test services.

[Cory Rahman]

Results (tested multiple times)

SOURCE / DL mbps / UP mbps

On my home 5Ghz network:

MLAB speed test / 40 / 76

Ookla / 83 / 84

AT&T / 82 / 83

Verizon / 82 / 89

On my home 2.4Ghz network:

MLAB speed test / 45 / 62

Ookla / 82 / 86

AT&T / 82 / 88

Verizon / 81 / 90

Does anyone know why these MLAB speed test results would be so different than the other online speed tests?

[Cory Rahman]

I did come across this page:

https://www.measurementlab.net/faq/#why-are-my-m-lab-results-different-from-other-speed-tests

Which seems to claim that the M-Lab tests are more accurate. Is this the case? Are the other 3 speed tests inaccurate?

[Livingood, Jason]

IIRC from prior discussions here, I think it was due to a single connection vs. multiple connections for the other tests.

 

Jason

--
You received this message because you are subscribed to the Google Groups "discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@measurementlab.net.
To post to this group, send email to dis...@measurementlab.net.
Visit this group at https://groups.google.com/a/measurementlab.net/group/discuss/.

[Sascha Meinrath]

This may be an artifact of multi-threading; but it may also reflect on- vs.
off-network test servers (i.e., are you testing the connection within an ISPs
network or to a location that is outside of that single ISPs system).

--Sascha

On 08/21/2018 11:30 AM, Livingood, Jason wrote:
> IIRC from prior discussions here, I think it was due to a single connection vs.
> multiple connections for the other tests.
>
>  
>
> Jason
>
>  
>

> *From: *Cory Rahman <cory....@gmail.com>
> *Date: *Monday, August 20, 2018 at 9:10 PM
> *To: *discuss <dis...@measurementlab.net>
> *Subject: *[EXTERNAL] [M-Lab-Discuss] Internet Speed Test results are at odds

> with other speed test services.
>
>  
>
> Results (tested multiple times)
>
>  
>
> SOURCE / DL mbps / UP mbps
>
>  
>
>  
>
> On my home 5Ghz network:
>
>  
>
> MLAB speed test / 40 / 76
>
> Ookla / 83 / 84
>
> AT&T / 82 / 83
>
> Verizon / 82 / 89
>
>  
>
>  
>
> On my home 2.4Ghz network:
>
>  
>
> MLAB speed test / 45 / 62
>
> Ookla / 82 / 86
>
> AT&T / 82 / 88
>
> Verizon / 81 / 90
>
>  
>
>  
>
> Does anyone know why these MLAB speed test results would be so different than
> the other online speed tests?
>
>  
>
>  
>
>  
>
> --
> You received this message because you are subscribed to the Google Groups
> "discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to discuss+u...@measurementlab.net

> <mailto:discuss+u...@measurementlab.net>.

> To post to this group, send email to dis...@measurementlab.net

> <mailto:dis...@measurementlab.net>.

> Visit this group at https://groups.google.com/a/measurementlab.net/group/discuss/.
>

> --
> You received this message because you are subscribed to the Google Groups
> "discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to discuss+u...@measurementlab.net

> <mailto:discuss+u...@measurementlab.net>.

> To post to this group, send email to dis...@measurementlab.net

> <mailto:dis...@measurementlab.net>.

[Chris Ritzo]

Hi Cory,

As others have mentioned, there are a variety of factors which are in play here. The FAQ you reference explains several reasons why the M-Lab test reports lower results than other speed tests, including single versus multi-threaded tests as Jason mentioned, and on-net versus off-net placement of servers as Sascha mentioned. M-Lab doesn't claim that our test is more accurate than others, but that it is measuring different network conditions. Different network performance tests use differing methodologies as well, which can account for different measurements. All the tests are as accurate as they can be for what they are measuring, and because they report similar metrics it is often assumed that their results should be equal.  I would characterize them as comparable and each accurate in their own right for what is being measured. The M-Lab NDT test measurements are more in line with Akamai tests, in that the server endpoints are off-net. Whether a single threaded test or a multi-threaded test is more accurate is still a question up for debate in the academic internet measurement community-- the article Jason referenced is one supporting multi-threaded tests, and earlier research points to single-threaded testing to be more precise in assessing overall performance. In any case, whatever the test used, they all provide some sense of a connection and I think that combined the results of each one can help paint a nuanced picture of your quality of access overall.

Best regards,
Chris

--

Chris Ritzo

Measurement Lab Operations & Support

o...@measurementlab.net | sup...@measurementlab.net

Senior Technologist, Open Technology Institute @ New America

740 15th Street NW, Suite 900 

Washington, DC 20036      

--

You received this message because you are subscribed to the Google Groups "discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@measurementlab.net.
To post to this group, send email to dis...@measurementlab.net.

[Jim Partridge]

This clearly seems to be an issue of multi-threaded tests versus the single-threaded M-Lab NDT test. I would point you to the Comments that Nick Feamster filed with the FCC in the Restoring Internet Freedom docket (link also pasted below) which included the observation that - The M-Lab NDT test consistently underestimates access link throughput. Off-net versus on-net servers aren't going to result in performance metrics that are half of actual capacity. It's certainly interesting to note how closely aligned the results are from the three other tests, Ookla, AT&T, and Verizon. Of note, Professor Feamster also states that "As access link speeds continue to increase...the underestimation {of the single-threaded NDT test} is likely to become even more severe."

https://ecfsapi.fcc.gov/file/1083088362452/fcc-17-108-reply-aug2017.pdf

[Simone Basso]

In the infosec community it often happens that people asking security questions is asked about their threat model. Likewise, I'd say that perhaps in this community we should help people making questions by consistently asking what they would like to measure.

AFAICT, it's non-controversial that parallel streams tests better approximate the last mile speed, however they also fail to reveal losses in the network (or elsewhere) that become obvious with a single stream test. In the same vein, on-net and off-net tests have pros and cons.

Also, speaking of cognitive dissonance, seeing a much lower than expected result from a single stream test can be as surprising as seeing an all-green report from a multi stream test when interactive communication is stuttering, or streaming does not load.

Best,

Simone

--
You received this message because you are subscribed to the Google Groups "discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@measurementlab.net.

[Livingood, Jason]

I think it was Jim Partridge that cited the research, not me. 😉 But in any case that citation was relatively recent, as is the research from Steve Bauer et. al. on the subject of measurement tests. The citation on single threading you noted (from 1997 on “the macroscopic behavior of the TCP congestion avoidance algorithm”) is probably a bit outdated considering how substantially TCP congestion control and threading has changed in 21 years. But I know one of the co-authors (Matt Mathis) is on this list so he’d know more than anyone how all the recent changes may or may not affect their prior conclusions, since I am pretty sure he’s on the leading edge of congestion control R&D.

 

In any case, it seems you are right to focus on the right test for the right question; that is the root of it. It seems most folks think NDT is useful for identifying the location of network bottlenecks, rather than aggregate network capacity (speed). But it seems like a lot of people use it for the latter case when another tool may be superior for that question.

 

JL

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@measurementlab.net.

To post to this group, send email to dis...@measurementlab.net.
Visit this group at https://groups.google.com/a/measurementlab.net/group/discuss/.

 

--

You received this message because you are subscribed to the Google Groups "discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@measurementlab.net.

[Livingood, Jason]

+1, Simone. All very good points!

 

From: Simone Basso <basso...@gmail.com>
Date: Tuesday, August 21, 2018 at 1:43 PM
To: "dis...@measurementlab.net" <dis...@measurementlab.net>
Subject: [EXTERNAL] Re: [M-Lab-Discuss] Re: Internet Speed Test results are at odds with other speed test services.

 

In the infosec community it often happens that people asking security questions is asked about their threat model. Likewise, I'd say that perhaps in this community we should help people making questions by consistently asking what they would like to measure.

 

AFAICT, it's non-controversial that parallel streams tests better approximate the last mile speed, however they also fail to reveal losses in the network (or elsewhere) that become obvious with a single stream test. In the same vein, on-net and off-net tests have pros and cons.

 

Also, speaking of cognitive dissonance, seeing a much lower than expected result from a single stream test can be as surprising as seeing an all-green report from a multi stream test when interactive communication is stuttering, or streaming does not load.

 

Best,

 

Simone

 

Il giorno mar 21 ago 2018 alle ore 19:23 Jim Partridge <jpartr...@gmail.com> ha scritto:

This clearly seems to be an issue of multi-threaded tests versus the single-threaded M-Lab NDT test. I would point you to the Comments that Nick Feamster filed with the FCC in the Restoring Internet Freedom docket (link also pasted below) which included the observation that - The M-Lab NDT test consistently underestimates access link throughput. Off-net versus on-net servers aren't going to result in performance metrics that are half of actual capacity. It's certainly interesting to note how closely aligned the results are from the three other tests, Ookla, AT&T, and Verizon. Of note, Professor Feamster also states that "As access link speeds continue to increase...the underestimation {of the single-threaded NDT test} is likely to become even more severe."

 

Error! Filename not specified.