MLab speed test is incorrect?

[David Radin]

I'm on my office network, which is not heavily utilized and has a 75/75 fiber connection to the internet. On speedtest.net or dslreports.com speed test, I get results that are in the neighborhood of 70/65, which is what I'd expect when an office of 40ish people are using the internet.

MLab's test consistently shows speeds in the 12/6 range. Why would its results be so much slower?

[Ron Dallmeier]

Latency. The mlab server you are testing against is likely not nearby from a network perspective, while the speedtest sites are hosted in many places and quite likely your ISP is even hosting one.

...Ron

On Thu, Jan 26, 2017 at 10:17 AM, David Radin <david...@gmail.com> wrote:

I'm on my office network, which is not heavily utilized and has a 75/75 fiber connection to the internet. On speedtest.net or dslreports.com speed test, I get results that are in the neighborhood of 70/65, which is what I'd expect when an office of 40ish people are using the internet.

MLab's test consistently shows speeds in the 12/6 range. Why would its results be so much slower?

--
You received this message because you are subscribed to the Google Groups "discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@measurementlab.net.
To post to this group, send email to dis...@measurementlab.net.
Visit this group at https://groups.google.com/a/measurementlab.net/group/discuss/.

[Kim Prince]

It seems to me that the usual speed testing sites are well suited to testing the user's 'last mile', and perhaps other 'reasonably local' elements of their connection.  By contrast, the M-Lab tests are designed to help the user understand their overall internet performance, i.e. what performance they can expect when downloading/uploading to a 'representative' location on the internet.

By testing with the usual sites, you can confirm that your ISP is living up to their promised connection speed.  To understand how well that ISP is 'interconnected' to the rest of the web however, you need to consider the M-Lab results.

I picked this up from reading the M-Lab page at http://www.measurementlab.net/observatory/#tab=help&, particularly under the heading 'The Internet'.  Am I on the right track?

[☕Peter Boothe]

The intent of the Measurement Lab speed test is to test your Internet speed from your location to places where content lives. This is why Measurement Lab test servers are hosted in data centers that also contain major content providers.  On the other hand, as Kim Prince mentioned, many other speed tests attempt to measure only your last-mile performance, and as such are frequently hosted within the ISP they are testing.

Unless you are getting geographically mis-routed to a cluster that is unreasonably far away from you (possible, but hopefully unlikely) then it sounds like you are connected at high speeds to your local ISP, but your connection through that ISP to content on the Internet may not be as good due to latency (more likely) or congestion along the path (less likely, but also possible)*.

* - There are of course more than two opportunities, and network people are often pedantic, so let me note here that there are other things that could cause the slowdown. But latency and congestion are the two most likely culprits.

  -Peter

ᴹ̶LAB | Measure the Internet, save the data, and make it universally accessible and useful.

[Ron Dallmeier]

From the FAQ:

Why are my M-Lab results different from other speed tests?

Internet performance tests may provide different results for a lot of reasons. Three of the main reasons for different results among tests are listed below:

1. Differences in the location of testing servers

Every performance test has two parts:

• client: This is the software that runs on the user’s machine and shows the user their speed results.
• server: This is the computer on the Internet to which the client connects to complete the test.

A test generates data between the client and the server, and measures performance between these two points. The location of these two points is important in terms of understanding the results of a given test.

If the server is located within your Internet Service Provider’s (ISP’s) own network (also known as the “last mile”), this is referred to as an “on-net” measurement. This approach lets you know about how your Internet connection is performing intra-network within your ISP, but it does not necessarily reflect the full experience of using the Internet, which almost always involves using inter-network connections (connections between networks) to access content and services that are hosted somewhere outside of your ISP. Results from on-net testing are often higher than those achieved by using other methods, since the “distance” traveled is generally shorter, and the network is entirely controlled by one provider (your ISP).

“Off-net” measurements occur between your computer and a server located outside of your ISP’s network. This means that traffic crosses inter-network borders and often travels longer distances. Off-net testing frequently produces results that are lower than those produced from on-net testing.

M-Lab’s measurements are always conducted off-net. This way, M-Lab is able to measure performance from testers’ computers to locations where popular Internet content is often hosted. By having inter-network connections included in the test, test users get a real sense of the performance they could expect when using the Internet.

2. Differences in testing methods

Different Internet performance tests measure different things in different ways. M-Lab’s NDT test tries to transfer as much data as it can in ten seconds (both up and down), using a single connection to an M-Lab server. Other popular tests try to transfer as much data as possible at once across multiple connections to their server. Neither method is “right” or “wrong,” but using a single stream is more likely to help diagnose problems in the network than multiple streams would. Learn more about M-Lab’s NDT methodology.

I would suggest a couple of things. When you run the m-lab test, note which server it is running the test against. It shows you as the test is running. It would be something like: NDT.IUPUI.MLAB1.ORD02.MEASUREMENT-LAB.ORG

ORD is the airport code for Chicago O'Hare in this example. It is certainly important to note, which server it used.

Another suggestion would be to select the "Details" from the results page. In here you will get a summary of the test results. This can shed some light as to whether the test observed any potential causes for bad results. Two big factors for TCP based throughput are Packet Loss and Latency. Here is an example:

Your system: -
Plugin version: - (-)

TCP receive window: 893408 current, 894848 maximum
0.00 % of packets lost during test
Round trip time: 21 msec (minimum), 68 msec (maximum), 33 msec (average)
Jitter: -
0.00 seconds spend waiting following a timeout
TCP time-out counter: 232
473 selective acknowledgement packets received

No duplex mismatch condition was detected.
The test did not detect a cable fault.
No network congestion was detected.

0.9633 % of the time was not spent in a receiver limited or sender limited state.
0.0000 % of the time the connection is limited by the client machine's receive buffer.
Optimal receive buffer: - bytes
Bottleneck link: -
469 duplicate ACKs set

...Ron

[David Radin]

Thanks, all. 

[ma...@hoppes.us]

Sorry, but I don't buy this for one second.

While it's true any number of factors can affect your Internet service speeds - I can run a speedtest.net speedtest to a server well outside my local geographic area and get the expected speeds.  Same with openspeedtest.com  Neither of these are local to the network.

Run an MLab and the results are 3-4megabits while on a 65megabit connection.  Someone at MLab needs to fix their servers.

[ma...@hoppes.us]

Running a second test I got 3megabits the first time and 60 the second to Washington D.C.  Explain please?

[Collin Anderson]

Hello Discuss List,

My apologies for not following up on this thread publicly. We had received a couple of similar emails at the same time to our support address and replied directly rather than publicly. However, since this is a public list I wanted to provide a bit more clarity in a way that I hope may be broadly useful, rather than address one particular network. 

While there are a number of speed tests that are available, they all have differences related to measurement methods and server placement that lead to divergent numbers. There are a number of papers out there that attempt to account for these differences, and we have tried to surface some of those themes in the FAQ in a manner that is comprehendible to the lay reader. M-Lab has always taken the position that 'more is better,' even where M-Lab isn't used, and that the critical factor is understanding why those differences exist and how they tell different stories about accessibility and performance.

On occasion we receive inquiries from Internet service providers, typically small-to-medium networks that are interested in why their numbers are different from other tests. What we often find is that the issue is related to peering and topology between those providers and the networks that we are located in. M-Lab chooses prominent transit and content providers as neutral places for our sites that should reflect user experience – the names well known among this community, and not arcane locations. It's difficult to remotely diagnose particular networks from the other side, however, we generally find that in those cases where there are lower-than-expected numbers, some intermediary provider is experiencing degraded conditions, e.g. packet loss and higher latency. In one of the recent cases, we found that there were two or three networks between us and the provider, and there appeared to be substantial loss on that ISP's upstream – neither a failure with M-Lab nor within their network, but a real issue facing users. 

In the past three years, we have expanded our presence in each market to include multiple sites, covering different providers for each location. This was meant to be a diagnostic resource for the community. I realize that the "Washington, D.C." label in some of the integrations of NDT is not so descriptive of these variables in a way that would better enable accounting of differences for those who know – especially when for each refresh of the page, D.C. might mean Level 3, Tata, X.O, etc. As I understand some other tests select servers based on latency, whereas most integrations of M-Lab and NDT randomly select a server within the same geographic location. Sometimes this sole difference is the most significant factor in the result, but both tests are correct within their context. This topology and selection factor seems like a more prevalent matter for smaller ISPs that are reliant on one or two networks for transit – in those cases, one directly-connected network will perform well, whereas others where there are additional intermediaries in the path do not perform as expected. 

So, rather than platform issues, we tend to find those differences result from the more common infrastructure challenges that all operators contend with, which does have the impact on consumers reflected in the measurement. When questions arise, we encourage networks to check their peering between them and our sites in BGP, run traceroutes (check reverse traceroutes in M-Lab's BigQuery dataset), and conduct comparative tests across all sites in the region to see how they perform differently. 

We have long meant to expose more technical information about our platform for those who are interested in metadata and documentation. As we update our documentation, we will aim to also have a "FAQs for Operators" that more clearly lays out how methodological and topological difference lead to different-than-expected results, and how to use M-Lab as a diagnostic resource. In the interim, we take such questions quite seriously and try to provide support to operators where we can. 

Please feel free to reach out directly to me and our support address (support@) in the future. 

Cordially,

Collin Anderson

[dominic...@yarris.com]

I'm getting unexpected results from the test. My upload speed is twice that of my download speed.

I don't believe that is correct.

[xgma...@gmail.com]

Modern modems are run with multiple channels now. You're one internet connection can be a 2 or 3 separate connections or data channels tied together in a bundle to give you speeds faster than the technology alone. An example of this is WOW's Docsis modems. With a Docsis 2 modem you have 2 channels and speeds up to 60mbps. Docsis 3 modem 3 channels are now used and speeds can be upwards of 100mbps. 

Measurement Labs does not support this technology which I think is a crock because all the major ISPs have shifted to this methodology for high-speed connections. I have stopped using their testing platform because it only runs on one channel and only effectively tests 1 third of my connection capabilities. I believe that Measurement Labs is an obsolete service and no longer offers valuable real-time information. This information is available on their own website if you read through their FAQ. If you're still using a 2-20mbps connection it may still be a valid test but in this day in age, it's just obsolete.

On Thursday, January 26, 2017 at 11:17:17 AM UTC-5, David Radin wrote:

[Livingood, Jason]

On 1/18/18, 10:59 AM, "xgma...@gmail.com" <xgma...@gmail.com> wrote:

 

> Modern modems are run with multiple channels now. You're one internet connection can be a 2 or 3 separate connections or data channels tied together in a bundle to give you speeds faster than the technology alone. An example of this is WOW's Docsis modems. With a Docsis 2 modem you have 2 channels and speeds up to 60mbps. Docsis 3 modem 3 channels are now used and speeds can be upwards of 100mbps. 

 

[JL] FWIW, according to a 2016 Arris report (https://www.arris.com/globalassets/resources/white-papers/scte-future-directions-for-fiber-deep-hfc-deployments.pdf) in the near future most DOCSIS 3.0 networks will move to 32 bonded 3.0 channels and 4 DOCSIS 3.1 OFDM channels. This enables 1 Gbps services, which Comcast and other ISP networks are providing over DOCSIS networks (obviously also over fiber, and there are many other fiber-based ISPs). I don’t think there are many measurement tools that can adequately measure those speeds at scale (such as the scale at which Ookla operates).

 

> Measurement Labs does not support this technology which I think is a crock because all the major ISPs have shifted to this methodology for high-speed connections. I have stopped using their testing platform because it only runs on one channel and only effectively tests 1 third of my connection capabilities. I believe that Measurement Labs is an obsolete service and no longer offers valuable real-time information. This information is available on their own website if you read through their FAQ. If you're still using a 2-20mbps connection it may still be a valid test but in this day in age, it's just obsolete.

 

[JL] Steve Bauer from MIT has done some thinking about this. See his section on the M-Labs test in https://www.measurementlab.net/publications/understanding-broadband-speed-measurements.pdf (Section 4.2.5) and his presentation on testing in the gigabit era at https://www.caida.org/workshops/aims/1602/slides/aims1602_sbauer.pdf.

On Thursday, January 26, 2017 at 11:17:17 AM UTC-5, David Radin wrote:

I'm on my office network, which is not heavily utilized and has a 75/75 fiber connection to the internet. On speedtest.net or dslreports.com speed test, I get results that are in the neighborhood of 70/65, which is what I'd expect when an office of 40ish people are using the internet.

 

MLab's test consistently shows speeds in the 12/6 range. Why would its results be so much slower?

--

You received this message because you are subscribed to the Google Groups "discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@measurementlab.net.

[xgma...@gmail.com]

All of that is assuming the ISP has a gigabit capable network in the area which ours does not. I was told specifically that ours runs on three channels. I have the same ISP at home with the same modem and I get exactly the same results at almost exactly 1/3rd of my capacity. I'm just trying to offer this as an explanation for the results the other user posted. I have seen the exact same thing and found an explanation for it. Also I wouldn't focus on speed and latency out on the web as much as I would good house keeping on my own router. I've found the most impact I've had on my networks with all my testing and tweaking has been to get my buffer-bloat under control which has made noticeable differences in our speeds and reduces connection timeouts to nearly non-existent. My point is Measurement Labs played no part in our improvements other than to throw us off and confuse us making us think there were issues when according to EVERY other speed/network check, we were right on the money. I measure RESULTS, not fanboyism for an outdated service. Thousandeyes helped me verify the bull%$@ that is ML. 

[David Radin]

ML measures the "real world" throughput, where peering connections between ISPs can become a bottleneck.

DSLReports and Ookla tend to measure the "last mile" throughput of the connection between customer premises and the ISP's headend.

Both are valid approaches. It just depends what you're trying to determine.

--
You received this message because you are subscribed to a topic in the Google Groups "discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/a/measurementlab.net/d/topic/discuss/vOTs3rcbp38/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss+unsubscribe@measurementlab.net.

[Srikanth S]

On Thu, Jan 18, 2018 at 11:15 AM, David Radin <david...@gmail.com> wrote:

ML measures the "real world" throughput, where peering connections between ISPs can become a bottleneck.

This is debatable, due to the ML infrastructure. We looked at interconnects between access ISPs and ML targets and access ISPs and Alexa 500 sites, and found there isn't much overlap. It's not a given that ML is measuring real-world throughput.

See 5.3 here: https://conferences.sigcomm.org/imc/2017/papers/imc17-final100.pdf

 

[Woundy, Richard]

> An example of this is WOW's Docsis modems. With a Docsis 2 modem you have 2 channels and speeds up to 60mbps. Docsis 3 modem 3 channels are now used and speeds can be upwards of 100mbps.

 

For this thread, are we talking about performance testing over DOCSIS 2, or over DOCSIS 3?

 

DOCSIS 2 uses load balancing to distribute traffic among multiple channels, whereas DOCSIS 3 uses channel bonding.

 

As I understand, ML’s NDT performance test uses a single TCP connection. (Also confirmed in section 4.2.5.1 of the MIT paper cited below.)

 

DOCSIS 2 uses load balancing, so the NDT TCP connection packets will be transmitted over only one of the DOCSIS channels (to keep packets in order), and that channel may be more or less congested than other channels.

 

DOCSIS 3 is more likely to use channel bonding than load balancing, so the NDT TCP connection packets would be striped (ie inverse-multiplexed) across all available channels for transmission.

 

This is less of an issue for Speedtest/Ookla because that performance test uses multiple HTTP threads (aka parallel TCP connections), and they can be load balanced across multiple channels. (see section 4.2.2.1 of the MIT paper).

[Nick Feamster]

> On Jan 22, 2018, at 4:48 PM, Woundy, Richard <Richard...@comcast.com> wrote:
>
> This is less of an issue for Speedtest/Ookla because that performance test uses multiple HTTP threads (aka parallel TCP connections), and they can be load balanced across multiple channels. (see section 4.2.2.1 of the MIT paper).

I’ll also add that the BISmark test has used multiple TCP threads since 2010, as well.

Even with channel bonding, a single-threaded test would have trouble filling a link to capacity. We saw this in experiments that are detailed in our SIGCOMM 2011 paper.

The inability of a single-threaded TCP connection to fill the link to capacity seems (1) independent of the underlying physical medium (i.e., cable, DSL, fiber); (2) to be present even for access links _well_ below Gigabit speeds.

-Nick

[trump...@gmail.com]

I just use 3 services for reliability http://www.speedtest.net/, https://speedof.me/ and http://internet-speed-test.online

четверг, 26 января 2017 г., 18:17:17 UTC+2 пользователь David Radin написал:

[James Miller]

The suite of tests run on Measuring Broadband America Program supported by Samknows are also multithreaded.  May be also worth noting that the number of threads are really import for understanding limits of TCP based tests but for UDP tests, for latency for example, the total number of datagrams exchanged has implications for accuracy and precision that can be attained by a given test.  

A well documented, public methodology for tests is critical to understanding measurement results.  

--
James Miller, Esq.

"Japanese is so Eighties..."
Anonymous FCC Colleague

--
You received this message because you are subscribed to the Google Groups "discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+unsubscribe@measurementlab.net.

[Julian Johnston Jr.]

I know this post  is old, but every time i use mlab, the measurement is too distant from what I have, just used the service, got 41mb from a 400mb speed but upload was correct, using other test I have seen 480mb, mlab has never been past 41mb.

On Thursday, January 26, 2017 at 11:17:17 AM UTC-5, David Radin wrote:

[Chris Ritzo]

There are multiple reasons why our test returns measurements that are slower than other tests. We outline a few of them in this FAQ article. You might also be interested in this blog post about speed tests, accuracy, and our primary performance measurement test, NDT.

One reason for the differences is that our servers are hosted outside last mile access networks, in data centers where networks "peer" or interconnect with one another. When providers are well peered, we see closer parity with other speed tests which often host servers at the edge of their last mile networks. The other major difference is that the architecture of our test itself differs from other tests. Our test is a single TCP stream test where others use multiple TCP streams. The use of multiple TCP streams emulates how web browsers typically operate when downloading resources you request from the Internet. Our NDT test's measurement using a single stream conforms more closely to accepted standards for performance measurement.

I hope this provides some more context for the differences in measurements from our test versus others.