Questions about MSAK Data Retrieval and Analysis

[Jiayi Liu]

Hello M-Lab community,

I've encountered some issues and questions while working with MSAK data. I'd greatly appreciate any insights or guidance on the following matters:

1. Querying the measurement-lab.msak.throughput1 table

When I try to retrieve mlab msak data, I first tried to query the measurement-lab.msak.throughput1 table like this to see its structure:

SELECT * FROM `measurement-lab.msak.throughput1` WHERE date BETWEEN '2024-09-25' AND '2024-09-27' LIMIT 1

I receive the following error: Cannot query over table 'mlab-oti.raw_msak.annotation2' without a filter over column(s) 'date' that can be used for partition elimination.

This is puzzling because I am using date as a partition. Could someone explain why this error occurs and how to resolve it?

2. Discrepancy in number of entries retrieved

I've recorded the IPv6 address and time for each device before and after each M-Lab speedtest. However, when I query the measurement-lab.msak_raw.throughput1 table using the IPv6 address and date, I'm getting more entries than the actual number of tests performed.

For example, with this query:

SELECT * FROM `measurement-lab.msak_raw.throughput1` WHERE date BETWEEN "2024-09-27" AND "2024-09-28" AND raw.Client LIKE '%specific_ipv6_address%' AND raw.Direction = 'download' ORDER BY date DESC

I performed around 16 tests, but I'm getting 63 entries. This pattern repeats for other devices I've tested - about 15-16 tests yield around 60 entries.

Questions:

1. How should I filter these data?
2. Are all these entries valid?
3. If they are valid, how can I group them into individual tests, considering some entries' raw.startTime falls between two test sessions?

3. Calculating download/upload speeds

Lastly, I'd like to confirm the correct method for calculating download/upload speeds from the retrieved data. Currently, I'm using this approach:

• For download speed: (ClientMeasurements' last entry Application BytesReceived / ElapsedTime) * 8 to get results in Mbps
• For upload speed: (ServerMeasurements' last entry Application BytesReceived / ElapsedTime) * 8

Is this calculation method correct? Are there any direct ways to obtain these speeds from the data?

Thank you in advance for any help or clarification you can provide. Your insights will be invaluable for my research.

Best,

Jiayi Liu

[Roberto D'Auria]

Hi Jiayi,

Regarding querying measurement-lab.msak.throughput1: This is an issue introduced when we added a mandatory date filter on the annotation2 table, for which we have now deployed a fix. Thank you for reporting it!

Discrepancy in number of entries retrieved: There are key differences between the MSAK data and the ndt7 data, primarily because MSAK provides the option to choose the number of streams (more info in our blog post here). Since each TCP connection on the M-Lab platform also has corresponding information from the sidecar services (such as tcpinfo), to allow joining with sidecar data, one row in the MSAK table represents just a single TCP stream. To join all the streams belonging to the same test, you should use the MeasurementID field:

- MeasurementID: Unique per MSAK test
- UUID: Unique per TCP stream in an MSAK test

Since your tests were not single-stream, the number of rows is not expected to be equal to the number of tests, but to n_tests * n_streams_per_test. In your example, 16 tests * 4 streams per test = 64 rows.

Calculating download/upload speeds: The presence of multiple streams adds a layer of complexity in interpreting MSAK results, since they must be aggregated to compute the average rate. We have recently created the view measurement-lab.msak.throughput1_downloads that implements one of the possible approaches to do so. This view's query takes all the TCPInfo snapshots between the first stream that started and the first stream that ended for a given MeasurementID, then computes the average throughput as the sum of all the BytesAcked in the last snapshots for each stream divided by MAX(ElapsedTime) (* 8 for Mbit/s).

To compute download throughput, using TCPInfo.BytesAcked on the server side is generally preferable, since it doesn't rely on the client's reporting being accurate. If you need the download goodput (i.e. application-level) instead, you can use a similar approach but with Application.BytesReceived, as you mentioned. In both cases, you need to aggregate the streams first.

Hope this helps!

--
You received this message because you are subscribed to the Google Groups "discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@measurementlab.net.
To view this discussion on the web visit https://groups.google.com/a/measurementlab.net/d/msgid/discuss/0d0711a4-ac32-48a5-9333-066d8360d6d2n%40measurementlab.net.

--

Roberto D'Auria

Platform Engineer

Measurement Lab

Code for Science & Society

[Jiayi Liu]

Hi Roberto,

Thank you for your detailed explanation. I appreciate your insights and would like to clarify my experiment method and follow up on a few questions.

My experimental approach was as follows:
For each device, I first recorded its IPv6 address and the current time.
I then accessed https://speed.measurementlab.net through a browser (Firefox).
After the test results appeared (or after a certain time threshold), I recorded the IPv6 address and time again to mark it ended.
After completing a series of tests (about 15 for each device), I used all recorded IPv6 addresses and times to query rows in BigQuery.

I did not specifically configure the number of MSAK streams. The results I previously shared were all the data I retrieved using the IPv6 addresses and time.

Based on your explanation, am I correct in understanding that the current web-based MSAK tests on https://speed.measurementlab.net are single-stream? Upon reviewing my filtered data, I noticed that for the MeasurementIDs with 3 rows, their client metadata indeed doesn't include browser information like "firefox".

To obtain experiment results suitable for comparison with other apps (as shown in the image below), I have a few questions:

For NDT data:
Should I use MeanThroughputMbps for both Download and Upload speeds?
For latency, should I use the minRTT from the download test?

For MSAK data:
Should I focus only on the results with a single row?
For these single-row results, should I calculate the speed using Application.BytesReceived?

Thank you again for your help in clarifying these points. Your guidance is invaluable in ensuring the accuracy of my comparative analysis.

P.S. I've included the Discuss group in this response to share the information with the community.

Thanks,

Jiayi

On Mon, Oct 7, 2024 at 10:38 AM Roberto D'Auria <rob...@measurementlab.net> wrote:

This returned 40 rows. After grouping by MeasurementID as you suggested, I got the following result:

These results look OK to me. I've looked at the raw.ClientMetadata fields to find out the client: the single-stream tests are from speed.measurementlab.net where we're currently running a calibration experiment against ndt7 and testing MSAK with different configurations (at that time, MSAK was running with a single BBR stream). Tests with 3 streams did not provide a "client_name" querystring parameter -- if that's your own client, I recommend sending "client_name" so they are easier to identify in the BQ data.

Then, I filtered these rows based on the start and end times I recorded for each M-Lab test, keeping only the rows where raw.startTime falls within these intervals. Here's what I found after filtering:

I believe the system clock on the machine where the client ran might be off by ~40 seconds, which in this case is impacting your analysis. All the M-Lab servers run systemd-timesyncd to keep the clocks synchronized via NTP. Data saved in BigQuery is generated by the MSAK server and the timestamp is the one from the server's system clock. I looked at the mlab1-nuq08 machine (the one from MeasurementID "5b0a5e50-2f71-4a86-8392-b83c01b05004") and the system clock is currently accurate to the second. I recommend not using the exact start time recorded by the client to aggregate the data (or, at least, account for +/- 1 minute in the query if you need to do so)

I'm curious about your mention of 4 threads. Is it possible that they're being split into 1+3?

No, you're just observing a combination of single-stream tests (the ones from speed.measurementlab.net) and tests from your client, which is configured to use 3 streams. I previously mentioned 4 streams as an example, MSAK's number of streams is configurable by the client and can change on a test-by-test basis.

Regarding calculating download/upload speeds: Do you plan to create a unified view for MSAK data similar to the NDT? This would allow us to directly obtain download/upload speeds through a MeanThroughputMbps field, which would be very helpful for analysis.

We plan on adding an msak.throughput1_uploads view which only contains uploads, aggregated with the same methodology as msak.throughput1_downloads.

Let me know if you have any further questions!

P.S. I don't know if it was intentional, but your reply didn't go to the Discuss group but to me only. I did not cc the group, but feel free to do it again if needed!

-Roberto

On Thu, Oct 3, 2024 at 8:13 AM Jiayi Liu <miali...@gmail.com> wrote:

Hi Roberto,

Thank you for your detailed response. I'm glad the issue with querying measurement-lab.msak.throughput1 has been resolved.

Regarding the discrepancy in the number of entries retrieved, I've done some further investigation based on your explanation. Here's what I found:

I queried the MSAK data using this SQL:

SELECT
  *
FROM `measurement-lab.msak_raw.throughput1`
WHERE

date BETWEEN "2024-09-29" AND "2024-09-30"
  AND raw.Client LIKE '%2600:1010:b047:e1eb:f0a5:4cd4:4861:2b37%'
  AND raw.Direction = 'download'
ORDER BY date ASC

This returned 40 rows. After grouping by MeasurementID as you suggested, I got the following result:

Then, I filtered these rows based on the start and end times I recorded for each M-Lab test, keeping only the rows where raw.startTime falls within these intervals. Here's what I found after filtering:

Interestingly, each MeasurementID within the time intervals appears only once. I've verified this pattern across multiple devices.
I'm curious about your mention of 4 threads. Is it possible that they're being split into 1+3? If so, I'm wondering why, for example, the MeasurementID 5b0a5e50-2f71-4a86-8392-b83c01b05004 has 3 rows all with a Start Time of 2024-09-30 11:21:25 (PDT), which is earlier than the start time I recorded for my first experiment? I have attached the related files below.

Regarding calculating download/upload speeds: Do you plan to create a unified view for MSAK data similar to the NDT? This would allow us to directly obtain download/upload speeds through a MeanThroughputMbps field, which would be very helpful for analysis.

Thank you again for your assistance. I look forward to your insights on these follow-up questions.

Best,

Jiayi Liu

[Roberto D'Auria]

Based on your explanation, am I correct in understanding that the current web-based MSAK tests on https://speed.measurementlab.net are single-stream? Upon reviewing my filtered data, I noticed that for the MeasurementIDs with 3 rows, their client metadata indeed doesn't include browser information like "firefox".

We are running a calibration experiment there, so the MSAK configuration varies over time. MSAK tests were single stream between 2024-02-27 and 2024-06-13, then we moved to using 2 streams between 2024-06-13 and 2024-07-23. Since then, we have reverted to a single stream, but with shorter test durations of 5 seconds.

I advise against using MSAK data from https://speed.measurementlab.net for comparisons unless tests are carefully selected based on specific configurations, including the number of streams, congestion control algorithm, and expected test duration. Given that MSAK allows for greater flexibility, any analysis should consider these variables.

For NDT data:
Should I use MeanThroughputMbps for both Download and Upload speeds?
For latency, should I use the minRTT from the download test?

Yes, MeanThroughputMbps is the value to use for both. It corresponds to TCPInfo.BytesAcked / ElapsedTime * 8 (for downloads) or TCPInfo.BytesReceived / ElapsedTime * 8 (for uploads) from the last TCPInfo snapshot.

As far as I know there is no "standard" way to measure latency during a speedtest. FWIW, on speed.measurementlab.net we show TCPInfo.MinRTT from the last download sample (here for ndt7 and here for msak).

For MSAK data:
Should I focus only on the results with a single row?
For these single-row results, should I calculate the speed using Application.BytesReceived?

All the streams belonging to the same test (= same MeasurementID) should always be considered. In case there are more than one, they need to be aggregated beforehand. The measurement-lab.msak.throughput1_downloads view provides aggregated MeanThroughputMbps for MSAK downloads regardless of the number of streams.

As per your last question, it depends on what definition of "speed" you're interested in:

- goodput (application-level bytes, without protocol headers, so not including WebSocket and TLS overhead): if it's a download, use the last ClientMeasurement's Application.BytesReceived / ElapsedTime. If it's an upload, use the last ServerMeasurement's Application.BytesReceived / ElapsedTime

- throughput (includes protocol headers, WS/TLS overhead, and probably what you want to use): if it's a download, use the last ServerMeasurement's TCPInfo.BytesAcked / ElapsedTime. If it's an upload, use the last ServerMeasurement's TCPInfo.BytesReceived / ElapsedTime. If you're using the throughput1_downloads view (and, in future, throughput1_uploads), you have this already as a.ThroughputMbps.

Thank you again for your help in clarifying these points. Your guidance is invaluable in ensuring the accuracy of my comparative analysis.

Glad to be helpful!

-Roberto

[Jiayi Liu]

Hi Roberto,

Thank you very much for the detailed explanations and guidance.

Based on your suggestions, I will refrain from using MSAK data as part of my comparative analysis for now, focusing solely on NDT data instead. Your insights have been extremely helpful in ensuring the accuracy and reliability of my work.

Thanks again for your support!

Best regards,
Jiayi