how to consistently get the higher throughput

[Sandeep Raman]

Hi,

Using the command stressapptest -m 288 -C 10 -s 200, ocassionally the throughput is:

Stats: Completed: 39026072.00M in 200.82s 194332.91MB/s, with 0 hardware incidents, 0 errors
Stats: Memory Copy: 39026072.00M at 194982.83MB/s

Otherwise it is always:

Stats: Completed: 6809244.00M in 200.90s 33893.27MB/s, with 0 hardware incidents, 0 errors
Stats: Memory Copy: 6809244.00M at 33921.13MB/s

2025/10/22-09:39:53(MDT) Log: Commandline - stressapptest -m 288 -C 10 -s 200
2025/10/22-09:39:53(MDT) Stats: SAT revision 1.0.9_autoconf, 64 bit binary
2025/10/22-09:39:53(MDT) Log: mockbuild @ 60f7effd08ba466099e31a8dafb3a554 on Fri Aug  9 19:27:10 UTC 2024 from open source release
2025/10/22-09:39:53(MDT) Log: 1 nodes, 288 cpus.
2025/10/22-09:39:53(MDT) Log: Total 1031176 MB. Free 1020883 MB. Hugepages 0 MB. Targeting 979425 MB (94%)
2025/10/22-09:39:53(MDT) Log: Prefer plain malloc memory allocation.
2025/10/22-09:39:53(MDT) Log: Using mmap() allocation at 0x7e4d75f00000.
2025/10/22-09:39:53(MDT) Stats: Starting SAT, 979425M, 200 seconds
2025/10/22-09:40:56(MDT) Log: region number 54 exceeds region count 8
2025/10/22-09:41:01(MDT) Log: Region mask: 0xff
2025/10/22-09:41:12(MDT) Log: Seconds remaining: 190
2025/10/22-09:41:22(MDT) Log: Seconds remaining: 180
2025/10/22-09:41:32(MDT) Log: Seconds remaining: 170
2025/10/22-09:41:42(MDT) Log: Seconds remaining: 160
2025/10/22-09:41:52(MDT) Log: Seconds remaining: 150
2025/10/22-09:42:02(MDT) Log: Seconds remaining: 140
2025/10/22-09:42:12(MDT) Log: Seconds remaining: 130
2025/10/22-09:42:22(MDT) Log: Seconds remaining: 120
2025/10/22-09:42:32(MDT) Log: Seconds remaining: 110
2025/10/22-09:42:42(MDT) Log: Seconds remaining: 100
2025/10/22-09:42:52(MDT) Log: Seconds remaining: 90
2025/10/22-09:43:02(MDT) Log: Seconds remaining: 80
2025/10/22-09:43:12(MDT) Log: Seconds remaining: 70
2025/10/22-09:43:22(MDT) Log: Seconds remaining: 60
2025/10/22-09:43:32(MDT) Log: Seconds remaining: 50
2025/10/22-09:43:42(MDT) Log: Seconds remaining: 40
2025/10/22-09:43:52(MDT) Log: Seconds remaining: 30
2025/10/22-09:44:02(MDT) Log: Seconds remaining: 20
2025/10/22-09:44:12(MDT) Log: Seconds remaining: 10
2025/10/22-09:44:40(MDT) Stats: Found 0 hardware incidents
2025/10/22-09:44:40(MDT) Stats: Completed: 6809244.00M in 200.90s 33893.27MB/s, with 0 hardware incidents, 0 errors
2025/10/22-09:44:40(MDT) Stats: Memory Copy: 6809244.00M at 33921.13MB/s
2025/10/22-09:44:40(MDT) Stats: File Copy: 0.00M at 0.00MB/s
2025/10/22-09:44:40(MDT) Stats: Net Copy: 0.00M at 0.00MB/s
2025/10/22-09:44:40(MDT) Stats: Data Check: 0.00M at 0.00MB/s
2025/10/22-09:44:40(MDT) Stats: Invert Data: 0.00M at 0.00MB/s
2025/10/22-09:44:40(MDT) Stats: Disk: 0.00M at 0.00MB/s
2025/10/22-09:44:40(MDT)
2025/10/22-09:44:40(MDT) Status: PASS - please verify no corrected errors
2025/10/22-09:44:40(MDT)


numastat -p 330169                                                                                                                                                                          

Per-node process memory usage (in MBs) for PID 330169 (stressapptest)
                           Node 0          Node 1           Total
                  --------------- --------------- ---------------
Huge                         0.00            0.00            0.00
Heap                         0.00            0.03            0.03
Stack                        0.00            0.02            0.02
Private                 462550.72       508797.05       971347.78
----------------  --------------- --------------- ---------------
Total                   462550.72       508797.10       971347.82

The numastat shows equal memory usage across both the NUMA.
Any pointers how to consistently get the higher throughput <<Stats: Memory Copy: 39026072.00M at 194982.83MB/s>>?

Thanks.

[Nick Sanders]

You can try "--local_numa", but I'm not sure it's still compatible with the most up to date numa APIs. 

I'd also check /var/log/messages to see if the system state is different when you get different results. Check thermal throttling, other jobs running, any quota your system may be applying. Also, check those numbers against the theoretical max of your system. 

--

---
You received this message because you are subscribed to the Google Groups "stressapptest-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stressapptest-di...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/stressapptest-discuss/5e9f7383-80aa-4fc5-b0c7-b25e43d7a0e2n%40googlegroups.com.

[Sandeep Raman]

Thanks,  --local_numa seems to slightly improve the throughput.

Stats: Completed: 8811196.00M in 200.87s 43864.93MB/s, with 0 hardware incidents, 0 errors

Stats: Memory Copy: 8811196.00M at 43982.61MB/s

The  /var/log/messages doesn't have anything logged related to these and doesn't look unusual or different. No other jobs running or quotas applied.

The  theoretical max of the system is ~800 GB/s.

I noticed one thing. Immediately after rebooting the system, I ran the command <<stressapptest -m 288 -C 10 -s 200 --local_numa>>. I got a pretty good throughput -> 202249.05MB/s. I recollected that the previous instance as well where the throughput was 194332.91MB/s was right after a reboot.

Stats: Completed: 40604848.00M in 200.77s 202249.05MB/s, with 0 hardware incidents, 0 errors
Stats: Memory Copy: 40604848.00M at 202911.97MB/s

I executed a few test runs. In the 4th run, the throughput dropped and it stays around this low number. Is stressapptest saturating any sub-system or any idea what could be happening here.

1st run after reboot: 202249.05MB/s
2nd run after reboot: 193562.94MB/s
3rd run after reboot: 204115.41MB/s
4th run after reboot: 45260.54MB/s

[Nick Sanders]

I wonder if the system's pagetable is getting full/slow? 

Can you try hugepages?

https://github.com/stressapptest/stressapptest/blob/master/src/os.cc#L380

To view this discussion visit https://groups.google.com/d/msgid/stressapptest-discuss/06d976e4-4baa-4015-96af-560e0e53924cn%40googlegroups.com.

[Nick Sanders]

Also, if you have a log of system temperatures, check if anything is getting hot. Check CPU/memory controller/memory clocks and see if they've been throttled without logging.

[Sandeep Raman]

I wonder if the system's pagetable is getting full/slow? Can you try hugepages?

I had 1G and 2M hugepages initially. I had remove the hugepage config because with 1G default hugepage stressapptest failed. https://groups.google.com/g/stressapptest-discuss/c/hnsA8I6Yp34/m/G54JtChXBQAJ

Also, if you have a log of system temperatures, check if anything is getting hot. Check CPU/memory controller/memory clocks and see if they've been throttled without logging.

I think I may have stumbled upon the cause. It looks like the Power, Memory and CPU related settings in the BIOS. With these changes, I now get consistent and good results.
I plan to run few more tests to narrow which specific BIOS settings really matter.

Thanks for the pointers.