AQM Evaluation Suite Upgrade | GSoC '25

[David Lin]

Hi everyone!

By way of introduction, I'm David and I'm currently a final year undergraduate from UofT! I'm deeply passionate about computer networks and it's exciting to meet the ns-3 community!

I will be working on the project "Upgrade the AQM Evaluation Suite for ns-3" over the summer, the main topics for improvement have been outlined in this issue and more information and updates can also be found on the wiki page.

If anyone has any feedback or question, feel free to drop by a message and I'd love to chat!

Best,

David

[David Lin]

Hi everyone!

I’m writing to share the progress update for my project on upgrading and extending the AQM Evaluation Suite in ns-3. Below is a summary of work completed during the Community Bonding Period through Week 2, along with next steps discussed in our recent project meeting.

Community Bonding Period (May 8–31, 2025)

• Migrated the legacy AQM Evaluation Suite codebase from GitHub to the new GitLab repository for centralized and collaborative development.

• Conducted a thorough review of the original AQM Evaluation Suite paper, extracting key insights and compiling a structured action plan aligned with recent ns-3 advancements.

• Identified behavioral discrepancies in the AggressiveSender scenario from ns-3.41 onward, where aggressive TCP variants (BIC, HighSpeed, YeAH) diverged from previous outputs. Traced the root cause to pacing settings and restored expected behaviour. Notably, TCP NewReno remains unaffected.

Week 1 (June 1–7, 2025)

• Migrated the build system from Waf to CMake to ensure compatibility with the latest ns-3 versions. Validated successful builds across platforms and ensured test suite integrity.

• Replaced deprecated TCP BIC with TCP CUBIC (now supported in ns-3) as the default for aggressive senders, addressing original limitations and aligning with [RFC 7928].

• Validated that TCP NewReno is explicitly configured in all relevant scenarios to avoid future regressions, as ns-3 now defaults to TCP CUBIC.

Week 2 (June 8–14, 2025)

• Implemented Explicit Congestion Notification (ECN) support as recommended in RFC 7928 §4.5. ECN is now configurable via --ecn=true and activates both at the TCP sender and AQM queue level (e.g., RED).

• Performance evaluation showed that ECN-enabled scenarios yield more concentrated ellipsoids in queue delay vs. goodput plots, indicating reduced variance and improved stability. As expected, Pfifo remains unaffected due to lack of ECN support.

Next Steps

Our next steps focus on preparing the AQM Evaluation Suite as a contributed module for inclusion in the ns-allinone-3.45 release. We will create the new repository and structure the module to meet contrib standards by June 24–25. I’ll finish integrating new AQM algorithms, clean up ECN logic, and squash commits—especially for the CMake migration—for review. We aim to ensure the module builds cleanly in allinone and is well-documented for reproducibility and ease of use.

Please feel free to review the GitLab repository and the Wiki, and reach out with any questions or suggestions. I’ll follow up again after the Week 4 milestone with further updates.

Best,

David

[David Lin]

Hi everyone!

Here’s a quick update on the progress for the AQM Evaluation Suite over the past weeks:

Week 3 (June 15–21, 2025)

• Improved output handling: Each test run is now saved to a separate timestamped folder (aqm-eval-output/{timestamp}), with a run_config.json file capturing the configuration for traceability and reproducibility.
  
• Integrated additional AQM algorithms: Added support for ns3::CobaltQueueDisc, ns3::FqCobaltQueueDisc, ns3::FqCoDelQueueDisc, ns3::FqPieQueueDisc, and ns3::TbfQueueDisc, including ECN support where available.

Week 4 (June 22–28, 2025)

• Improved dependency management: The suite now checks for required external tools (Python, GNUplot, ImageMagick) before running. If any are missing, it logs a warning and exits cleanly, instead of failing during the run.
  
• Refactored queue delay logic: Removed the redundant EvalTimestampTag class. Now using QueueDiscItem's built-in timestamp for queue delay calculations, reducing code duplication.
• Centralized output management: Replaced scattered system calls for directory creation and file management with a dedicated OutputManager class, making the codebase cleaner and more maintainable.
  

Next Steps

I’ll be continuing the preparations of the contributed module for the ns-allinone-3.45 release—polishing documentation, migrating commits to the new module repository, and ensuring smooth integration.

Feel free to review the repository and reach out with feedback or questions. Another update will follow after the next milestone.

Best,

David

[David Lin]

Hi everyone!

Apologies for the typos in the previous versions, here’s a quick update on the progress for the AQM Evaluation Suite over the past weeks:

Week 5 (June 29–July 5, 2025)

• Contrib module migration
  • Migrated the AQM Evaluation Suite into the ns-3 contrib repository as a standalone module. Updated all sources, scripts, and documentation (including README.md and Sphinx docs) to reflect the new directory layout and dependencies.
• Release-readiness for ns-allinone-3.45
  • Ran compatibility tests against the full ns-3.45 and verified that example workflows execute out-of-the-box as a contributed module.
• Clean commit history
  • Used interactive rebase to squash minor commits, applied a centralized commit-message format consistent with ns-3 practices, and established the cleaned-up branch as the new master.

Week 6 (July 6–July 12, 2025)

• Repo refactoring & MR checkpoint
  
  • Refactored the contrib module’s repository for improved maintainability. Checkpointed the first-half milestone with MR for initial review.
• ImageMagick compatibility
  
  • Discovered that ImageMagick 6 uses convert while 7 uses magick. Enhanced the test runner to detect and invoke the appropriate command at runtime, ensuring the compatibility and consistency with the behaviour that ns-3 uses to generate sphinx documentations.

Next Steps

• Incorporate reviewer feedback on MR, finalize install rules, and polish example launch scripts.
  
• Start investigating into the objectives for the next milestone while prioritizing on addressing the reviewer feedbacks.

Feel free to take a look at the MR and share any feedback or questions. More updates to follow!

Best,
David

[David Lin]

Hi everyone!

Time for an update on the progress for the AQM Evaluation Suite over the past weeks:

• The merge request I've submitted is pending review, which I've prioritized addressing the reviews. A few notable problems that I've addressed from the reviewer's comment is
  • Refactored logging, there were originally many calls to cout, which didn't utilize the logging module that comes with ns-3.
  • The sphinx documentation wasn't in the standard format, which I've discovered from the Zulip chat that there is tutorial and examples on creating documentation for the module.
• Standardized the integration of custom AQM algorithms, which is one of the key feature in order to ensure a smooth developer experience. Currently the built-in AQM algorithms are hard-coded in multiple places, for example in the runner and helper, as well as the Python scripts. This makes adding new AQM algorithms hard as there are multiple individual copies saved in different places, and in different format (For example, in the runner it was saved as the short format "PfifoFast" but in the helper it was saved as "ns3::PfifoFastQueueDisc") this requires many hard-coded processing required and can create inconsistencies.I'm addressing this issue by creating a singleton registry of all the AQM algorithms with the complete metadata, such as name and the actual ns-3 typeid, along with other attributes, and reference this as source of truth in all places. During the process I've encountered one difficulty that there appears to be some discrepancy between the output data of the original implementation and the new implementation, despite the configurations should be unchanged, and after investigation it came from the randomness, since originally, the order of the aqm algorithms in the array was fixed, and the new registry which replaced the fixed array had a different execution order of AQM algorithms and thus created minor discrepancy between the output due to the randomness and order of execution, which is acceptable and still accurate in my opinion. The code can be found in this MR which will be finalized after the previous has been merged, the actual code is in the last commit.
  

For the next step, I'm actively working on bringing multi-bottlenck topology support, with the inspiration from the Generic Fairness Configuration - 2 (GFC-2), and analyze its performance.

Best,

David

[David Lin]

Hi everyone!

Unfortunately I was sick since this Wednesday and wasn't been able to make significant progress, but I would still like to give some quick updates as I've been catching up.

• The MR that contains the major upgrade to the aqm-eval-suite has been merged in the contributed module repository, which is a big milestone.
• The MR for the central AQM registry is ready for review, after fixing a few bugs related to how the AQM was registered in the registry, I though I'd reflect on why it was designed this way. As previously described it was hard to modify the AQM algorithms, as they are scattered around. The design principles originally was to allow the user to add custom AQM algorithms with flexibility, ideally at the example level. I've considered multiple designs such as using a config file, however it might require parsing structured files which then requires external dependencies, there is also one main difficulty to achieve the example level registry without config file is that the aqm-eval-runner runs child processes and the aqm registry needs to be synchronized for correct processing. The current design, we simply treat the aqm registered in aqm-registry.cc as source of truth for all registered algorithms, and is the only place where change is needed, sort of like a config file. I'm open to suggestions here if anyone has idea for a better design.
• Started the investigation of multi-bottleneck topology upgrade, the changes is also harder than anticipated. I've tried some simple approaches and the current challenges is that the current EvalutionTopology class is fixed to the dumbbell topology and is used in all examples, which leads to another problem of how we should present this option to the user, should it be additional examples with the multi-bottleneck topology or should it also be a flag like how we enable ECN. Additionally, the current tracing are also based on the assumption of single bottleneck such as the queue delay, if we have multiple queues how should we present this visually, maybe potentially with multiple graphs? Nevertheless I've been working on creating a good abstraction as the BaseEvaluationTopology which can be derived to either DumbbellTopology or MultiBottleneck, I think this is a good starting point as it can provide a standardized API to use in the examples in the future. This is mainly the next step, and I'm also open for any ideas if anyone has done something similar.

Best,

David

[Tom Henderson]

David, inline below.

On 8/3/25 12:20 PM, David Lin wrote:

Hi everyone!

Unfortunately I was sick since this Wednesday and wasn't been able to make significant progress, but I would still like to give some quick updates as I've been catching up.

• The MR that contains the major upgrade to the aqm-eval-suite has been merged in the contributed module repository, which is a big milestone.

I agree that this is a notable milestone; I was able to successfully add it to the ns-allinone-3.45 manifest as a contributed module.  

• The MR for the central AQM registry is ready for review, after fixing a few bugs related to how the AQM was registered in the registry, I though I'd reflect on why it was designed this way. As previously described it was hard to modify the AQM algorithms, as they are scattered around. The design principles originally was to allow the user to add custom AQM algorithms with flexibility, ideally at the example level. I've considered multiple designs such as using a config file, however it might require parsing structured files which then requires external dependencies, there is also one main difficulty to achieve the example level registry without config file is that the aqm-eval-runner runs child processes and the aqm registry needs to be synchronized for correct processing. The current design, we simply treat the aqm registered in aqm-registry.cc as source of truth for all registered algorithms, and is the only place where change is needed, sort of like a config file. I'm open to suggestions here if anyone has idea for a better design.

I left some review comments in the MR.

• Started the investigation of multi-bottleneck topology upgrade, the changes is also harder than anticipated. I've tried some simple approaches and the current challenges is that the current EvalutionTopology class is fixed to the dumbbell topology and is used in all examples, which leads to another problem of how we should present this option to the user, should it be additional examples with the multi-bottleneck topology or should it also be a flag like how we enable ECN. Additionally, the current tracing are also based on the assumption of single bottleneck such as the queue delay, if we have multiple queues how should we present this visually, maybe potentially with multiple graphs? Nevertheless I've been working on creating a good abstraction as the BaseEvaluationTopology which can be derived to either DumbbellTopology or MultiBottleneck, I think this is a good starting point as it can provide a standardized API to use in the examples in the future. This is mainly the next step, and I'm also open for any ideas if anyone has done something similar.

Can you point to where a multi-bottleneck topology is defined?  I did not see it defined in the AQM evaulation IETF draft.

- Tom

[Mohit P. Tahiliani]

Hi Tom,

Thanks for the feedback.

On Tue, Aug 5, 2025 at 2:00 AM 'Tom Henderson' via ns-developers <ns-dev...@googlegroups.com> wrote:

[cut]

Can you point to where a multi-bottleneck topology is defined?  I did not see it defined in the AQM evaulation IETF draft.

It is described in Section 10 of RFC 7928: https://datatracker.ietf.org/doc/html/rfc7928#section-10

Best Regards,

Mohit P. Tahiliani

[David Lin]

Hi everyone!

Giving a quick update on the AQM evaluation suite:

• Been implementing a simple examples of the multi-bottleneck according to the GFC-1 topology (diagram available on issue). After discussion in the previous meeting, we would like to first implement it as a simple example, with the same AQM algorithm across the routers and explore streaming output to multiple files based on the router, in the future it would be good to make it more customizable by allowing users to set different AQM algorithms across the routers. One of the difficulty encountered was the topology has a quite different interface compared to existing EvaluationTopology class, and the ScenarioImpl class that is used in the examples has a dependency on EvaluationTopology. The solution I have for now is turning ScenarioImpl into a template class so the user is able to choose whether the scenario is based on original EvaluationTopology or the new multi-bottleneck topology. This design has made existing code that uses the original topology to work with minimal changes while allowing the extension of the new topology, perhaps in the future we can work on standardizing the abstraction for the evaluation topologies. I've studies some existing implementation of the GFC-1 topology available on GitHub and implemented a simple example in AQM evaluation suite, and successfully demultiplexed the output data based on the router, currently testing the behaviours and parameters of the topology to make sure the output will be viable.
• I will be giving a lightning talk at icns3 about the AQM evaluation suite, covering the background, how it works and some of the upgrades I've made to the suite during the summer. Feel free to drop by virtually if anyone is interested!

Best,

David