Intent to Implement: Migrating test manifests to TOML

[Andrew Halberstadt]

Hello everyone,

For more than a decade now, we've been using a bespoke ini(ish) parser to process our test manifests. While it has served us well, it is time to switch to a standard format.

We are planning to migrate all manifestparser based manifests (e.g mochitest.ini, browser.ini,  xpcshell.ini, etc) to TOML. Notably, web-platform-tests and reftests use their own manifest formats and will remain unaffected.

## Why TOML?

There are several reasons we chose TOML.

1. Standard. It is simple and widely used, parsers exist for it in every major language.

2. Tooling. There are parsers that support round-trip comments (tomlkit), which means we can finally write tools that can edit the manifests without losing comments.

3. Performance. There are several performance optimized parsers (implemented in Rust or C++, like rtoml) that should give appreciable boosts to the build backend responsible for processing manifests.

4. Familiar. If you squint, it looks close enough to the existing ini format that it should feel somewhat familiar to developers.

5. Maintenance. We no longer need to maintain a bespoke ini parser.

## Why now?

As Firefox's CI has continued to grow in complexity, the number of test settings (i.e a unique test configuration) and migrations (i.e, switching the environment a test runs in) has steadily been growing as well. It's gotten to the point where we are unable to keep up. When adding a new test setting, or performing a migration, the most time consuming part is always triaging tests. This process involves a ton of manual manifest editing in order to skip all the new failures that inevitably arise.

In order to keep up with the pace of CI, we need tools to aid us. Which means we need a way to automate the editing of manifests, which is not possible with our current parser (without losing comments).

## How will the migration work?

The migration will be internal to manifestparser, so if you maintain any tools that process our test manifests, the switch should be completely transparent (though you may need to update to a newer version of manifestparser if you are using it out-of-tree).

Rather than a hard cutover, there will be a transition period where both manifest formats are supported. This will help reduce conflicts and allow us to migrate a small subset of manifests first, and work through any issues that arise before changing the entire tree at once.

The work is being tracked in bug 1821199. Big thanks to Tom Marble for driving this project!

Please reach out if you have any questions or concerns.

Cheers,

Andrew