ESMification update

[Tooru Fujisawa]

Hi everyone.

ESMification is a project to switch the Mozilla-specific JSMs in privileged code to the standard ECMAScript modules (ESM).

Recently, the preparatory phase has started to impact larger parts of the codebase. In the interest of keeping everyone informed, we will send out a few updates regarding the transition process. This is the first such update on what's going on and future planning

Ongoing Preparation Work

Preparation work is split into 4 parts.

Add New APIs

Just like `ChromeUtils.import` and friends for JSM, we're going to add new APIs for importing and handling ESMs.

• Basic import API: `ChromeUtils.importESM`
• Defining lazy getter: `ChromeUtils.defineESMGetters`
• Registering window/process actors: `esmURI` property for each existing API
• Static component registration: `esm` property in components.conf
• Querying loaded modules: `Cu.isESMLoaded` and `Cu.loadedESMs`

NOTE: the name of the APIs is subject to change

Most of the work is almost ready, and we can land them once the API naming convention gets fixed.

The filename extension for the system (privileged) ESM is `.sys.mjs`.  This is to distinguish between regular ES module files that use `.mjs`.

Add Compatibility Layer for existing APIs

To make the migration easier, and also to avoid breaking out-of-tree code, the existing APIs like `ChromeUtils.import` are automatically redirected to ESM-ified files, if the JSM is already ESM-ified.

• Import APIs such as `ChromeUtils.import` 
• Querying loaded modules: `Cu.isModuleLoaded` and `Cu.loadedModules`

This allows migration to be done per single file, or subtree, without affecting other files, or out-of-tree code.

Also, lexical variables in JSM are now exposed to `Cu.import` return value, and there’s no need to copy lexical variables to the global `this` property by `this.foo = foo;`.

Those layers are already available in 103.

Rewrite ESM-incompatible part in JSM

JSM has a per-module global `this` object, which has been used to declare global variables and lazy getters. The standard ECMAScript module does not have the global `this` object, and code that relies on it needs to be rewritten.

• Rewrite `this.foo = ...` with `const foo = ...`
• Rewrite lazy getters to use a plain `lazy` object
• Use plain object instead of global `this` for `Cu.cloneInto`
• Use `.jsm` extension consistently

Most of the work is almost done, or getting reviewed.

Please refer to the Lazy Getter Update email and the Lazy Getter Implementation document for more information.

Add ESLint rules

We’ll prepare ESLint rules both for JSMs and ESMs.

For JSMs, new rules will be mostly to avoid adding ESM-incompatible code, and some of them will be applied also to ESMs:

• Reject global `this`
• Reject modification to `globalThis`
• Check `lazy` object properties

For ESMs, new rules will be to catch some edge cases:

• Reject top-level await (this is not supported yet, because of synchronous-ness)
• Avoid misuse of `import` declarations
• Standardize the lazy object name

Planning

In-tree JSM-to-ESM Migration Phase 1

The first phase of the migration covers the following:

• Rename `*.jsm` to `*.sys.mjs`
• Replace `EXPORTED_SYMBOLS` with `export` declaration
• Replace `ChromeUtils.import` for the module with either:
• static `import` declaration, if it’s in the system ESM’s top-level
• `ChromeUtils.importESM` otherwise
• Rewrite lazy getter for the module
• Rewrite `moz.build` for the module
• Rewrite `components.conf` for the module
• Rewrite `ChromeUtils.registerWindowActor` for the module
• Rewrite `ChromeUtils.registerProcessActor` for the module
• Rewrite `Cu.isModuleLoaded` for the module
• Rewrite `Cu.loadedModules` for the module

The migration will be semi-automatic, with scripts for simple cases, and manual work for edge cases. Each team will work at their own pace to apply the scripts and identify edge cases. 

It’s not necessary to migrate all JSMs to ESM. Difficult cases can be left as JSM.

We’ll send the detailed document once it’s ready.

In-tree Migration phase 2

Once all in-tree code becomes ESM, each team can rewrite those files to be more standard ESM-style, such as using `export default` etc.

Out-of-tree Migration

There are multiple out-of-tree consumers of JSM:

• Firefox’s privileged extensions
• Thunderbird’s extensions
• non-sandboxed AutoConfig scripts

And we need to keep the compatibility with them.

During the in-tree migration phase 1, the above compatibility layer guarantees the compatibility as long as:

• The set of exported symbols is same
• The behavior of the exported symbols is same
• The URL is same, except for the extension part (`.jsm` to `.sys.mjs`)

Once the in-tree migration phase 1 finishes, we’ll call for the out-of-tree migration, to use the new APIs.

The out-of-tree code migration period will take some cycles.

Cleanup on the API

Once the out-of-tree migration finishes, we’ll perform cleanup, including gradually removing the deprecated APIs or special behavior around that:

• Remove 2nd parameter of `ChromeUtils.import`
• Stop exposing the global object via Cu.import return value, or maybe remove Cu.import

Non-deprecated JSM APIs such as `ChromeUtils.import` aren’t removed, for in-tree difficult-to-migrate cases, and also for out-of-tree compatibility.