Frequent major-version changes and the current plugin compatibility model

[Daniel Song]

Hi all,

Following a brief exchange on the forums with Dan, I am moving this discussion here.

I would like to raise a concern about Zotero’s increasingly frequent major-version changes, particularly because the application’s major version is currently used as a plugin compatibility boundary...

I would like to raise a concern about Zotero’s increasingly frequent major-version changes, particularly because the application’s major version is also being used as a plugin compatibility boundary.

The current stable release is Zotero 9, there is already a 10.0 development branch, and the `main` branch now contains `11.0.SOURCE`:

https://github.com/zotero/zotero/blob/main/version

I understand that a major version does not necessarily have to represent a complete redesign. Zotero 8 included a significant Mozilla platform upgrade and changes such as the JSM-to-ESM migration, removal of Bluebird, and changes to internal Mozilla APIs. Zotero 9 also introduced a number of new features.

However, from an ordinary user’s perspective, Zotero 8 and 9 do not feel like entirely new generations of the application. More importantly, versions 8, 9, 10, and 11 are appearing in relatively rapid succession. This would mostly be a cosmetic issue if the version number were only a product label, but it has real consequences for the plugin ecosystem.

The main problem is not the number itself. The problem is that the Zotero product version is being used as a proxy for plugin API compatibility.

Under the current model:

* A plugin may work perfectly well with a new Zotero version but still be disabled because its `strict_max_version` has not been increased.
* Plugin developers must test and update compatibility metadata for every new major version, even when no code changes are required.
* Users can upgrade Zotero and unexpectedly lose important parts of their workflow.
* The maintenance and coordination cost of frequent major releases is transferred to every plugin developer and every plugin-dependent user.
* There is no sufficiently visible pre-upgrade report showing which installed plugins will be disabled.

I understand the official reasoning that minor Zotero releases are guaranteed not to break plugins, while major releases may include UI or internal API changes. Requiring plugin developers to test a potentially breaking release is reasonable in isolation.

But this model becomes increasingly difficult to justify when major releases are frequent. It creates many cases of “administrative incompatibility”: the plugin code remains compatible, but Zotero disables it until somebody changes a version number in the manifest or update feed.

This suggests that the current compatibility model is not sufficiently granular.

Would it be possible to separate the Zotero product version from the plugin API version? For example:

```text
Zotero application version: 11.0
Stable plugin API version: 3
Mozilla platform version: 140
```

Plugins using only the stable API could declare compatibility with plugin API version 3 and remain enabled across Zotero 9, 10, and 11. Plugins using privileged internal APIs could continue to use stricter application-version limits.

A more sustainable model might include:

1. A separately versioned plugin API or ABI.
2. Capability-based compatibility declarations instead of relying mainly on the application’s major version.
3. Automatic forward compatibility for plugins that use only stable, sandboxed APIs.
4. Strict version limits only for plugins that access Zotero internals or Mozilla/XPCOM APIs.
5. A pre-upgrade compatibility check listing plugins that will be disabled.
6. Remote compatibility overrides for plugins known to work, without requiring a new XPI release.
7. A clearly announced API freeze and testing period before each compatibility-breaking release.
8. Possibly an LTS or extended-support channel for institutions and users whose workflows depend heavily on plugins.

I would also appreciate clarification on the intended meaning of a Zotero major version under the new release model:

* Does it represent a major set of user-visible features?
* A Mozilla platform upgrade?
* A potentially breaking change to internal APIs?
* Or simply a new release train?

If every release train is assigned a new major number and every major number becomes a new plugin compatibility boundary, the plugin ecosystem will face increasing maintenance costs even when the actual API changes are small.

I am not arguing that Zotero should never break internal APIs or modernize its platform. The concern is that application release numbering, internal platform changes, and plugin API compatibility are currently too tightly coupled.

Frequent releases are understandable. Frequent compatibility resets are much more disruptive.

Could the Zotero team explain whether the planned sandboxed plugin APIs will eventually allow plugins using only stable APIs to remain compatible across future major releases without repeatedly increasing `strict_max_version`?

[Dan Stillman]

First, make sure you've read A Faster Release Cycle for Zotero, which explains the reasons for the faster cadence. Our first priority is our users, and the faster cycle lets us get both stable features and bug fixes out to users more quickly.

A major release is legitimately the boundary for plugin compatibility testing. As you note, Zotero 7 to 8 involved major changes, including a Mozilla platform update. Zotero 9 to 10 involved extensive internal changes. Zotero 11 will involve (at least) another Mozilla platform update. Only Zotero 8 to 9 involved no major internal changes, but that's likely to be the exception rather than the rule. And even small changes could conceivably interact badly with a plugin.

Current plugins have full access to everything in Zotero, so there's no way to predict with certainty what might trigger breakage (or worse) from a plugin. As long as that's the case, plugin developers truly do need to test their plugins at each major version. If they can't commit to that, they should not ask users to run their code. We also need to know that developers are actively maintaining their plugins, because if a problem did arise, we need to know that they would be around to address it immediately.

One of the benefits of the new release cycle, though, is that major updates are smaller, so updates in many cases will take much less time. And within a major version, we now guarantee that there will be only bug fixes, whereas for years, technically "minor" versions often received major new features or involved breaking API changes. There were just a lot fewer Zotero plugins, so it didn't matter as much.

As we've said, we're working on new sandboxed APIs, and developers who switch to those may be able to avoid `strict_max_version` bumps or switch to a different gating mechanism that needs less-frequent updating. But that's not something we can guarantee yet. Our priority is giving users visibility into and control over what plugins can do, not taking away the broad access that makes the Zotero plugin ecosystem so vibrant. Any sufficiently broad API surface is liable to be affected somehow as we make major changes in Zotero.

To quickly address a few other points:

5. A pre-upgrade compatibility check listing plugins that will be disabled.

No, upgrades will remain automatic.

7. A clearly announced API freeze and testing period before each compatibility-breaking release.

I announced exactly this for Zotero 10 here a week ago, and we'll do so for each major release. Plugin developers need to monitor this list. In the future, we hope to offer other options for important development notifications like these (including, potentially, automated warnings about compatibility status).

6. Remote compatibility overrides for plugins known to work, without requiring a new XPI release.

Already possible, as explained in that announcement (and previously).

8. Possibly an LTS or extended-support channel for institutions and users whose workflows depend heavily on plugins.

We don't think an ESR makes sense for Zotero.

- Dan

--
You received this message because you are subscribed to the Google Groups "zotero-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/zotero-dev/9f468fca-6319-4497-a98a-ab82056ab947n%40googlegroups.com.

[XY Wong]

Regarding the plugin API version:

1. For the plugin API, we always try our best not to introduce breaking changes once it's in the stable release of Zotero. For any breaking changes that will cause plugins using the APIs to be incompatible, they come with a Zotero major version update.

2. For other Zotero internal code or platform code that plugins may use for now, there's no guarantee of compatibility across major versions, though by design of the release cycle of Zotero app, non-major version changes don't actually introduce big features and thus usually don't bring compatibility issues for plugins within a major version.

For reasons above, as long as developers make sure their plugin is compatible with the latest major version, it is likely not to get broken in future updates of the same major version. Thus I don't see the necessity of introducing a separate "plugin API version".

On the other hand, plugin developers are obligated to track each major release of Zotero and test their plugins, fix any issues, and bump the max-version accordingly. With the planned sandboxing, one should be able to see fewer compatibility issues across versions (ideally none, though from time to time we have to introduce breaking changes, like the recent multi-collection-selection affecting some APIs), but developers should still always test plugins for each major version update. And actually, per the announcement of the release cycle, it would be only several times (around 6-8 times)  a year that a major version is released; it should be manageable, especially considering that for most plugins, the compatibility check and max-version bump can be largely covered by automated CI tests.