Release plans and stability for extensions

[Antonin Delpeuch (lists)]

Hello all,

Following the discussion about the integration of the RDF extension
(https://github.com/OpenRefine/OpenRefine/issues/4518), I have been
thinking about how we can better support extension developers.

The first low hanging fruit is to upload our Maven artifacts to Maven
Central so that extension developers can have a simple build system. We
are already doing this for snapshot releases, we just need to add it to
the normal release process (and do it retrospectively for 3.5, maybe).
Unless anyone has a problem with this, I would do this soon.

Second, I would propose that we avoid any large-scale upgrades in the
3.x release series from now on, keeping it stable for extensions. By
large-scale upgrade I mean things like the migration to Jackson,
migration to the Wikimedia i18n plugin or jQuery updates. Those upgrades
can be done in 4.x instead. Migrations in 3.x would still be allowed to
fix serious vulnerabilities (log4shell) or address similar major issues.

For instance, we currently have a PR open which migrates to jQuery 3.6.0
(https://github.com/OpenRefine/OpenRefine/pull/3087). This is something
that will require extensions to adapt. Although they are helped in this
process by the warnings that should be emitted when using the extension
with OpenRefine 3.5, this can still be a big piece of work, especially
without a good test suite. Therefore I propose that we do this upgrade
in 4.x only.

More generally, I am keen to think about a better extension model for
4.0. I am trying to find out good examples of extension architectures
for the web frontend, for the backend, and how the two can be tied
together so that users can install an extension once (patching the
frontend and backend at the same time). If you have any suggestions of
examples to look at, I would be very interested to know.

Cheers,

Antonin

[Thad Guidry]

Hi Antonin,

I've also given considerable thought to a few extension architectures, but I'll keep things concise for now with a single need:

I would like to see the dev experience to support HMR hot module reloading which allows a dev to see changes on the screen quicker or immediately as you make them and while maintaining the state of the application.

(I myself have built some simple things with SvelteKit which uses Snowpack to deliver HMR and really enjoy Svelte's simpler component experience )

It's extremely empowering to be able to make a small change to a single small component in an possible extension to the application and then seeing the changes without refreshing the page.

However, I think I cannot begin my extension adventures for records as long as we are still on Butterfly which as I understand prevents any HMR for devs?

Thad

https://www.linkedin.com/in/thadguidry/

https://calendly.com/thadguidry/

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine-dev/2ebd7b17-f7f1-ab75-c45a-8eab5048821d%40antonin.delpeuch.eu.

[Thad Guidry]

It also seems like we might make use of Java's ServiceLoader to enable a plugin experience for developers without the overhead of something like OSGI.

In fact, many other data applications use ServiceLoader for this: Apache NiFi, Talend, Liferay, Jboss, and others.

Examples:
https://pedrorijo.com/blog/java-service-loader/
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#processor_api

After trying out a few Svelte / Java / Quarkus application templates I am convinced that Quarkus is not for the faint of heart (non-trivial) for our particular application stack.

Spring has done a good job of documenting some of the drawbacks of a true GraalVM native project which Quarkus can build as.

But at least Spring is looking to the future and following closing the GraalVM ecosystem with their Spring Native project: https://docs.spring.io/spring-native/docs/current/reference/htmlsingle/

So rather than Quarkus which directly doesn't solve some of our scaling problems for future plans of extensions, api, and multi-tenancy, we might look to Spring.

So Spring might be our overall architecture we could utilize for several things to help extension developers (we don't have to use Spring's ecosystem at all, or only pieces that help us solve our needs):

1. migrating away from Butterfly and just use Spring Boot which creates an application context (just like Butterfly) and instantiates objects and adds them to the context and easily can do this for extensions with @ComponentScan

2. extensions can be excluded with includeFilters / excludeFilters

3. from the Spring ecosystem, we might use Spring HATEOAS to rebuild the OpenRefine API (and document and version it) to leverage Hydra and make it easier for extension authors to quickly come to grips with the API interactively instead of reading lots of our technical documentation.

4. Spring Webflux would give us a reactive core (where we can still hang on to Jetty since Spring Boot can automate startup)