Future use of Glassfish and/or Payara

[Oliver Bertuch]

Dear devs,

I would like to bring to your attention that we need to talk about using
(Eclipse) Glassfish 5.1 or Payara 5.x for the future of Dataverse.

Context first: at FZJ we want to use Dataverse as our primary repository
for research data. Current plan is running a Kubernetes Cluster in a
medium sized HA service configuration including storage in custom S3
service (Ceph). To reach that goal we want a) up-to-date software, b) a
stable platform and c) a more microservices oriented application.

In our humble opinion, the current options to run Dataverse in
containers (docker-aio, Openshift and dataverse-docker) all suffer from
trying to depict the "classic" setup into containers, making them kind
of VM-like. As we know that more people are opting for running Dataverse
in Docker / Kubernetes / OpenShift, we are willing to help getting this
more mature for production usage and contribute time and ressources to
this wonderfull OSS project.

We are well aware that this needs changes to codebase and environment
and that this takes time and effort. We don't want to break someone
elses installation and know that Docker is not the holy grail - others
might still prefer the classic installation for good reasons and that
should continue to work.

After having said that, lets go "in medias res":
One of those steps to do is about getting rid of the EOL Glassfish 4.x.
This discussion is meant to be about how to replace it and the steps
that need to be done for this.

What we already know:
1) Devs at IQSS are leaning towards staying in the "Glassfish and
compatible" universe and not go with OpenLiberty/Wildfly/....
2) Devs at IQSS want security patches for application servers ASAP.
3) Some people at IQSS like the options of MicroProfile ConfigAPI, some
don't.
4) Developers want to focus on code, not app servers. Building an image
and testing it should be as comfortable as possible.
5) Uploading an image to the Docker Hub or other registries shouldn't
take long for small changes. Create a reasonable layer structure that
avoids unnecessary big push/pull.fixscreen
6) People using Docker/Kubernetes really like using "upstream images"
that just need configuring via ENV or other easy mechanisms, but don't
need them to build an inherited subimage. Dockerfiles are for builders,
(configurable) images are for admins and users.

What we need to discuss:
A) Philosophy beyond container images
B) Security needs

About A)
IMHO container images for a product like Dataverse should be
i) usable directly from upstream releases,
ii) stateless,
iii) compatible with Kubernetes ConfigMaps and Secrets plus Docker Secrets,
iv) rolling-update ready,
v) scalable to a multinode cluster and
vi) configureable via easy mechanisms like ENV vars, files etc (despite
config changes may need a restart of the container).

The current Perl based installer is IMHO not a good option for
containers. It's kind of bloated and not easy to maintain for both
classical and container usage. IMHO the codebase should be refactored to
a more hybrid model offering better container and classic support at the
same time plus specialized Docker stuff within Maven, scripts and
Dockerfiles.

Images should have no test data or similar by default to ensure
statelessness. That is currently not the case with dataverse-docker;
Openshift unclear (haven't digged deep yet); docker-aio was not meant
for production anyway.

Easy to configure and support for secrets handling? Not yet. Will need
support from the application server to make it easier to maintain. Keep
idempotency in mind - a restart may not result in a redeployment of
database structures or similar kaboom stuff.

About B)
If you take a look at CVEs for different application servers for the
last years, there hasn't been much potential of high risk security
fixes. It is unlikely this will change in the future. Seems like most of
the upstream Java ecosystem vendors know what they are doing. Even if
you look at bigger platforms as JBoss AS, they normally offer quarterly
releases including security fixes.

When discussing about a "Glassfish or compatible" application server,
there are only to candidates:

O1) Eclipse Glassfish first release 5.1 around December - the new kid on
the block. Release cycles yet unknown. Security patch release policy yet
unknown. Owned by Eclipse Foundation, current maintainers from Oracle
and Payara.
O2) Payara 5.x, mature ecosystem, quarterly releases without
subscription. Security patches included with quarterly updates. Owned by
Payara Foundation (UK Nonprofit). People at Payara Services Ltd (offers
subscriptions and support) seem to work for both plus Eclipse.

When you look at Eclipse Glassfish 5.1 histoy, you might note that there
have not been many changes feature wise
(https://github.com/eclipse-ee4j/glassfish/commits/master). Especially
the enhanced support for variables and pwd aliases in configuration
places (see
https://docs.payara.fish/documentation/payara-server/server-configuration/var-substitution)
is not yet in Eclipse Glassfish. This is pretty essential stuff when
addressing the concern mentioned for A). And I don't even suggest going
for ConfigAPI - that is another story not told here.

Conclusion and proposal)
IMHO we should opt for Payara 5.x and let Eclipse Glassfish project
settle down and get mature first. And we should go for application
scoped resource configuration instead of domain scoped. If you want to
have an example in what direction this is going, have a look at
https://github.com/poikilotherm/dataverse/tree/5292-small-container.

I am very curious what you think of my thoughts and proposal.
Please give loads of feedback and join the discussion.
(Crossing fingers nobody will throw stones in my direction...)

Cheers!
Oliver