ElasticSearch 5.6.x EOL and the move to ElasticSearch 6 or 7

[kjbo...@gmail.com]

Hello all,

I searched the group but didn't see any posts about this.

Our central IT group here at Carleton is nervous about our use of products which have reached EOL.

Since ES 5.6.x reached EOL in March of last year, is AtoM 2.6.0 expected to support a current version of ES?

A move to ElasticSearch 7.6.x would future proof AtoM until mid 2021.

Thank you,

Kevin Bowrin

Carleton University Library

[Dan Gillean]

Hi Kevin, 

Thanks for raising this topic. At Artefactual, we have an Elasticsearch upgrade to 7.x on our list of target goals for 2.6. That said, I can't at this time guarantee if it will be included in 2.6 or not - unless Carleton would like to help make this happen! More details below. 

Internally, we've been undergoing a company-wide reorganization, part of which has included the creation of a Software and Architecture Development team, whose focus is solely on the health, maintenance, and improvement of our open source projects (i.e. separate from our additional paid services and the general day to day operation of the company). This includes analyzing and addressing technical debt wherever possible. We have already upgraded from MySQL 5.7 to MySQL8 in the upcoming 2.6 release, as well as replaced the deprecated Flash-based media player (flowplayer) with a well-maintained, modern HTML5 player that still includes a Flash-fallback option for legacy users. AtoM also still has one Flash based component (the digital object multi-uploader), which we are prioritizing replacing prior to Flash's full browser support deprecation in December 2020. We have also been compiling a list of other major and minor areas of technical debt, and devising strategies for addressing them as we are able, prioritizing those with past or upcoming end of life dates. This includes trying to upgrade AtoM (and Archivematica) to newer versions of Elasticsearch as soon as possible. 

However, this is where Artefactual's business model can make this kind of work difficult to undertake! 

Essentially: we give away all our software, and supporting resources (like docs, slides, webinar recordings, training resources, etc) freely under open licenses. We also commit company time to writing docs, answering forum questions, giving demos, fixing bugs, prepping releases, and doing testing, without any cost to our user community. However, we are a small company, we are not eligible for grants, and we don't have external investors or venture capital - there is a limit to the amount of unpaid work we can perform while ensuring that Artefactual itself remains a viable employer and maintainer of our projects. 

Consequently, we depend on our community for major development in our core projects - either via code contributions, or via paid development support. This model has worked well for over 11 years now, and has lead to the many wonderful features available to all in AtoM - whenever someone sponsors development, we work with them to devise a solution that will meet their needs but also conform to relevant standards and be useful to the broader community, and then we incorporate this into the next public release. Therefore, everything you see in AtoM is the result of dozens (hundreds?) of institutions and individuals contributing to the project over its many years. We have further details on this model in the following page: 

• https://wiki.accesstomemory.org/Development/Philosophy  
  

While this model has worked great for feature development so far, it's MUCH harder to find development support for maintenance work - rarely do we find institutions willing to sponsor thousands of dollars worth of effort to have AtoM do the same thing at the end, but with updated dependencies in the background. People are generally focused on deliverables that they can use to justify budget requests against business needs. 

The formation of our new SAD business unit (as we jokingly refer to it internally) is one of the ways we are trying to address this, to free up more internal resources for maintenance work - but as a small company with limited reserves and an ethics of openness, there are still limits to what we can undertake without support. In the 2.5 release, we managed to upgrade from Elasticsearch 1.x to Elasticsearch 5.6 - however, because Elastic has a "move fast and break things" approach to development, the major changes between ES versions required thousands of lines of code changes in AtoM - you can see the initial commit implementing these changes here: 

• https://github.com/artefactual/atom/commit/eabbac520d765a4a903e01f0ef77bc068d71b7a7  
  

Once again, ES versions 6 and 7 include major non-backwards compatible changes, requiring an equal level of analysis and development effort in AtoM to be able to support the upgrade - more effort if we want to make changes to take advantage of some of the new options and methods available in newer Elasticsearch versions. 

With no community support, Artefactual will find a way to get there eventually - but as I said, I can't guarantee this will be in the 2.6 release or not. It will depend on some of the other factors we may need to prioritize, and the amount of internal development leeway we have - particularly in this challenging global moment. 

So, here comes the pitch: 

If Carleton considers an Elasticsearch upgrade a priority for the upcoming 2.6 release, please consider if you might be able to contribute to its inclusion! If you have developers, we would be happy to provide guidance on how to undertake the work required to submit a pull request upgrading the ES version -and we also have a number of Development resources available on our wiki. Alternatively, if your institution would be interested in sponsoring this enhancement, feel free to contact me off-list, and Artefactual can prepare some estimates. We would also be happy to share those estimates publicly once prepared, so that multiple institutions might collaborate in sponsoring the work if the expected cost is too much for a single institution. 

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/1476fccf-c8ef-4187-97ad-81b4ad3eb367%40googlegroups.com.