In general packages like numpy or scipy do not have as many breaking changes as in astropy. For scipy, it "upstreams" i.e. absorbs lots of changes that in Astropy are released as affiliated packages, and contributions for a long time were evaluated on whether they break existing user code. For Astropy, pipelines and analysis tools built on a combination of affiliated and semi-affiliated/dependent packages can quickly fall out of date with the changes and improvements made to Astropy core, and require bumping the version, which then breaks user code. I have personally seen many examples of this among colleagues and students.
I would like to offer a solution that may help. Currently, although Astropy has some of the most user-friendly installation and development documentation for a large python package that I've ever seen,
we do not help with taking advantage of the semversion system.The most we say is try to use a conda environment with anaconda default (
https://docs.astropy.org/en/stable/install.html#using-conda), which is extremely unreproducible and this generic, all-in-one environment does not facilitate more frequent updates. Updates will break things, especially older code, and leave users bewildered. And we actually recommend running `conda update anaconda` in this environment, which can be not nice to recover from.
I believe this is the wrong approach. We should show and encourage users to create a NEW python environment for a given specific task, with FIXED versions of astropy. Such an environment can be exactly re-created when the user moves to the next postdoc and gets a new computer in 1.5 years, allowing their old code to keep working, or needs to come back and run the same model 5 years later upon request from a new paper wanting to build on the results.
If people are familiar with `python poetry`, it is now in a stage where it works well alongside conda, to easily create version locked manageable virtual environments. I think the conda users were initially ignored by the target demographic of poetry, but the support has been mostly forced onto it now through relentless community requests. It is not the only tool for creating reproducible environments, but I think it may ultimately be the most user-friendly. In any case:
If we instructed users how to create reproducible environments for their code (a concept they are hopefully familiar with from writing clear instructions on how to reproduce their scientific results 😉), then they could lock their script for paper xyz or pipeline code with astropy==4.1 photutils==0.9 specutils==1.0.1. Just locking only these versions will go a long way, as I believe astropy devs currently backport breaking dependency changes, except for that one jinja2 incident (I may be wrong about this). But even if only locking the astropy related packages is not always enough (due to unlocked secondary dependencies like numpy or jinja), then we could go one step further and instruct on how to create an environment dump, using conda/poetry/pipenv etc. and then also how to re-install from this conda environment yaml or poetry/pip pyproject.toml/lockfile (or even requirements.txt).
I believe this would allow for astropy to not bother as much with LTS and backports, as the number of requests from users would go down significantly.
Ultimately, the biggest problem is creating broken complex dependencies for users to just run their previously working code again. Getting new features into old versions is a nice to have that does not really have to be entertained. The previous goal is best achieved by a combination of instructing users how to actually create environment that are impossible to break, and being considerate in not making too many breaking changes too quickly.
Best regards,
Emir
---
Emir Karamehmetoglu