Compiling LANDIS-II

[Adnan Hafeez]

Hello, has anyone here successfully been able to compile LANDIS-II and its extensions for an HPC Linux system operating under RHEL OS?

Looking through the website here https://www.landis-ii.org/install I Do not see any mention of supporting systems other than Windows. I would like to confirm If its possible to run this under Linux HPC before I put considerable amount of effort into it. 

Thank you for any of your support. 

Kind regards,

Adnan

[Robert Scheller]

Adam,

See:  https://github.com/LANDIS-II-Foundation/Core-Model-v8-LINUX

Melissa Lucash and James Lamping led that project.

Cheers,

Rob

[Alex Chubaty]

Hi Adnan,

Others have got LANDIS-II running successfully on Linux (typically Ubuntu I believe). Getting v7 going appears to be best documented:

- https://github.com/LANDIS-II-Foundation/Core-Model-v7-LINUX

- https://github.com/LANDIS-II-Foundation/Tool-Linux-Singularity

However, with the release of v8, Ive been trying to get that version going following the instructions here:

- https://github.com/LANDIS-II-Foundation/Core-Model-v8-LINUX

I've been able to build the various components per the instructions, but haven't yet managed to run any scenarios. I will resume working through this in the next few days, but here are a few pinch points so far:

- There appears to be a problem with some of the prebuilt binaries, namely a path reference issue (hardcoded path where it's failing to resolve "..\extensions", instead of using "../extensions").

- There may also be a discrepancy in the  location of the prebuilt binaries for the parts of the code that aren't extensions or support libraries ("build/Release" vs "build/extensions"). 

- Need to copy/symlink the five GDAL-related *so objects rather than the *.dlls to build/Release.

Hopefully others can provide additional guidance.

[Adnan Hafeez]

Also, do you think this will only work on Ubuntu? Our HPC is RHEL only. 

Thank you,

Adnan

[Adnan Hafeez]

Thank you for your replies. I will look into v7. The issues regarding v8 is there any way to fix those?

On Monday, October 14, 2024 at 10:27:57 AM UTC-4 Alex Chubaty wrote:

[Klemet]

Hello Adnan !

Based on my own work, I could recommend you look into compiling LANDIS-II-v7 into an Apptainer; it's basically an image containing Ubuntu (or any other version of Linux you want) that you can then "run" anywhere, with its own internal dependencies being taken care of.

The Apptainer image can then be used on supercomputing clusters or personal computers without messing too much with dependencies. It also makes simulations that are very easily replicable.

I already have an Ubuntu apptainer image with LANDIS-II-v7 and most extensions compiled, along with a Python environment that i use for some scripts. I can send it to you if you need it, along with some notes on how to create and use an apptainer image. I'll prepare a LANDIS-II-v8 apptainer soon when all extensions are ready for v8 : )

Good luck with using LANDIS-II !

[Matthew Garcia]

Klemet--

If you’d be willing to share those instructions with this message group, I think building a containerized package with LANDIS-II would be useful to many in the community. If someone wants the actual Apptainer image, perhaps that can be shared individually or via download link?

--Matt Garcia

[Klemet]

Ok, so I made a repository with one Apptainer file (using the Core v7 and 14 different LANDIS-II extensions with older versions) and all of the instructions to use and make an Apptainer :

https://github.com/Klemet/LANDIS-II-Apptainer

I've tried to be as detailed as possible in how to make an Apptainer, but you can also download the latest version of the one I've been using these past few years in the release section.

I hope this will help you all !

[Klemet]

Hey Matthew 😄 !

It's a great idea; I'll make a GitHub repository with instructions + an Apptainer file. I'll post this here as soon as it's ready.

Le mardi 15 octobre 2024 à 05:30:29 UTC-4, Matthew Garcia a écrit :

[Alex Chubaty]

Thank you for making this avalaible. Are you able to share the .sif file in the repo too?

I'm working on Docker images that I've put together for use which includes LANDIS-II v7 with several extensions, and an R environment (with Rstudio for interactive use) for working with data preparation and anlyses of simulation outputs. It'd be great to ensure I'm following suggested/best practice with respect to the LANDIS-II pieces.

Have you had a chance to look at a LANDIS-II v8 version yet?

Alex

[Klemet]

Hello Alex 😄 !

Yes, the .sif is shared through the "Release" section of the repo, as it allows the sharing of large files (up to 2GB). The current .sif that I uploaded is around 1GB in size.

Melissa Lucash and you just notified my of the existence of the Docker images of LANDIS-II ; I think it's awesome !

I've tried to read more about Docker, to see if I should choose between investing time in a develloping Docker image (and maybe join your efforts) or an Apptainer, or both of them. It looks to me like Docker has some advantages (maybe smaller sizes of containers, easier to pull, easier to deal with several container at once, etc.); but an Apptainer also seem to have some key features here.

In particular, this reddit thread mentions that Docker is not always available on High-Performance Computing environnements (HPCs), such as supercomputing clusters, making Apptainers (formerly Singularity) more adapted to these environments. Some users mention that docker images can also be pulled from inside an Apptainer; it could make the two compatible, although I think it would be redondant in the case of LANDIS-II.

So, I tend to think that Apptainers might be more adapted for the use-case of LANDIS-II; and maybe easier, as user just have to download one file and learn one command to use it.

For v8 : I will probably create the image in the coming weeks. I was discussing with Robert Scheller if it was the right time to do it; he seemed to say that it was ! But I'll take the time to properly test each extension first, and maybe several at once for compatibility issues.

[Alex Chubaty]

THanks for pointing out the release files. I misspoke though - I meant .sdef instead of .sif. Having a text version of the image build "recipe" is important for reuse etc.

There are certainly pros/cons of each approach (Apptainer vs Docker). Singulariy/Apptainer is dominant on HPC systems because it doesn't require elevated privileges the way Docker does, but Docker seems to be used more for CI/CD, and I'm more familiar with the prebuilt solutions in the Docker space (e.g., Rstudio server images and reproducible R environments) which makeit easy to add on a LANDIS-II layer to the images to keep complete workflows in the same system. I know it was possible to convert image formats (e.g., Docker to Apptainer) so that may be an option. Having both Apptainer and Docker recipes (i.e., sdef and Dockerfile) and available for people to use as they see fit would be a huge help. (I'll add here too, that my current Docker build recipe includes an R script to automate the tedious manual editing of the extension csproj files, and allows the user to select which modules they want/need in their image. I imagine such a script would be useful in an Apptainer build process too.)

I've tried v8, with little success so far. I would be happy to share notes / coordinate with you or others. As part of testing, I think it would be incredibly beneficial to include the extension tests as part of the image build stage, so having the sequence of commands you use for testing scripted/recorded would be a huge bonus.

Thanks,

Alex

[Klemet]

Thank you for your answer, Alex !

> I misspoke though - I meant .sdef instead of .sif. Having a text version of the image build "recipe" is important for reuse etc.

I gotta be honest; I had no idea of what .sdef files where, and I think it's awesome ! Will definitly look into it.

> There are certainly pros/cons of each approach (Apptainer vs Docker). Singulariy/Apptainer is dominant on HPC systems because it doesn't require elevated privileges the way Docker does, but Docker seems to be used more for CI/CD, and I'm more familiar with the prebuilt solutions in the Docker space (e.g., Rstudio server images and reproducible R environments) which makeit easy to add on a LANDIS-II layer to the images to keep complete workflows in the same system. I know it was possible to convert image formats (e.g., Docker to Apptainer) so that may be an option. Having both Apptainer and Docker recipes (i.e., sdef and Dockerfile) and available for people to use as they see fit would be a huge help. (I'll add here too, that my current Docker build recipe includes an R script to automate the tedious manual editing of the extension csproj files, and allows the user to select which modules they want/need in their image. I imagine such a script would be useful in an Apptainer build process too.)

I find the idea of automating the editing of extension csproj file an amazing idea; I never thought about it ! Clearly, that's the way to go.

Plus, if it's possible to transform a docker image into an Apptainer, then I think that this is definitly the way to go (according to perplexity, it's looks pretty simple). I think that we should focus on collaborating on one Docker image, and make the Appimages directly available for people that work on HPC systems.

Melissa Lucash told me that her group has built a docker image for LANDIS-II v8 with the support of another group, and that they will finish testing in the next two weeks. Maybe I should launch a discussion with the three of us about it ?