A few questions about a server or multi-user setup for AiiDA

[Holger Sassnick]

Dear all,

First off, I would like to thank you for developing such a nice tool which really helped me and others in our group to facilitate and manage our high-throughput calculations. I very much like the sophisticated design of the package and the concept of provenance that it implements.

For more complex and larger projects I thought that it would be nice to have an AiiDA setup with the following features:

• Having multiple users accessing the same database/repository and therefore natively sharing/reusing calculation results (As far as I know this is probably not yet possible, but I think that it would be nice to have).
• A sort of server/client implementation as setting up AiiDA locally demands a bit of technical knowledge and needs a linux operating system. Especially when working with students I don't think I can expect them to go through this process.

Initially I (naively) assumed that setting up a shared remote postgreSQL database would resolve the issues (I didn't realize that data was also stored in the file-repository at that point...). The daemon could run on one of our linux office computers and the data can be remotely analyzed by accessing the database. I am still using this kind of setup for my own calculations and even though this use-case was not intended, it works quite well and I wouldn't want to miss the flexibility that it gives me.
I noticed that there are developments on different storage backends for the aiida-core library. I was wondering whether this would enable a complete remote storage option at some point where several computers could access all the data of an AiiDA profile?

After taking this detour, I guess the easiest way would be to have the database and aiida-core running on one server computer and accessing it via e.g. a web interface. As far as I know this is precisely the approach that aiidalab is taking. Before going deeper into this I would be very thankful for your opinion and how the initially described requirements may or may not be accomplished..

I also saw that the AiiDA rest API is continouosly extended (also towards starting and managing AiiDA processes), would it make sense to set up an interface via this API in the future? What are the use-cases intended for this API?

Finally, I wish you a relaxing end of holiday and a great year 2023.
Thank you for going through my post, I am very much looking forward to a follow up discussion.

All the best,
Holger 

[Jonathan Chico]

Hi Holger!

I'm not an AiiDA main developer but I use it quite a lot and developer my own workflows and plugins, so I can just tell you a bit on how we use AiiDA to address some of the issues you raise.

• Having multiple users accessing the same database/repository and therefore natively sharing/reusing calculation results (As far as I know this is probably not yet possible, but I think that it would be nice to have).

As far as I know there is no way to have several users accessing the same database/repository as you have found out, the approach that we have taken is to have a "main" repository where calculations are uploaded from each of the users profiles once they are deemed to be good enough to be shared. This is not ideal, but it is the best that I have been able to come up with until now. Some  

• A sort of server/client implementation as setting up AiiDA locally demands a bit of technical knowledge and needs a linux operating system. Especially when working with students I don't think I can expect them to go through this process.

Aiidalab aims to make the setup of aiida much simpler and there are some nice ansible roles and playbooks that can be used to automate the deployment. 

 

I also saw that the AiiDA rest API is continouosly extended (also towards starting and managing AiiDA processes), would it make sense to set up an interface via this API in the future? What are the use-cases intended for this API?

The AiiDA REST-API is indeed very nice, we use it as a way to share our data in a centralized way right now, as the functionality grows I think that controlling AiiDA itself would be possible.

I do agree however that  multi-user support would be great, since this will allow for easier collaboration and sharing, making certain projects advance much faster.

Cheers

[Sebastiaan Huber]

Dear Holger,

 

You are right that currently using a multi-user setup for AiiDA is not straightforward.

However, I think it should be possible as the basic infrastructure is there.

 

In principle it is possible to have multiple clients connect to the same storage, which as you have figured out consists by default of a Postgres database  and a file repository on the local file system.

As long as the file system is accessible on all clients, this setup should work just fine.

Each client would create its own AiiDA instance (essentially just the `.aiida` config folder with the `config.json` file) and configure a profile that points to the Postgres database and the file repository.

Each client could for example mount the file system of the file repository through sshfs if the machine where it is mounted can be accessed over SSH.

 

As you also already mentioned, as of AiiDA v2.1 the storage backend is pluginnable.

I have used this concept to create the `aiida-s3` plugin (https://github.com/sphuber/aiida-s3) which replaces the file repository on the local file system with an object store (AWS S3 and Azure Blob Storage are currently implemented).

Although this makes connecting multiple clients to the same storage more straightforward, one should of course expect a non-negligible performance hit.

Whether this is acceptable depends on your usage and use-case.

I am working on improving the performance of these solutions, but this will be a long-term project.

 

Concerning the REST API: there are currently two implementations:

1. The REST API that ships with `aiida-core`: this currently only supports read-only operations, so essentially just querying
2. The `aiida-restapi` (https://github.com/aiidateam/aiida-restapi/) plugin: provides both querying but also process management

 

The aiida-restapi plugin was created some time ago, but found little traction for the time being.

Recently, I have been updating it to add more control features, such as starting/stopping the daemon, and playing/pausing/killing of processes.

I think the feature set it has now is quite complete.

If you find this useful and think there are features missing, please feel free to open feature requests on the Github repo.

I will most probably be able to dedicate time to implement them as we are also using it now ourselves.

If this project is useful, I am thinking of writing a Python client library to make using the API less verbose (currently one has to type a lot of boilerplate).

Let me know if this would be useful to you and I can keep you in the loop of its design and development.

 

Hope this was useful and feel free to ask if you have any more questions.

 

Regards,

 

Sebastiaan

--
AiiDA is supported by the NCCR MARVEL (http://nccr-marvel.ch/), funded by the Swiss National Science Foundation, and by the European H2020 MaX Centre of Excellence (http://www.max-centre.eu/).
 
Before posting your first question, please see the posting guidelines at http://www.aiida.net/?page_id=356 .
---
You received this message because you are subscribed to the Google Groups "aiidausers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aiidausers+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/99e0c4f9-2c08-4f07-aaaf-f8573b7f6797n%40googlegroups.com.

 

[Holger Sassnick]

Dear Jonathan and Sebastiaan,

thank you for the replys and all the information. I think I sort of have a better idea now about how a setup could look like for us.

I would have one more question concerning aiida-lab setup on e.g. the materialscloud page. Unfortunately, my knowledge on docker and ansible is rather limited.
Is it true that each user is run in an individual docker container on a server?
Do you think that it would be possible to also do that on a smaller scale and have each docker container connect to a shared file repository and database?
This way I could circumvent all the problems, people could access the server regardless of the operating system since I would rely on the aiida-lab user interface and still calculation results could be shared if needed..

Have a nice weekend and all the best,
Holger

[Jusong Yu]

Dear Holger,

You are correct about AiiDAlab, it is a web interface, a containerized Jupyter environment for AiiDA deployed in a server. It should have all the functionalities of the AiiDA but nothing more.

Therefore if AiiDA has an elegant way to support accessing shared  DB and file repository in the future, the AiiDAlab should support it. 

> Is it true that each user is run in an individual docker container on a server?

Indeed, each user is run in a container and inside the container is the independent database for each user. 

The ansible/k8s recipe for deploying the AiiDAlab is now all using aiidalab-docker-stack full-stack which has PostgreSQL DB installed and used inside the container. 

The "full-stack" image is built based on the "aiidalab" stack which only has AiiDA installed and configured with jupyter infrastructure but not PostgreSQL (If you only need aiida but not the aiidalab web interface, there are images "base" and "base-with-services" parallel to "aiidalab" and "full-stack". For more details please check https://github.com/aiidalab/aiidalab-docker-stack#docker-stack-for-aiidalab). This hints at a possibility that the DB can be deployed separately and the file repository can be mounted independently to accomplish what you mentioned about multi-user shared AiiDAlab deployment. 

It is worth a try without considering the security problem like the users can override data or write to the DB backend at the same time. But at the moment, unfortunately, we don't have it supported. 

Best,

Jason

[Holger Sassnick]

Dear Jason,

thank you for the response, I will then try out the different images and see if I can find a good set up for our use case.

All the best,

Holger