Array Jobs and Aiida
Joseph Weston
Dear Aiida users/developers,
I’ve got a question about combining Aiida with the “job array” feature present in several popular schedulers (Slurm, PBS, …).
Essentially I want to run a large number of calculations of identical type with different inputs. The number of calculations
is sufficiently large that submitting individual jobs to the cluster is not feasible (the scheduler daemon falls over).
I would like to know whether there is some feature that I am not aware of that will enable this out of the box, and if not
I would like to start a discussion about how this could be best implemented in Aiida.
My current solution is to create a custom Calculation type (ArrayCalculation) that accepts Aiida Arrays/Lists as input, and sets the appropriate
scheduler command (e.g. “#SBATCH –array=0-100” for Slurm). Each task in the array job then writes its output to a separate file,
and the parser then constructs an Array/List for the output.
This just about works, however there are several problems:
- The Calculation does not have access to the Scheduler type that it’s being run under, meaning I currently hard-code the
“scheduler_command” to use “#SBATCH –array=…”. This could be improved by extending Aiida’s “job template” to include
An array job specification.
- Caching is much less useful. Aiida hashes the inputs/outputs as a whole, meaning that parts of the outputs from previous
ArrayCalculations cannot be used. Consider the difference between running 3 separate Calculations with inputs 1, 2, and 3 and running
a single ArrayCalculation with input [1, 2, 3].
I searched the mailing list/issue tracker for a related issue, and saw that Pezhman had already asked a similar question on this mailing list a couple of years ago, but did not receive any replies.
Here’s hoping someone can help me out!
Thanks,
Joe
Joseph Weston
Software Development Engineer, Microsoft Quantum
Zhu, Bonan
Hi Joe,
If my understanding is correct what you want to achieve is essentially some kind of task farming. At the moment AiiDA maps each CalcJob to a job of the scheduler. I would image that the array job approach would not be implemented easily as often that the unique id of a job inside the array is not available at the time of submission, and this may also vary among different types of schedulers.
However, you can achieve what you want (e.g. running many “small” AiiDA calculation inside some limited numbers of big/long cluster job) by the `aiida-fireworks-scheduler` plugin (https://github.com/zhubonan/aiida-fireworks-scheduler, full disclosure – I am the author).
This package does require some additional setup though. What it does is that instead of submitting the jobs directly to the scheduler of the cluster, they get submitted to another workflow engine (fireworks) acting as a meta-scheduler. Each cluster job can then run multiple of such workflows in series or in parallel. Afterwards, job retrieval and parsing are handled by AiiDA as usual. Using this approach, the user is responsible for submitting the cluster job (for running fireworks) manually, but this is something we seek automate in the future within AiiDA. If you have any question about the plugin, please feel free to open issues on Github.
Best wishes,
Bonan
--
AiiDA is supported by the NCCR MARVEL (http://nccr-marvel.ch/),
funded by the Swiss National Science Foundation, and by the European H2020 MaX Centre of Excellence (http://www.max-centre.eu/).
Before posting your first question, please see the posting guidelines at
http://www.aiida.net/?page_id=356 .
---
You received this message because you are subscribed to the Google Groups "aiidausers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
aiidausers+...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/aiidausers/CO1PR21MB1283ED6719DA1FA2C49F9510DEC89%40CO1PR21MB1283.namprd21.prod.outlook.com.
Joseph Weston
Hi Bonan,
Thanks for your reply!
Task farming could indeed be a solution. From speaking with Dominik Gresch I am aware that there is a plan to put some form of task farming into Aiida;
is there an ongoing mailing list discussion about this? (I searched but could only find an old one referencing a hackathon >1 year ago).
If task farming were implemented then it seems that the caching problem would solve itself also.
However, there would still be the issue of efficiently referring to all the calculations as a unit. AFAIK the only grouping mechanism in Aiida to use “node groups”,
Which cannot themselves be used as inputs to a calculation. I see that others have used calcfunctions to fulfil a similar purpose
(see How to do provenance with items in an aiida.orm.nodes.data.list.List (google.com)), however I am unsure how Aiida would perform if I were to create
a single process Node with ~1M inputs (in order to simulate, say, an array).
Thanks,
Joe
From: aiida...@googlegroups.com <aiida...@googlegroups.com>
On Behalf Of Zhu, Bonan
Sent: Friday, August 27, 2021 4:02 PM
To: aiida...@googlegroups.com
Subject: [EXTERNAL] [aiidausers] RE: Array Jobs and Aiida
|
You don't often get email from bona...@ucl.ac.uk. Learn why this is important |
To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/DB3PR0102MB337051636743EE81EA9E0CA4B9C89%40DB3PR0102MB3370.eurprd01.prod.exchangelabs.com.
Giovanni Pizzi
To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/CO1PR21MB12832496B9989D1AF98E4725DEC99%40CO1PR21MB1283.namprd21.prod.outlook.com.
Joseph Weston
Hi Giovanni
We're in the process of brainstorming what level of task farming we can provide in AiiDA.
We've been discussing this also today some of these aspects at the fortnightly AiiDA developer meeting (notes here: https://hackmd.io/reNOJUwNRReIxLSwkAwI1g)
In essence, there are a few options on the table: at the moment the one that has a working implementation is Bonan's aiida-fireworks-scheduler, and we're discussing some small changes to AiiDA to make that plugin (but potentially any other doing a similar job in acting as a meta-scheduler) easier to implement and use: https://github.com/aiidateam/aiida-core/issues/5084
But we're also discussing if there are other engines that we could use as "meta-schedulers" (e.g. Greasy that is offered e.g. in the Swiss supercomputer centre: https://user.cscs.ch/tools/high_throughput/).
The plan is to discuss task farming in a few months, after we finalise some changes to AiiDA that are currently in the develop branch and we'd like to release first (soon).
Thanks for the links!
I also see that your use case is however a bit different, you'd also like to "group" many calculations together, and I'm curious to understand more about your use case. So, if you'd be interested in discussing this further, please contact me in private, and I'll set up a call also including any other developer on our side that might be interested in joining the discussion.
Thanks, I’ll reach out in a separate email.
Best regards,
Joe