CalcJobfunction

[Otto Kohulák]

Dear AiiDA community,

I would like to ask a question, but first let me explain a little bit what I'm trying to do.

There are two main ways how to do a calculation:

• calcfunction
• CalcJob

The calcfunctions are convenient for small-scale operation running that one likes to run on local a machine, like interpolations. Whereas CalcJobs are more suitable for running external codes. However, with CalcJobs comes much bigger infrastructure. Correct me if I'm wrong, but it is not possible in calcfunction to have help strings for inputs, namelist inputs, automatic input validation or ExitCode results. What I try to do is to use one particular python library (that requires disk IO). I can write my calcfunctions but I will lack all of the above mentioned features, like instead of using input namelists I have to use less readable dynamical keywords, or like I cannot have ExitCode environment. On the other hand, if I use CalcJob, the framework assumes I want to use external code. I can do workarounds, like doing everything in preparation for submission and than run blank code. Or preparing python code as input and running python interpreter as the external code. But they are still workarounds.

I tried all of the above, they work, but I was happy with none of them. I would like to ask AiiDA experts how they would approach this problem in the most AiiDAic way.

If there is something that is not clear about my question, please tell me I will do amends.

Best,

O.

[Zhu, Bonan]

Hi Otto,

 

Assuming that your code does not do anything “heavy”, what you can do is to wrap the running of the calcfunction inside a minimal WorkChain. Then you can enjoy all the features of CalcJob, such as input validations and help strings.

 

If your code is CPU heavy then I think it is still best to run it as a CalcJob. This is because running a long `calcfunction` will block the python process (daemon) and if it gets kill due to out-of-memory your daemon process would be lost. In addition, there is no way to know if it encounter any errors. This is not a problem for CalcJob as it is run as external code, and any errors encountered can be inspected using the stdout/stderr.

 

Best wishes,

Bonan

 

From: aiida...@googlegroups.com <aiida...@googlegroups.com> On Behalf Of Otto Kohulák
Sent: 14 June 2022 11:20
To: aiida...@googlegroups.com
Subject: {SPAM?} [aiidausers] CalcJobfunction

 

⚠ Caution: External sender

 

--
AiiDA is supported by the NCCR MARVEL (http://nccr-marvel.ch/), funded by the Swiss National Science Foundation, and by the European H2020 MaX Centre of Excellence (http://www.max-centre.eu/).
 
Before posting your first question, please see the posting guidelines at http://www.aiida.net/?page_id=356 .
---
You received this message because you are subscribed to the Google Groups "aiidausers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aiidausers+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/CAK%3DGx5imdLZ5YOZSrB6gY-r%3DBr4N1JEQBLdb0rQcNDMa6eD1fA%40mail.gmail.com.

[Otto Kohulák]

Dear Bonan,

first of all, thank you for very fast answer.

ut 14. 6. 2022 o 12:29 Zhu, Bonan <bona...@ucl.ac.uk> napísal(a):

Hi Otto,

 

Assuming that your code does not do anything “heavy”, what you can do is to wrap the running of the calcfunction inside a minimal WorkChain.

The truth is yes and no. Most of the time the jobs are very light, but some of them might require a lot of memory (like ~15GB), but are not very difficult to do. Maybe it is better to run them as a CalcJobs.

 

Then you can enjoy all the features of CalcJob, such as input validations and help strings.

 

If your code is CPU heavy then I think it is still best to run it as a CalcJob. This is because running a long `calcfunction` will block the python process (daemon) and if it gets kill due to out-of-memory your daemon process would be lost. In addition, there is no way to know if it encounter any errors. This is not a problem for CalcJob as it is run as external code, and any errors encountered can be inspected using the stdout/stderr.

So it looks to me it, there are not many options if I want to run python-heavy code in AiiDA. Correct me if I'm wrong, but if I would like to run a python-heavy code as a Calcjob, I would need to prepare the source in preparation for submission and execute the python interpreter as an external code. Isn't there a aiida-python plugin for automatization? If there's not, I will suggest it aiida-core issues or even contribute because I have more than one project that will benefit from it.

Best regards,

Otto

 

To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/DB9PR01MB917568EA661FB2D6135C3B03B9AA9%40DB9PR01MB9175.eurprd01.prod.exchangelabs.com.

[Giovanni Pizzi]

Dear Otto,

maybe you want to look into what Microsoft is working on in the AiiDA dynamic workflows package?

https://github.com/microsoft/aiida-dynamic-workflows

I think that this is probably covering some of your needs, and people from the Microsoft team should also be in this mailing list and can probably comment more in detail.

Best,

Giovanni

On 14 Jun 2022, at 13:40, Otto Kohulák <pra...@gmail.com> wrote:

Dear Bonan,

first of all, thank you for very fast answer.

ut 14. 6. 2022 o 12:29 Zhu, Bonan <bona...@ucl.ac.uk> napísal(a):

Hi Otto, 

 

Assuming that your code does not do anything “heavy”, what you can do is to wrap the running of the calcfunction inside a minimal WorkChain.

The truth is yes and no. Most of the time the jobs are very light, but some of them might require a lot of memory (like ~15GB), but are not very difficult to do. Maybe it is better to run them as a CalcJobs.

 

Then you can enjoy all the features of CalcJob, such as input validations and help strings.

 

If your code is CPU heavy then I think it is still best to run it as a CalcJob. This is because running a long `calcfunction` will block the python process (daemon) and if it gets kill due to out-of-memory your daemon process would be lost. In addition, there is no way to know if it encounter any errors. This is not a problem for CalcJob as it is run as external code, and any errors encountered can be inspected using the stdout/stderr.

So it looks to me it, there are not many options if I want to run python-heavy code in AiiDA. Correct me if I'm wrong, but if I would like to run a python-heavy code as a Calcjob, I would need to prepare the source in preparation for submission and execute the python interpreter as an external code. Isn't there a aiida-python plugin for automatization? If there's not, I will suggest it aiida-core issues or even contribute because I have more than one project that will benefit from it.

Best regards,

Otto

 

 

Best wishes,

Bonan 

 

From: aiida...@googlegroups.com <aiida...@googlegroups.com> On Behalf Of Otto Kohulák
Sent: 14 June 2022 11:20
To: aiida...@googlegroups.com
Subject: {SPAM?} [aiidausers] CalcJobfunction

 

⚠ Caution: External sender

 

Dear AiiDA community, 

 

I would like to ask a question, but first let me explain a little bit what I'm trying to do.

 

There are two main ways how to do a calculation:

• calcfunction
• CalcJob

The calcfunctions are convenient for small-scale operation running that one likes to run on local a machine, like interpolations. WhereasCalcJobs are more suitable for running external codes. However, with CalcJobs comes much bigger infrastructure. Correct me if I'm wrong, but it is not possible in calcfunction to have help strings for inputs, namelist inputs, automatic input validation or ExitCode results. What I try to do is to use one particular python library (that requires disk IO). I can write my calcfunctions but I will lack all of the above mentioned features, like instead of using input namelists I have to use less readable dynamical keywords, or like I cannot have ExitCode environment. On the other hand, if I use CalcJob, the framework assumes I want to use external code. I can do workarounds, like doing everything in preparation for submission and than run blank code. Or preparing python code as input and running python interpreter as the external code. But they are still workarounds.

I tried all of the above, they work, but I was happy with none of them. I would like to ask AiiDA experts how they would approach this problem in the most AiiDAic way.

 

If there is something that is not clear about my question, please tell me I will do amends.

 

Best,

 

O.

-- 
AiiDA is supported by the NCCR MARVEL (http://nccr-marvel.ch/), funded by the Swiss National Science Foundation, and by the European H2020 MaX Centre of Excellence (http://www.max-centre.eu/).
 
Before posting your first question, please see the posting guidelines at http://www.aiida.net/?page_id=356 .
--- 
You received this message because you are subscribed to the Google Groups "aiidausers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aiidausers+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/CAK%3DGx5imdLZ5YOZSrB6gY-r%3DBr4N1JEQBLdb0rQcNDMa6eD1fA%40mail.gmail.com.

-- 
AiiDA is supported by the NCCR MARVEL (http://nccr-marvel.ch/), funded by the Swiss National Science Foundation, and by the European H2020 MaX Centre of Excellence (http://www.max-centre.eu/).
 
Before posting your first question, please see the posting guidelines at http://www.aiida.net/?page_id=356 .
--- 
You received this message because you are subscribed to the Google Groups "aiidausers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aiidausers+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/DB9PR01MB917568EA661FB2D6135C3B03B9AA9%40DB9PR01MB9175.eurprd01.prod.exchangelabs.com.

-- 
AiiDA is supported by the NCCR MARVEL (http://nccr-marvel.ch/), funded by the Swiss National Science Foundation, and by the European H2020 MaX Centre of Excellence (http://www.max-centre.eu/).
 
Before posting your first question, please see the posting guidelines at http://www.aiida.net/?page_id=356 .
--- 
You received this message because you are subscribed to the Google Groups "aiidausers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aiidausers+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/CAK%3DGx5hEwAUXFO81hCDmtLenxd8YHT%2Bk6LpjGe2m1V41Mz3LQg%40mail.gmail.com.

[Otto Kohulák]

Dear Giovanni,

thank you for your suggestion, aiida-dynamic-workflows looks interesting. However, if I understand it correctly, it allows you to store your workflows as a node inside the database. Interesting concept, but I am aiming to have a new package that could operate with a python library.

To express myself I prepared a very simplistic plugin showing what I have in mind.

g...@github.com:addman2/aiida_python.git

The Idea goes like this. You can define a class with CalcJobPython as a parent. Prepare a calc plugin as if it was a normal CalcJob and define the run_python method as you "script". The example is in tests. I would like to see this as a part of aiida-core, but anyways if not I will start to develop such plugin.

I will suggest it on aiida issue tracker.

Best regards,

O.

ut 14. 6. 2022 o 21:49 'Giovanni Pizzi' via aiidausers <aiida...@googlegroups.com> napísal(a):

To view this discussion on the web visit https://groups.google.com/d/msgid/aiidausers/E783225B-BAF2-4139-A870-CE0C2FDE2ADF%40epfl.ch.