Dear Atomate Team
I’ve encountered some difficulty customizing a the atomate.vasp.fireworks.core.LepsFW to output more information into the task document than the default.
In the original firework, the LepsFW terminates after generating a task document in MongoDB that reports the following fields in the output taskdoc: epsilon_static, epsilon_static_wolfe, epsilon_ionic, piezo_ionic_tensor, piezo_tensor. However, since the DFPT calculation generates other information in calcs_reversed/output section as well, such as force_constants, normalmode_eigenvals etc, I want to customize the firework to report these information in the output taskdoc too.
May I know what is the cleanest way of doing this?
What I have tried is the following:
The Original LepsFW parses the vasp output to MongoDB tasks collection using the atomate.vasp.firetasks.parse_output.VaspToDb, which in turn invokes the atomate.vasp.drones.VaspDrone to do the actual parsing. I made copies of
in my own user directory in a folder named “custom” and inserted a sys.path.append command in both core.py and parse_output.py so that the modified files load each of their subsidiary from
The modified drones.py’s line 299 is then edited to include terms such as “force_constants” etc. so that they will be parsed to the “output” taskdoc.
I thought this will be sufficient but the following error popped up:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Traceback (most recent call last):
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py\", line 1461, in _refresh_wf
updated_ids = wf.refresh(fw_id)
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py\", line 1005, in refresh
updated_ids = updated_ids.union(self.apply_action(m_action, fw.fw_id))
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py\", line 813, in apply_action
apply_mod(mod, self.id_fw[cfid].spec)
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py\", line 1765, in spec
return self.partial_fw.spec
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py\", line 1838, in partial_fw
self._fw = Firework.from_dict(data)
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 155, in _decorator
new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 155, in <dictcomp>
new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 118, in _recursive_load
return {k: _recursive_load(v) for k, v in obj.items()}
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 118, in <dictcomp>
return {k: _recursive_load(v) for k, v in obj.items()}
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 121, in _recursive_load
return [_recursive_load(v) for v in obj]
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 121, in <listcomp>
return [_recursive_load(v) for v in obj]
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 113, in _recursive_load
return load_object(obj)
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 329, in load_object
mod = __import__(modname, globals(), locals(), [classname], 0)
ModuleNotFoundError: No module named 'custom'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/rocket.py\", line 357, in run
lp.complete_launch(launch_id, m_action, final_state)
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py\", line 1272, in complete_launch
self._refresh_wf(fw_id)
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py\", line 1476, in _refresh_wf
raise RuntimeError(err_message)
RuntimeError: Error refreshing workflow. The full stack trace is: Traceback (most recent call last):
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py\", line 1461, in _refresh_wf
updated_ids = wf.refresh(fw_id)
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py\", line 1005, in refresh
updated_ids = updated_ids.union(self.apply_action(m_action, fw.fw_id))
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py\", line 813, in apply_action
apply_mod(mod, self.id_fw[cfid].spec)
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py\", line 1765, in spec
return self.partial_fw.spec
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py\", line 1838, in partial_fw
self._fw = Firework.from_dict(data)
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 155, in _decorator
new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 155, in <dictcomp>
new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 118, in _recursive_load
return {k: _recursive_load(v) for k, v in obj.items()}
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 118, in <dictcomp>
return {k: _recursive_load(v) for k, v in obj.items()}
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 121, in _recursive_load
return [_recursive_load(v) for v in obj]
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 121, in <listcomp>
return [_recursive_load(v) for v in obj]
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 113, in _recursive_load
return load_object(obj)
File \"/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py\", line 329, in load_object
mod = __import__(modname, globals(), locals(), [classname], 0)
ModuleNotFoundError: No module named 'custom'
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
It seems that there are more files and classes involved even after drones.py; however, I am unable to locate them. At this point, the process starts to look very complicated.
May I know if there is a cleaner way of achieving this objective? Or if not, which other files should I modify to overcome this error?
Best wishes
Yaze
p.s.
I’ve tried modifying task_materials.py in atomate/vasp/builder to parse the sections in “calcs_reversed”, but it seems that the “calcs_reversed” is a list with only 1 element. I can only parse the entire “calcs_reversed” to a new taskdoc but am unable to parse only a section of it. Testing with adding the following line
doc["testing"] = taskdoc["calcs_reversed"][1]
to task_materials.py leads to IndexError: list index out of range
When FireWorks bootstraps your Firetask from a database definition, it needs to know where to look for Firetasks.
First, you need to make sure your Firetask is defined in a file location that can be found by Python, i.e. is within Python’s search path and that you can import your Firetask in a Python shell. If Python cannot import your code (e.g., from the shell), neither can FireWorks. This step usually means either installing the code into your site-packages
directory (where many Python tools install code) or modifying your PYTHONPATH
environment variable to include the location of the Firetask. You can see the locations where Python looks for code by typing import sys
followed by print(sys.path)
. If you are unfamiliar with this topic, some more details about this process can be found here, or try Googling “how does Python find modules?”
Second, you must register your Firetask so that it can be found by the FireWorks software. There are a couple of options for registering your Firetask (you only need to do one of the below):
fireworks.user_objects
or it subdirectories - it will be automatically be found there.USER_PACKAGES
variable of the FW config to include the package for where to find the Firetask, e.g. “mypackage.my_subpackage”. Note that FireWorks will search within subpackages automatically, so you can just put a root package (but loading will be slightly slower).You are now ready to use your Firetask!