PyTasks code seems recorded in MongoDB. Is it possible to modify it ?

[David Michéa]

Hi,

I use a lot of PyTask to implement my fireworks.

Sometimes, tasks are FIZZLED because of a bug in the function passed to the PyTask. But when I fix the bug and re_run the FW, It ends up with the same error message : the bug seems not fixed, althought the python module as been updated. 

I suppose this is due to the fact that the code of the function seems to have been registered in MongoDB ... Am I wrong ?

Is it possible to fix the code in MongoDB so that I can rerun these FWs without error ? Or is there a way to force the FW to reload the function from its module ?

Best regards,

--

David Michéa
HPC Software Engineer
Tél : +33 (0)6 52 67 58 36
mic...@unistra.fr

[Anubhav Jain]

Hi David

PyTask shouldn't be storing the function contents, only the function name (e.g. "my_module.my_function") and the desired arguments and keyword arguments into that function (e.g., param1="bob"). If this is incorrect, can you send me the contents that you are seeing in MongoDB regarding storing the code of the function itself?

If you update the code on your Worker, it should also update when you rerun a PyTask. You should confirm that the updated version of your code is indeed being loaded when you load the same function name manually, i.e., not through FWS. If it still seems like a FWS problem, then maybe send me a dump of the JSON of the FW that is causing problem from your MongoDB.

Best

Anubhav

--
You received this message because you are subscribed to the Google Groups "fireworkflows" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflows+unsubscribe@googlegroups.com.
To post to this group, send email to firewo...@googlegroups.com.
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/CANr6Pra5yFa1VRcYZLVzJpx1mZLgWrOv0_hVgcYS%3D7gENogEaQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Best,
Anubhav

[David Michéa]

Hi Anubhav,

For exemple, here is a stacktrace of a FIZZLED PyTask after correction of the bug:

• "_stacktrace": "Traceback (most recent call last):\n File \"/b/home/ipgs/dmichea/venv_a2s/lib/python3.6/site-packages/fireworks/core/rocket.py\", line 224, in run\n m_action = t.run_task(my_spec)\n File \"/b/home/ipgs/dmichea/venv_a2s/lib/python3.6/site-packages/fireworks/user_objects/firetasks/a2s_tasks.py\", line 44, in run_task\n coregis_wf = coregis_S2.get_coregis_wf(src_id)\n File \"/b/home/ipgs/dmichea/A2S/code/WFlows/coregis_S2.py\", line 220, in get_coregis_wf\n coregis_fws = create_coregis_fws(src_id)\n File \"/b/home/ipgs/dmichea/A2S/code/WFlows/coregis_S2.py\", line 205, in create_coregis_fws\n fw_coregis = Firework([check_scratch_task, do_coregis_task], name=\"coregistration\", spec={\"_category\": \"multiseq\"})\nNameError: name 'fw_check_scratch' is not defined\n"

Here is the line 205 of /b/home/ipgs/dmichea/A2S/code/WFlows/coregis_S2.py before correction : 

fw_coregis = Firework([check_scratch_task, do_coregis_task], name="coregistration", parents = [fw_check_scratch], spec={"_category": "multiseq"})

Here is the line after correction (the fw defined as parent does not exists anymore -> changed to a task included in this FW):

fw_coregis = Firework([check_scratch_task, do_coregis_task], name="coregistration", spec={"_category": "multiseq"})

As you can see, the stacktrace print the line 205, and there is no more 'parents' parameter in it (fw_coregis = Firework([check_scratch_task, do_coregis_task], name=\"coregistration\", spec={\"_category\": \"multiseq\"})). I can check it importing the module manually : the code is correct now

And it stops on the same bug ...

But I was wrong in my first post : this is not a PyTask, it is a custom Task : class AppendCoregisTask in module a2s_tasks: 

/b/home/ipgs/dmichea/venv_a2s/lib/python3.6/site-packages/fireworks/user_objects/firetasks/a2s_tasks.py
->

#!/usr/bin/env python3

import logging

from fireworks.core.firework import FWAction, Firework, FireTaskBase, Workflow

import mpic_S2

import attributes_S2

import coregis_S2

import correl_S2

import A2S_config

import import_S2

...

class AppendCoregisTask(FireTaskBase):

        _fw_name = 'AppendCoregisTask'

        required_params = ["src_id"]

        def run_task(self, fw_spec):

                (A2S_logger, __) = A2S_config.init_loggers()

                src_id = self["src_id"]

                coregis_wf = coregis_S2.get_coregis_wf(src_id)

                if coregis_wf:

                        A2S_logger.info("create & append co-registration workflow")

                        return FWAction(additions=coregis_wf)

                else:

                        A2S_logger.fatal("coregis_S2.get_coregis_wf failed : no co-registration workflow created for src id %s" % str(src_id))

                return FWAction()

...

So could it be linked to my problem ?

PS: The JSON of the FW (seen from the web GUI, I don't now how to have this info from CLI ...)

Firework 2359 : append Coregistration WF

created on 07/28/2017Collapse Expand Toggle Toggle level1 Toggle level2

{

• "_id": null,
• "archived_launches": [ ],
• "created_on": "2017-07-28T08:57:45.892309",
• "fw_id": 2359,
• -
  "launches": [
  • -
    {
    • "_id": null,
    • -
      "action": {
      • "additions": [ ],
      • "defuse_children": false,
      • "defuse_workflow": false,
      • "detours": [ ],
      • "exit": true,
      • "mod_spec": [ ],
      • -
        "stored_data": {
        • -
          "_exception": {
          • "_details": null,
          • "_failed_task_n": 0,
          • "_stacktrace": "Traceback (most recent call last):\n File \"/b/home/ipgs/dmichea/venv_a2s/lib/python3.6/site-packages/fireworks/core/rocket.py\", line 224, in run\n m_action = t.run_task(my_spec)\n File \"/b/home/ipgs/dmichea/venv_a2s/lib/python3.6/site-packages/fireworks/user_objects/firetasks/a2s_tasks.py\", line 44, in run_task\n coregis_wf = coregis_S2.get_coregis_wf(src_id)\n File \"/b/home/ipgs/dmichea/A2S/code/WFlows/coregis_S2.py\", line 220, in get_coregis_wf\n coregis_fws = create_coregis_fws(src_id)\n File \"/b/home/ipgs/dmichea/A2S/code/WFlows/coregis_S2.py\", line 205, in create_coregis_fws\n fw_coregis = Firework([check_scratch_task, do_coregis_task], name=\"coregistration\", spec={\"_category\": \"multiseq\"})\nNameError: name 'fw_check_scratch' is not defined\n"
          },
        • "_message": "runtime error during task",
        • -
          "_recovery": {
          • "_all_mod_spec": [ ],
          • "_all_stored_data": { },
          • "_all_update_spec": { }
          },
        • -
          "_task": {
          • "_fw_name": "AppendCoregisTask",
          • "src_id": 16314
          }
        },
      • "update_spec": { }
      },
    • "fw_id": 2359,
    • -
      "fworker": {
      • "category": "multiseq",
      • "env": { },
      • "name": null,
      • "query": "{}"
      },
    • "host": "hpc-n567",
    • "ip": "192.168.1.218",
    • "launch_dir": "/b/home/ipgs/dmichea/A2S/launchdir/multiseq/block_2017-07-26-12-37-31-425159/launcher_2017-07-28-08-42-25-415961/launcher_2017-07-28-09-05-21-905507",
    • "launch_id": 2317,
    • "runtime_secs": 0.37215,
    • "state": "FIZZLED",
    • -
      "state_history": [
      • -
        {
        • "created_on": "2017-07-28T09:05:21.919802",
        • "state": "RUNNING",
        • "updated_on": "2017-07-28T09:05:22.286604"
        },
      • -
        {
        • "created_on": "2017-07-28T09:05:22.291952",
        • "state": "FIZZLED"
        }
      ],
    • "time_end": "2017-07-28T09:05:22.291952",
    • "time_start": "2017-07-28T09:05:21.919802",
    • "trackers": [ ]
    }
  ],
• "name": "append Coregistration WF",
• -
  "spec": {
  • "_category": "multiseq",
  • -
    "_tasks": [
    • -
      {
      • "_fw_name": "AppendCoregisTask",
      • "src_id": 16314
      }
    ]
  },
• "state": "FIZZLED",
• "updated_on": "2017-07-28T09:05:22.307789"

}

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/CAKubV2JqpGa9b3retb%3DPX1_GVC8jPPKz3pkbiemKxbbR0o8mtw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

[David Michéa]

Erratum : This is not a PyTask (as said in my first statement), it is a custom firetask which executes this code from one of its imported module.

[Anubhav Jain]

Hi David

I am not really sure how to solve this; we do update our Firetasks code all the time and upon rerun the updated version is loaded. FireWorks does not store the code anywhere.

As far as I can tell, some outdated version of coregis_S2 is being loaded.

On your worker, with the same Python environment loaded as when FWS is executing, you could try to execute the command: 

python -c "exec(\"import coregis_S2\\nprint(coregis_S2.__file__)\")" >> test_import.txt

Then, check text_import.txt and ensure that the printed out location of co_regis_S2 is the correct location of your code.

If you are running through qlaunch, you can also add that command to the pre_rocket field of your my_qadapter.yaml file, which will ensure that the command is executed in exactly the same Python environment as FireWorks and on the same machine, etc. Check test_import.txt again after executing the FW rerun and see what it says. Whatever file is printed out there should be the same file that FWS is finding for coregis_S2.

Best,

Anubhav