Execution stops at gaussian scale space, but with fastr execute it works

7 views
Skip to first unread message

Mattias Hansson

unread,
Jun 28, 2017, 2:27:07 PM6/28/17
to Fastr Users
When i run this script it fails at gaussian scale space, but with fastr execute it works.

Script: /home/mhansson/fastr-scripts/PROOF_scripts/MultiAtlasApp_PROOF_FemkeSegmTEST_LEFT_TrainClassifier.py

Excerpt from output:

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF227_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO] drmaaexecution:0205 >> Job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF311_MRI_L_T30 not found in DRMAA lookup

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF311_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF390_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO] drmaaexecution:0205 >> Job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF390_MRI_L_T0 not found in DRMAA lookup

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF390_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF244_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO] drmaaexecution:0205 >> Job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF380_MRI_L_T0 not found in DRMAA lookup

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF380_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF380_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF197_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF197_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF036_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF036_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF173_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF173_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF173_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF227_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF380_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF128_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF103_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF197_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF357_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF345_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF244_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO] drmaaexecution:0205 >> Job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF345_MRI_L_T30 not found in DRMAA lookup

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF345_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF390_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF244_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF227_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF128_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF269_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF357_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO] drmaaexecution:0205 >> Job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF357_MRI_L_T0 not found in DRMAA lookup

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF357_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO] drmaaexecution:0205 >> Job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF269_MRI_L_T0 not found in DRMAA lookup

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF269_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO] drmaaexecution:0205 >> Job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF403_MRI_L_T0 not found in DRMAA lookup

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF403_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF103_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF103_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF345_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF311_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF150_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___morph___PROOF150_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/morph/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF128_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF311_MRI_L_T30 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF403_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___threshold___PROOF269_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/threshold/outputs/image in {}!

[INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___scalespacefilter___PROOF403_MRI_L_T0 with status JobState.cancelled

[ERROR]      node:1192 >> Could not find required data for fastr:///networks/MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier/nodelist/scalespacefilter/outputs/image in {}!

Traceback (most recent call last):

  File "MultiAtlasApp_PROOF_FemkeSegmTEST_LEFT_TrainClassifier.py", line 124, in <module>

    main()

  File "MultiAtlasApp_PROOF_FemkeSegmTEST_LEFT_TrainClassifier.py", line 120, in main

    network.execute(sourcedata, sinkdata)

  File "/home/mhansson/virtual_env_fastr_develop/fastr/fastr/core/network.py", line 995, in execute

    joblist = node.execute()

  File "/home/mhansson/virtual_env_fastr_develop/fastr/fastr/core/node.py", line 1165, in execute

    for sample_index, sample_id, job_data, job_depends, failed_annotations in self.input_group_combiner:

  File "/home/mhansson/virtual_env_fastr_develop/fastr/fastr/core/node.py", line 469, in iter_input_groups

    for sample_payloads in itertools.product(*[ig.iterinputvalues for ig in self.input_groups.values()]):

  File "/home/mhansson/virtual_env_fastr_develop/fastr/fastr/core/node.py", line 300, in iterinputvalues

    yield self[index]

  File "/home/mhansson/virtual_env_fastr_develop/fastr/fastr/core/node.py", line 164, in __getitem__

    target_index=index)

  File "/home/mhansson/virtual_env_fastr_develop/fastr/fastr/core/node.py", line 260, in find_source_index

    target_index=target_index)

  File "/home/mhansson/virtual_env_fastr_develop/fastr/fastr/core/node.py", line 288, in solve_broadcast

    raise exceptions.FastrSizeMismatchError('The estimated size after broadcast matching is incorrect! (estimated {}, actual {})'.format(estimated_source_size, source_size))

Mattias Hansson

unread,
Jun 28, 2017, 2:28:11 PM6/28/17
to Fastr Users
I am running fastr develop

Hakim Achterberg

unread,
Jun 29, 2017, 8:31:13 AM6/29/17
to Fastr Users
Hi Mattias,

The reason fastr crashes and throws and exception is because you appearantly are using 13 normal images and 15 label images in your sources:
16 [INFO]      node:1380 >> Preparing Node images with size [13] and dimnames ('atlas',)

42 [INFO]      node:1380 >> Preparing Node label_images with size [15] and dimnames ('atlas',)

that don't match and give trouble in the sampler node where they should come together.

I am not sure why you think the GaussianScaleSpace fails, but it seems to be doing okay. There was an issue with a parameter file that couldn't be found:
104 [INFO]   network:1109 >> Finished job MultiAtlasApp_PROOF_FEMKE_LEFT_TrainClassifier___param_file___id_0 with status JobState.failed

which caused all the classification jobs to cancel at some point.

Cheers,
Hakim
Reply all
Reply to author
Forward
0 new messages