Why does this fail?

18 views
Skip to first unread message

sprock

unread,
Apr 8, 2022, 12:28:23 PM4/8/22
to Cooperative Computing Tools
Hello,

Here is my makeflow file:
URL="/home/rmason/Scratch/elkjobs"
WD="1x1x1_220408_132509"
CP=/bin/cp
MAKEFLOW_INPUTS=$(WD)/

$(URL)/$(WD): $(WD)
    $(CP) -r $(WD) $(URL)

# #$(URL)/$(WD)/runsbatch: ./runsbatch.sh
# #    $(CP) runsbatch.sh $(URL)/$(WD)/runsbatch

$(URL)/$(WD)/runspecies.txt: $(URL)/$(WD)/runspecies.sh
    cd $(URL)/$(WD) && ./runspecies.sh > runspecies.txt

# $(URL)/$(WD)/dirs: $(URL)/$(WD)/runspecies.txt
#     cd $(URL)/$(WD) && find . -type d -depth 1 > dirs

# $(URL)/$(WD)/done: $(URL)/$(WD)/dirs
#     cd $(URL)/$(WD) && ./runsbatch.sh

Most of it is commented out for testing.

If i run only the first rule, it succeeds.  If I run the first and third rules  I get:

parsing runelk_1x1x1_220408_132509.mfl...
local resources: 1.000 cores, 0 MB memory, 2061949 MB disk
max running local jobs: 1
checking runelk_1x1x1_220408_132509.mfl for consistency...
runelk_1x1x1_220408_132509.mfl has 2 rules.
recovering from log file runelk_1x1x1_220408_132509.mfl.makeflowlog...
checking for old running or failed jobs...
checking files for unexpected changes...  (use --skip-file-check to skip this step)
error: /home/rmason/Scratch/elkjobs/1x1x1_220408_132509/runspecies.sh does not exist, and is not created by any rule.
found 1 errors and 0 warnings during consistency check.
workflow failed.

Indeed,  /home/rmason/Scratch/elkjobs/1x1x1_220408_132509/runspecies.sh does not exist because the first rule fails!

What is going on?

Thanks for any help,
Roger

Ben Tovar

unread,
Apr 8, 2022, 1:31:16 PM4/8/22
to cctoo...@googlegroups.com
Roger,

makeflow does not have a way of knowing that the first rule generates the file $(URL)/$(WD)/runspecies.sh. It only knows to check that the path $(URL)/$(WD) is created and has read permissions. Thus, it does not know that the first rule in the file should be executed before the second, and does not even try to run the workflow. Note that no rule fails, the workflow is failing a consistency check. Rules are executed according to their input and output dependencies, and not according to their order in the file.

A quick fix for the above would be:


URL="/home/rmason/Scratch/elkjobs"
WD="1x1x1_220408_132509"
CP=/bin/cp
MAKEFLOW_INPUTS=$(WD)/

$(URL)/$(WD)/runspecies.sh: $(WD)
    $(CP) -r $(WD) $(URL)

$(URL)/$(WD)/runspecies.txt: $(URL)/$(WD)/runspecies.sh
    cd $(URL)/$(WD) && ./runspecies.sh > runspecies.txt

--
You received this message because you are subscribed to the Google Groups "Cooperative Computing Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cctools-nd+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/abe7ece7-bac7-4d8d-90db-d9847a2f0486n%40googlegroups.com.

Roger Mason

unread,
Apr 8, 2022, 1:50:27 PM4/8/22
to cctoo...@googlegroups.com
Hello Ben,

Ben Tovar <bto...@nd.edu> writes:

> A quick fix for the above would be:
>
> URL="/home/rmason/Scratch/elkjobs"
> WD="1x1x1_220408_132509"
> CP=/bin/cp
> MAKEFLOW_INPUTS=$(WD)/
>
> $(URL)/$(WD)/runspecies.sh: $(WD)
> $(CP) -r $(WD) $(URL)
>
> $(URL)/$(WD)/runspecies.txt: $(URL)/$(WD)/runspecies.sh
> cd $(URL)/$(WD) && ./runspecies.sh > runspecies.txt

Unfortunately that makes no difference.

Thanks for your help.
Roger

Ben Tovar

unread,
Apr 8, 2022, 1:55:41 PM4/8/22
to cctoo...@googlegroups.com
Rogerm

What do you get? Do you still get a consistency error?

--
You received this message because you are subscribed to the Google Groups "Cooperative Computing Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cctools-nd+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/y655ynjo467.fsf%40mun.ca.

Roger Mason

unread,
Apr 8, 2022, 2:27:25 PM4/8/22
to cctoo...@googlegroups.com
Hello Ben,

Ben Tovar <bto...@nd.edu> writes:

> Rogerm
>
> What do you get? Do you still get a consistency error?
>
makeflow runelk_1x1x1_220408_132509.mfl

parsing runelk_1x1x1_220408_132509.mfl...
local resources: 1.000 cores, 0 MB memory, 2061375 MB disk
max running local jobs: 1
checking runelk_1x1x1_220408_132509.mfl for consistency...
runelk_1x1x1_220408_132509.mfl has 2 rules.
recovering from log file runelk_1x1x1_220408_132509.mfl.makeflowlog...
checking for old running or failed jobs...
checking files for unexpected changes... (use --skip-file-check to skip this step)
error: /home/rmason/Scratch/elkjobs/1x1x1_220408_132509/runspecies.sh does not exist, and is not created by any rule.
found 1 errors and 0 warnings during consistency check.
workflow failed.

Thanks again,
Roger

Ben Tovar

unread,
Apr 8, 2022, 2:37:49 PM4/8/22
to cctoo...@googlegroups.com
Roger,

It does seem to work for me over here:

Makeflow file:

URL="/home/rmason/Scratch/elkjobs"
WD="1x1x1_220408_132509"
CP=/bin/cp
MAKEFLOW_INPUTS=$(WD)/

$(URL)/$(WD)/runspecies.sh: $(WD)
    $(CP) -r $(WD) $(URL)

$(URL)/$(WD)/runspecies.txt: $(URL)/$(WD)/runspecies.sh
    cd $(URL)/$(WD) && ./runspecies.sh > runspecies.txt


At the command line:
# show the inputs:
makeflow_analyze -I Makeflow  
1x1x1_220408_132509

# show the outputs:
makeflow_analyze -O Makeflow
/home/rmason/Scratch/elkjobs/1x1x1_220408_132509/runspecies.txt
/home/rmason/Scratch/elkjobs/1x1x1_220408_132509/runspecies.sh


Before retrying, could rerun makeflow using the -c switch, just to make sure we are not picking a log from a previous run, e.g.: makeflow -c runelk_1x1x1_220408_132509.mfl




--
You received this message because you are subscribed to the Google Groups "Cooperative Computing Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cctools-nd+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/y65wnfzmnwc.fsf%40mun.ca.

Roger Mason

unread,
Apr 8, 2022, 4:36:09 PM4/8/22
to cctoo...@googlegroups.com
Hello Ben,

Ben Tovar <bto...@nd.edu> writes:

> Before retrying, could rerun makeflow using the -c switch, just to make sure we are not picking a log from a previous run, e.g.: makeflow -c runelk_1x1x1_220408_132509.mfl
>

Same result:

makeflow -c runelk_1x1x1_220408_132509.mfl
parsing runelk_1x1x1_220408_132509.mfl...
local resources: 1.000 cores, 0 MB memory, 2061895 MB disk
max running local jobs: 1
checking runelk_1x1x1_220408_132509.mfl for consistency...
runelk_1x1x1_220408_132509.mfl has 2 rules.
recovering from log file runelk_1x1x1_220408_132509.mfl.makeflowlog...
checking for old running or failed jobs...
checking files for unexpected changes... (use --skip-file-check to skip this step)
error: /home/rmason/Scratch/elkjobs/1x1x1_220408_132509/runspecies.sh does not exist, and is not created by any rule.
found 1 errors and 0 warnings during consistency check.
workflow failed.

Roger

Ben Tovar

unread,
Apr 8, 2022, 4:58:29 PM4/8/22
to Cooperative Computing Tools
Could you send me your makeflow file?

Ben

--
You received this message because you are subscribed to the Google Groups "Cooperative Computing Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cctools-nd+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/y65sfqnmhxl.fsf%40mun.ca.

Roger Mason

unread,
Apr 8, 2022, 5:02:53 PM4/8/22
to cctoo...@googlegroups.com

Ben Tovar <bto...@nd.edu> writes:

> Could you send me your makeflow file?

Attached.

Cheers,
Roger

runelk_1x1x1_220408_132509.mfl

Ben Tovar

unread,
Apr 9, 2022, 8:44:33 AM4/9/22
to cctoo...@googlegroups.com
Roger,

It seems that in the workflow you sent me the first rule still does not list $(URL)/$(WD)/runspecies.sh as an output. The first rule should be something like:

$(URL)/$(WD)/runspecies.sh: $(WD)
    $(CP) -r $(WD) $(URL)

If I create a mock file 1x1x1_220408_132509/runspecies.sh and a mock file then the workflow executes correctly for me. Please let me know if that works for you.


Ben

--
You received this message because you are subscribed to the Google Groups "Cooperative Computing Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cctools-nd+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/y65o81bmgp2.fsf%40mun.ca.

Roger Mason

unread,
Apr 9, 2022, 10:44:12 AM4/9/22
to cctoo...@googlegroups.com
Hi Ben,

Ben Tovar <bto...@nd.edu> writes:

> It seems that in the workflow you sent me the first rule still does
> not list $(URL)/$(WD)/runspecies.sh as an output. The first rule
> should be something like:
>
> $(URL)/$(WD)/runspecies.sh: $(WD)
> $(CP) -r $(WD) $(URL)
>
> If I create a mock file 1x1x1_220408_132509/runspecies.sh and a mock file then the workflow executes correctly for me. Please let me know if that works for you.

Yes, now it works. Clearly I need to be much more careful in
constructing worflows.

Many, many thanks for your help & patience.

Roger

Ben Tovar

unread,
Apr 9, 2022, 11:02:01 AM4/9/22
to Cooperative Computing Tools
Thanks for letting us know! Please don't hesitate if you have any more questions.

Best wishes,

Ben

--
You received this message because you are subscribed to the Google Groups "Cooperative Computing Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cctools-nd+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/y65a6cunwp3.fsf%40mun.ca.
Reply all
Reply to author
Forward
0 new messages