bash -e

[Koert Kuipers]

i want to use the shell protocol, but with "/bin/bash -e". is that possible?

basically i would like it to exit if any step in the series of commands fails

currently in this example b succeeds, which i consider undesirable.

$ more Drakefile
a.out <-
  echo "running a" > $OUTPUT

b.out <- a.out
  blech
  echo "running b" > $OUTPUT

[Artem Boytsov]

See my previous email - this could be a parameter to Drake itself (similar to bash -e), though I also tend to think this behavior should be the default one and I'm surprised Drake doesn't behave this way now...

--
You received this message because you are subscribed to the Google Groups "drake-workflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drake-workflo...@googlegroups.com.
Visit this group at http://groups.google.com/group/drake-workflow.
For more options, visit https://groups.google.com/d/optout.

--

Artem.

[Alan Malloy]

Yeah if one command in a step fails, the whole step should fail. If we don't already do that that's super-bad IMO.

[Koert Kuipers]

$ more Drakefile
a.out <-
  echo "running a" > $OUTPUT

b.out <- a.out
  blech
  echo "running b" > $OUTPUT

$ drake
The following steps will be run, in order:
  1: /home/koert/tmp/test_project/././a.out <-  [missing output]
  2: /home/koert/tmp/test_project/././b.out <- /home/koert/tmp/test_project/././a.out [projected timestamped]
Confirm? [y/n] y
Running 2 steps with concurrence of 1...

--- 0. Running (missing output): /home/koert/tmp/test_project/././a.out <-
--- 0: /home/koert/tmp/test_project/././a.out <-  -> done in 0.02s

--- 1. Running (missing output): /home/koert/tmp/test_project/././b.out <- /home/koert/tmp/test_project/././a.out
.drake/shell-11112bcfb7e7fe47a36f6dca55491dc5.bat: line 1: blech: command not found
--- 1: /home/koert/tmp/test_project/././b.out <- /home/koert/tmp/test_project/././a.out -> done in 0.02s
Done (2 steps run).

[Alza]

Isn't this behaviour happening because Drake executes all of the commands for a given step in a single script?

I am working around it using things like this in my steps:

some-command-that-can-fail || { echo "command failed" && exit 1; }

or for stderr instead of stdout:

some-command-that-can-fail || { >&2 echo "command failed" && exit 1; }

Of course this doesn't look nice when used on every command in a step..

[Koert Kuipers]

you can also do this

$ more Drakefile
a.out <-
  echo "running a" > $OUTPUT

b.out <- a.out

  set -e

  blech
  echo "running b" > $OUTPUT
$ drake
The following steps will be run, in order:
  1: /home/koert/tmp/test_project/././a.out <-  [missing output]
  2: /home/koert/tmp/test_project/././b.out <- /home/koert/tmp/test_project/././a.out [projected timestamped]
Confirm? [y/n] y
Running 2 steps with concurrence of 1...

--- 0. Running (missing output): /home/koert/tmp/test_project/././a.out <-
--- 0: /home/koert/tmp/test_project/././a.out <-  -> done in 0.02s

--- 1. Running (missing output): /home/koert/tmp/test_project/././b.out <- /home/koert/tmp/test_project/././a.out

.drake/shell-3016e2716abb1cde071422ffae74a6c8.bat: line 2: blech: command not found
Done (1 steps run).
clojure.lang.ExceptionInfo: shell command failed with exit code 127 {:file "set -e\nblech\necho \"running b\" > $OUTPUT\n", :msg "shell command failed with exit code 127", :cmd ("/bin/bash" ".drake/shell-3016e2716abb1cde071422ffae74a6c8.bat"), :opts {:die true, :no-stdin nil, :err [#<PrintStream java.io.PrintStream@699dbc28> #<BufferedWriter java.io.BufferedWriter@72ac1c43>], :out [#<PrintStream java.io.PrintStream@173ad86> #<BufferedWriter java.io.BufferedWriter@51853928>]}, :exit-code 127}
    at clojure.core$ex_info.invoke(core.clj:4403)
    at drake.shell$shell.doInvoke(shell.clj:143)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invoke(core.clj:624)
    at drake.protocol$run_interpreter.invoke(protocol.clj:86)
    at drake.protocol_interpreters$register_interpreter_BANG_$reify__3071.run(protocol_interpreters.clj:13)
    at drake.core$run_step.invoke(core.clj:354)
    at drake.core$attempt_run_step.invoke(core.clj:466)
    at drake.core$function_for_step$fn__5272.invoke(core.clj:489)
    at drake.core$trigger_futures_helper$fn__5283.invoke(core.clj:527)
    at clojure.core$binding_conveyor_fn$fn__4145.invoke(core.clj:1910)
    at clojure.lang.AFn.call(AFn.java:18)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)

[Artem Boytsov]

On Fri, Mar 6, 2015 at 5:43 AM, Alza <abc...@gmail.com> wrote:

Isn't this behaviour happening because Drake executes all of the commands for a given step in a single script?

It does, but the script returns last process' exit code as its exit code:

artem:~$ cat>test.sh
echo hey there
false
artem:~$ bash test.sh || echo failure
hey there
failure

So in the simplistic example you provided before, Drake should have failed the step and stop the workflow. 

I am working around it using things like this in my steps:

some-command-that-can-fail || { echo "command failed" && exit 1; }

If this is the last command of your step, it shouldn't be needed. But if this is in the middle of the step, this is what you would need to do. You can also try setting SHELL environment variable to "bash -e" (not sure if this would work inside the workflow, but maybe it should!). We could also make it a step option, but not for SHELL protocol as it's agnostic of the shell it uses. But it's possible to add BASH protocol to Drake which would have such option.

or for stderr instead of stdout:

some-command-that-can-fail || { >&2 echo "command failed" && exit 1; }

Of course this doesn't look nice when used on every command in a step..

On Friday, 6 March 2015 01:36:10 UTC, Koert Kuipers wrote:

$ more Drakefile
a.out <-
  echo "running a" > $OUTPUT

b.out <- a.out
  blech
  echo "running b" > $OUTPUT
$ drake
The following steps will be run, in order:
  1: /home/koert/tmp/test_project/././a.out <-  [missing output]
  2: /home/koert/tmp/test_project/././b.out <- /home/koert/tmp/test_project/././a.out [projected timestamped]
Confirm? [y/n] y
Running 2 steps with concurrence of 1...

--- 0. Running (missing output): /home/koert/tmp/test_project/././a.out <-
--- 0: /home/koert/tmp/test_project/././a.out <-  -> done in 0.02s

--- 1. Running (missing output): /home/koert/tmp/test_project/././b.out <- /home/koert/tmp/test_project/././a.out
.drake/shell-11112bcfb7e7fe47a36f6dca55491dc5.bat: line 1: blech: command not found
--- 1: /home/koert/tmp/test_project/././b.out <- /home/koert/tmp/test_project/././a.out -> done in 0.02s
Done (2 steps run).

On Thu, Mar 5, 2015 at 8:30 PM, Alan Malloy <al...@factual.com> wrote:

Yeah if one command in a step fails, the whole step should fail. If we don't already do that that's super-bad IMO.

--
You received this message because you are subscribed to the Google Groups "drake-workflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drake-workflo...@googlegroups.com.
Visit this group at http://groups.google.com/group/drake-workflow.
For more options, visit https://groups.google.com/d/optout.

--

Artem.

[Artem Boytsov]

Oh, nice, I completely forgot about "set -e". :)

--
You received this message because you are subscribed to the Google Groups "drake-workflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drake-workflo...@googlegroups.com.
Visit this group at http://groups.google.com/group/drake-workflow.
For more options, visit https://groups.google.com/d/optout.

--

Artem.