options in documentation get me 'Unrecognized option: did you mean target exclusion?'

[Jillian Rowe]

Hello,

Thanks a lot for Drake! Its really great! I've got it installed all over the place now. ;-)

I am using Drake Version 0.1.7 .

I am seeing a few options specified in the docs (https://docs.google.com/document/d/1bF-OKNLIG10v_lMes_m4yyaJtAaJKtdK0Jizvi_MNsg/edit#) that do report  'Unrecognized option: did you mean target exclusion?' when run.

"However, if the step has async option, or --async command line option was given"

 drake_workflow git:(develop) ✗ drake --async -w workflow.d 

drake: unrecognized option '--async'

?

The same for --history and --revert. Are these still in development?

Another documentation request is to add custom logging. Is this possible? Specifically I would like my jobs to log to a sqlite database.

Thanks for the great tool!

Best,

Jillian

[al...@malloys.org]

Those options are all colored in red which, if you refer to the color-coding guide at the top of the spec, means "not implemented". I guess they are "things somebody once thought would be a cool idea, but nobody has chosen to do". In the specific case of async, though, you may be satisfied with the -j flag (this is mentioned the output of drake --help, which is a more realistic guide to options available, since it is locked to options actually implemented).

As for logging to somewhere else, drake uses a logging framework that loads a logging implementation from the classpath. So you can effectively replace drake's logging statements by putting the logging library of your choice on the classpath before it (eg, via java -cp mylogger.jar:drake.jar drake.core ...), and then configuring your logging library however it expects to be configured (eg, by placing a log4j.properties file on the classpath).

[Jillian Rowe]

Ah alright. Thanks. I think the --branch --merge branch options are fine
for versioning.

The async option though would be VERY nice to have. Are there plans for
that?

I suppose I'll get working on the log4j custom logging.

Thanks!

Best,
Jillian

[al...@malloys.org]

Did you look at the -j option like I suggested? It is probably the same as what you want --async to be.

[Jillian Rowe]

I suppose it is just about the same. My machine only has so many threads. 

I saw here:

https://github.com/Factual/drake/wiki/Async-Execution-of-Steps

That with Async steps stdin redirection could fail.

does that mean something like

file < do stuff  

Would fail?

What about piped commands?

I'm asking because I'm thinking about extending a perl module I'm writing to support drake workflow outputs, but I work in the bioinformatics/genomics fields where most of the commands have pipes or stdin/stdout redirection. For instance

         bcftools view {$self->indir}/{$sample}.vcf.gz | sed 's/ID=AD,Number=./ID=AD,Number=R/' \

                | vt decompose -s - \

                | vt normalize -r $REFGENOME - \

                | java -Xmx4G -jar $SNPEFF/snpEff.jar -c \$SNPEFF/snpEff.config -formatEff -classic GRCh37.75  \

                | bgzip -c > \

                {$self->{outdir}}/{$sample}.norm.snpeff.gz && tabix {$self->{outdir}}/{$sample}.norm.snpeff.gz

Is the kind of command we see quite a bit.

To break it down a bit

bcftools view {$self->indir}/{$sample}.vcf.gz | sed 's/ID=AD,Number=./ID=AD,Number=R/' | vt decompose -s - 

in "-s -" the second "-" means read data from STDIN.

Would strange things happen if using drake and asynchronous workflows?

Best,

Jillian

[Alan Malloy]

You can do any amount of redirection between commands in a single task. The warning about redirection is just saying that you can't reliably read from the stdin of the drake process itself, if you use async tasks.