Job dependencies

[Emmanuel]

So because I cannot use both mysql and postgre together, I am force to run separately 2 different ctl. So I have a job that export from mysql to CSV , then I ll have one that does CSV to postgres..

The problem is how do I know the previous job is done running? can I use the etl execution to check? any good practice around that?

Emmanuel

[Thibaut Barrère]

> The problem is how do I know the previous job is done running? can I use the etl execution to check? any good practice around that?

The built-in way to handle that is to use a batch (search for "Batching" on https://github.com/activewarehouse/activewarehouse-etl/wiki/Documentation). This allows to run jobs sequentially.

If you need to also ensure that two batches won't run at the same time, then you'll have to make a lock-system yourself (I can share code to do that if you need).

cheers,

Thibaut

--

http://www.logeek.fr

[Thibaut Barrère]

> Yes, if you have some code lock between 2 batch, let me know. Otherwise I use imple file lock ;)

Here is what I used last time, relying on flock (tested on Windows and Mac if I remember well):

https://gist.github.com/3914149

-- Thibaut

[Emmanuel]

Ok !  same thing I use for single process already in other code :) I ll figure out a way to create a dependent batch lock using that technique then :)

[Thibaut Barrère]

> Ok !  same thing I use for single process already in other code :) I ll figure out a way to create a dependent batch lock using that technique then :)

If you need something more elaborate, maybe you need a custom scheduler that will remain up and running, then!

If you go single machine, you could have a look at rufus scheduler.

If you go multiple machines (wow!), then using sidekiq or resque comes to mind :-)

-- Thibaut

[Emmanuel]

or Redis Mutex :) But for now same machine. Multiple if perf will struggle on the future ;)