Jug hooks

[Luis Pedro Coelho]

Hello everyone,

Following a previous discussion on this group

https://groups.google.com/forum/#!topic/jug-users/Btmte0wRfYE

I implemented an experimental feature on jug to allow for, for example,
having execute exit after a certain period of time.

The idea is to allow hooks which are called after a task is executed and
which can do things including exiting the programme. So, you can
implement your own logic if you like, but I am doing a few basic ones.

This way, the main loop is not overly complicated with all these checks,
but you have a lot of flexibilty.

The attached jugfile demonstrates the use of one of the predefined checks:


from jug.hooks import exit_checks
exit_checks.exit_after_time(seconds=4)

*

Right now, this is only available on a special branch on github

https://github.com/luispedro/jug/tree/hooks

If this seems to work out, I'll merge into master. In the meanwhile, I'm
very open to feedback

See

https://github.com/luispedro/jug/blob/hooks/jug/hooks/exit_checks.py

for examples of how to write your own hooks

Tx
--
Luis Pedro Coelho | EMBL | http://luispedro.org

Recent stuff:
http://doi.org/10.1002/bies.201300143
http://dx.doi.org/10.1038/nm.3424
http://dx.doi.org/10.1038/nmeth.2693

[Sven]

Hello Luis,

I have been using the hooks branch since you announced it. For me
"exit_after_time()" and "exit_if_file_exists()" have been most helpful
so far.

The latter I used as a "make the already running tasks the last you run"
switch. (Simply by touching a file named "stop" in the code's directory
...).


However, I noticed (and I might be misinterpreting something here)
something in the logic of this particular hook:

Assume I have nothing running. Yet there are ready tasks. And the
aforementioned "stop" file exists. If I submit a job now, it will pick
one task, run it and only then take the "stop" signal into account.

Is that correct? If so, is that intended?

I at first thought this was on purpose to be able to run tasks
"stepwise". But then I noticed you provide "exit_after_n_tasks()", in
which one could use n=1 to achieve just that...


Anyway: Thanks for providing this wonderful package!
Best regards
Sven

[Luis Pedro Coelho]

Hi Sven!

> Assume I have nothing running. Yet there are ready tasks. And the
> aforementioned "stop" file exists. If I submit a job now, it will pick
> one task, run it and only then take the "stop" signal into account.
>
> Is that correct? If so, is that intended?

Yes, this is correct and is the current design.

Having said that, I think you are right that this is not very intuitive.
So, it is very good feedback. I'll consider this as a bug and fix it.

(This is exactly why it's nice to do this is a separate hook: once it's
released, it'll be harder to change things if people start expecting
them to work in a particular way [even if it's not the best way])

Thanks
Luis

[Sven]

Hi Luis,

I'm glad my comment was of any help!


I'd also have another proposal for an additional feature. However, I'm
not sure whether an implementation would be feasible, not even if one
was desirable, as it might well be that it is only necessary because I
use something wrongly...

Anyway: Could we have tasks that return something via value() (that is,
if there is already something to return), but are not started (that is,
if there is no value for them yet)?

This could be an optional boolean argument to the TaskGenerator(),
something like "dont_run=True".

Let me try to give a scenario where I think this might be useful: Assume
I had something running for some time, but then notice it is too slow
and I should do some changes. It might be that some result are already
there. I would not want those to be lost. Thus I might stop everything
(using the new hooks) and then I'd like to restart only the
postprocessing tasks (might be the sum/integration of some calculated
numbers, which was not calculated as the whole set of numbers was not
complete, or simply an output routine). Yet, some postprocessing tasks
might already be "ready". In this case it would be straight forward to
mark the heavy calculational tasks as "dont_run", and restart...



And another question: If a jug instance running in a queue (atleast in
the queue used here) is terminated by the queuing system, e.g., because
the maximum runtime is over, it seems it is not exiting cleanly. In the
task list those will still show as running. I suppose there is no way to
prevent this, besides making sure to not have tasks killed due to too
long running... But if it happens, is it safe to just remove the
respective .lock file from the juddata directory? I have done so
recently, and it seemed to work, but I'd rather make sure ...


Best!
Sven

[Luis Pedro Coelho]

On 03/19/2014 12:40 PM, Sven wrote:
> Anyway: Could we have tasks that return something via value() (that is,
> if there is already something to return), but are not started (that is,
> if there is no value for them yet)?
>
> This could be an optional boolean argument to the TaskGenerator(),
> something like "dont_run=True".

> Let me try to give a scenario where I think this might be useful: Assume
> I had something running for some time, but then notice it is too slow
> and I should do some changes. It might be that some result are already
> there. I would not want those to be lost. Thus I might stop everything
> (using the new hooks) and then I'd like to restart only the
> postprocessing tasks (might be the sum/integration of some calculated
> numbers, which was not calculated as the whole set of numbers was not
> complete, or simply an output routine). Yet, some postprocessing tasks
> might already be "ready". In this case it would be straight forward to
> mark the heavy calculational tasks as "dont_run", and restart...

I see the objective but am not 100% sure on the interface.

I'd have to think about this.

> And another question: If a jug instance running in a queue (atleast in
> the queue used here) is terminated by the queuing system, e.g., because
> the maximum runtime is over, it seems it is not exiting cleanly. In the
> task list those will still show as running. I suppose there is no way to
> prevent this, besides making sure to not have tasks killed due to too
> long running...

Dependends on how the job was killed by the queue and whether it had
enough time to clean up. The same happens to me.

But if it happens, is it safe to just remove the
> respective .lock file from the juddata directory? I have done so
> recently, and it seemed to work, but I'd rather make sure ...

Yes, that's what you should do.

You can also do it via:

jug cleanup --locks-only

HTH
Luis

[Sven]

Hi Luis,

> I see the objective but am not 100% sure on the interface.
>
> I'd have to think about this.

As said, I'm not entirely sure if a feature like this is actually
desirable. The problems it solves might as well be solved by writing
better post-processing tasks.

On the other hand, having a feature like this to salvage already made
calculations, might be helpful often enough to still have it. As things
might cause you problems that you didn't think of when writing the
post-processing routines...

> Yes, that's what you should do.
>
> You can also do it via:
>
> jug cleanup --locks-only

Ah, nice. Didn't see that switch ...

Thank you!
Sven

[Luis Pedro Coelho]

Just FYI:

I just pushed to the hooks branch a version where this issue is now fixed.

HTH
Luis

--
Luis Pedro Coelho | EMBL | http://luispedro.org

Recent stuff:
http://doi.org/10.1002/bies.201300143
http://dx.doi.org/10.1038/nm.3424
http://dx.doi.org/10.1038/nmeth.2693

On 03/19/2014 12:05 PM, Luis Pedro Coelho wrote: