Running jobs from threads

[mojmir....@gmail.com]

Hi,

I'm trying to run luigi job from http request using

luigi.build([task], local_scheduler)

However, I'm getting "ValueError: signal only works in main thread" (probably not Luigi's fault, but Twisted's). I'm able to start a job with command using os.system('luigi --module ...'), but I was hoping for more straightforward way. Perhaps submitting a job using REST api?

Thanks for any hints!
Mojmir

[Arash Rouhani Kalleh]

You're luckily not the first person with this problem. You can sacrifice the SIGUSR functionality and then run from any thread.

https://github.com/spotify/luigi/blob/64d6b831f/luigi/worker.py#L336-L338

Cheers,

Arash

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mojmír Vinkler]

Works perfectly, thanks! Here's a code for running task from thread if anyone stumbles upon this

def port_is_open(port):

    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    result = sock.connect_ex(('127.0.0.1', port))

    return result == 0

def run_task(task):

    """ Thread safe alternative of luigi.build """

    if port_is_open(8082):

        sch = luigi.rpc.RemoteScheduler()

    else:

        sch = luigi.scheduler.CentralPlannerScheduler()

    # no_install_shutdown_handler makes it thread safe

    w = luigi.worker.Worker(scheduler=sch, no_install_shutdown_handler=True)

    w.add(task)

    w.run()