Tasks disappearing from visualizer dashboard after a while

[hal...@mojn.com]

Hi all

After a while (what seems about 24 hours or less) a task, failed or completed will disappear from the dashboard. Is there any way we can persist the tasks longer than that?

With best regards

Chris Hallas

[Erik Bernhardsson]

Without using the task history db, your tasks will be pruned after a while

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[hal...@mojn.com]

Hi Erik. Thanks for taking the time to answer. Much appreciated.

How about persistence with the task history db setup? Do they sit there forever?

[hal...@mojn.com]

Hi Erik

Sorry to bother you again. Our scheduler is set up to use an sqlite.

I've taken a look at the Luigi source code (our organization loves Luigi btw, so thank you) and I found the following: https://github.com/spotify/luigi/blob/a0f73452c80173548a258c08f89b44edc4c771ae/luigi/scheduler.py#L80-L98

I'm having a harder time than I should understanding if I can use any of these options to either enable persistence forever or to increase the time limit before tasks are pruned.

Can you specify what I should do?

[Arash Rouhani]

Hi there,

Have you seen this document describing the task history service? http://luigi.readthedocs.org/en/latest/central_scheduler.html?highlight=task%20history#enabling-task-history

/Arash

On Thu, Apr 9, 2015 at 10:37 AM, <nexus...@gmail.com> wrote:

[Christoffer Hallas]

Hi Arash

Thanks. I can’t say that I have. I dunno how I missed it. I’m sorry that I’ve disturbed you guys.

Chris

—
Sent from Mailbox

[Arash Rouhani]

No worries! :)

If you find any improvements, please share it by sending a pull request!

Kind Regards,

Arash

[James Chen]

I apologize ahead if there's a known solution to this, but I still can't seem to get the task history showing on the task visualizer a bit longer.
Yes I've enabled job history logging to a sqlite database, and I can view the all history through http://host/history, but the tasks still disappear from the task visualizer 10 minutes after they run - even the failed ones.
I tried running tasks using "--scheduler-remove-delay 86400", but it didn't make any difference.
Am I still missing something?
Thanks in advance!

-James

[bcha...@gmail.com]

I am having the same issue my /history contains some tasks but the luigi UI does not show any Done tasks
I am using sqlite

[jun...@gmail.com]

On Wednesday, April 13, 2016 at 6:24:13 AM UTC+8, bcha...@gmail.com wrote:
> I am having the same issue my /history contains some tasks but the luigi UI does not show any Done tasks
> I am using sqlite

me too :(

[mgol...@gmail.com]

Any feedback on this issue? This is a major feature of luigi and I would like to be able to check reasons for failure and failed jobs outside of the 10 minute window

[limw...@gmail.com]

me too..any solution?

[Dave Buchfuhrer]

If you just want a record of error messages, you can turn on error e-mails.

--
You received this message because you are subscribed to the Google Groups "Luigi" group.

To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+unsubscribe@googlegroups.com.

[r.j....@gmail.com]

See https://stackoverflow.com/a/44708942/3752859

There is a configuration option that was incorrectly documented.