Long running jobs issue

[Drazen D. Babic]

Hi Andrew,

regarding the thread for translations (not easy to spot any more), could you advice on this please:

- is there anything Jam can do with a long running jobs on the server side like this one? The Page is BLANK for 20 minutes or so, and if interrupted I think the server side is stopped.

Would some kind of progress bar be better? But than, do we really care when and how the Server Side finishes the job? Because there is no visibility in success just like in Demo/Mail.

Thoughts?

[Andrew Yushev]

Dean, 

Requests from the client can be synchronous and asynchronous.

If the request is asynchronous the server processes it until the end, nevertheless of the actions on the client 

and then sends the response.

You can write the code for the progress bar. 

The way to do it is to control the execution and save the percentage of the work done in the record of some table.

Then you can read from the client this value and update the progress bar.

Unfortunately there is no general solution for progress bars.

вт, 29 июн. 2021 г. в 13:27, Drazen D. Babic <bab...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "Jam.py Users Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jam-py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jam-py/d82eafaf-df73-4504-aa04-1a2465194ff8n%40googlegroups.com.

[Drazen D. Babic]

Thanks

I am more after how can Jam do long running job which is started by the click on the button.

This is a good example:

https://nordicapis.com/asynchronous-apis-using-flask-celery-and-redis/

I think Fabio's translation solution is a candidate for a full Server background job. But than, we have no example at all which

covers running long jobs within Jam. Progress bar does not solve long running jobs....

This would be similar situation with a HUGE server side calculations started by the button.  

D.

[Maxwell Morais]

Hi Drazen!

Here is the documentation about how jam handles long pooling tasks

https://jam-py.com/docs/how_to/how_to_calculations_in_the_background.html

On my overview, it's not good enough since the GIL, you can replace `threading` by `subprocessing`, but still there's no guarantee that the Jam-py application will continue in memory when the response get finished!

A good way to go, will raise an webhook, from the long pooling task, or to update the status, or to store the result somewhere.

Regarding celery, I would suggest Huey, because it's simplicity, it just works with any framework

https://github.com/coleifer/huey

It support many backends, redis, sqlite, disk storage, and memory storage, without over complicate, the generation of tasks, like the case of celery.

The key point is start the worker server together with jam-py application, but a shell script can address this issue, or we can change the code of jam-py, for it automatically start an sub-process, if huey is not running.

Also, I do have my own task scheduler, https://gist.github.com/MaxMorais/98db2175d7a4e79a1316ee9bb6e24330, but I use it as an Microservice API Architecture, plug-in different  codes, implemented in different languages, but this code is small enough, for we build a zero-dependency for jam-py, but a CLI will be required to jam-py!

To view this discussion on the web visit https://groups.google.com/d/msgid/jam-py/153a81cf-c36e-493c-89fd-d33e3c0b1bfdn%40googlegroups.com.

--

Best Regards.

Maxwell Morais

Python Developer powered by Frappé frameworks <3

+55 11 947248832

[Drazen D. Babic]

Thanks Max, 

I am getting:

AttributeError: 'Item' object has no attribute 'lock'

from Jam doco procedure...

What I did, I just added above to mail.py Demo/Server Module, and mapped button to "background" instead of send_email to get it running when Email is sent - just as an example....

Something is wrong.

Andrew, thoughts?

[Drazen D. Babic]

If I copy this into Task/Server, it will work. However, I think that it is not possible to change anything in Task/Server while App is running.

Which means the App/HttpServer needs a restart. This rings a bell from years ago....

So two things, how to map this job to a button and how to change the above behavior? 

Thanks

[Maxwell Morais]

Drazen, regaring the attribute error, I think you need to enable the record lock

https://jam-py.com/docs/admin/record_locking.html

--
You received this message because you are subscribed to the Google Groups "Jam.py Users Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jam-py+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jam-py/cfa8c534-69cc-4099-9264-82d5d06fd1e3n%40googlegroups.com.

[Maxwell Morais]

I dont got the other points

[Drazen D. Babic]

Thx Max,

when we want to change something in a job code on Task/Server, I think the restart is needed.

[Maxwell Morais]

Drazen, if you aren't in debug mode that's true!, but keep the debug enabled is a security flaw!

Due this documented issue https://medium.com/swlh/hacking-flask-applications-939eae4bffed

It's an issue allowed by werkzeug!

An easy way to resolve that issue, was if the python code was somehow reflected in modules.

If that was the case, we can simply call `reload(my_module)` and it will refresh the code in memory.

To view this discussion on the web visit https://groups.google.com/d/msgid/jam-py/3c3d1033-98f2-428f-9d7c-8a90da5325f9n%40googlegroups.com.

[Drazen D. Babic]

True.

Re attribute error, I think Mail is a virtual table, there is no record lock. 

I think the below "load_all_cores" code never executes and can't figure why:

import threading

import time

import traceback

import datetime

from cpu_load_generator import load_all_cores

def background(task):

    now = datetime.datetime.now()

    print('background started' + str(now))

    interval = 3 * 10

    time.sleep(interval)

    while True:

        if not time:

            return

        with task.lock('background'):

            try:

                now = datetime.datetime.now()

                print('background load' + str(now))

                load_all_cores(duration_s=20, target_load=0.5)  

            except Exception as e:

                traceback.print_exc()

        time.sleep(interval)

def bck(task):

    bg = threading.Thread(target=background, args=(task,))

    bg.daemon = True

    bg.start()

The bck is mapped to a button. Only first print runs:

.

background started2021-07-02 15:25:50.269311

.

Mind you, if bck is replaced with on_created, and copy/paste to Task/Server, it will work. 

Use: pip install cpu-load-generator

Thoughts?

[Drazen D. Babic]

Hi, 

Map a button on some form like this:

    item.add_button(item.view_form.find(".form-footer"), 'Load CPU').click(function() {

        item.server('bck');

    });

Paste above code into form Server Module. Do the pip part. Click the button

load_all_cores will not run due to:

AttributeError: 'Item' object has no attribute 'lock'

Tested with v112 and v6...

Of course, this will not solve the issue with running jobs, even if the button works. It is just the matter of time when someone will need this imo.

Good reading:

https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xxii-background-jobs

So here is the thing, the above article is easy, but when I try to integrate with Jam, here is what happens in rq queue when "redis" is mapped to button:

from redis import Redis

import rq

def redis(item):

    print('Redis started)')

    queue = rq.Queue('microblog-tasks', connection=Redis.from_url('redis://127.0.0.1:6379'))

    job = queue.enqueue(example, 23)

def example(seconds):

    print('Starting task')

    for i in range(seconds):

        print(i)

        time.sleep(1)

    print('Task completed')

.

.

    raise ValueError('Invalid attribute name: %s' % name)

ValueError: Invalid attribute name: als.journals.email_aliases.example

So rq fails because I have no idea how to map "example" to a valid queue. Why is rq getting als.journals.email_aliases.example?

To demonstrate how nicely the above works from the article (I just created tasks.py in a folder):

>>> from redis import Redis

>>> import rq

>>> queue = rq.Queue('microblog-tasks', connection=Redis.from_url('redis://127.0.0.1:6379'))

>>> job = queue.enqueue('tasks.example', 23)

>>> job.is_finished

False

>>> job.meta

{}

>>> job.get_id()

'43b45dad-5d3c-4ffc-a236-3222fec1ed03'

>>> job.refresh()

>>> job.meta

{'progress': 56.52173913043478}

>>> job.refresh()

>>> job.meta

{'progress': 73.91304347826087}

>>> job.refresh()

>>> job.meta

{'progress': 95.65217391304348}

>>> job.is_finished

True

And here is the result from the  queue:

.

.

.

16:05:30 microblog-tasks: tasks.example(23) (43b45dad-5d3c-4ffc-a236-3222fec1ed03)

Starting task

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

Task completed

16:05:53 microblog-tasks: Job OK (43b45dad-5d3c-4ffc-a236-3222fec1ed03)

16:05:53 Result is kept for 500 seconds

.

.

Absolutely no idea how to do it with Jam....

Thanks

[Maxwell Morais]

Drazen, python rq, will require a dedicated process aside of jam-py, to process.

Like:

python server.py

python tasks.py

To view this discussion on the web visit https://groups.google.com/d/msgid/jam-py/534f21cf-9d1f-4994-a0c5-584cda276dcen%40googlegroups.com.

[Dražen Babić]

hi Max

I think Jam IS the process. Just like Flask IS the process.

Both require redis server running, plus rq, as in that link.

Here, we try to pass it to rq. But, its failing to accept the queue with below error. Jam has no error. Rq has it

Try it. It is simple.

On Mon, 5 Jul 2021, 21:24 Maxwell Morais, <max.mor...@gmail.com> wrote:

Drazen, python rq, will require a dedicated process aside of jam-py, to process.

Like:

python server.py

python tasks.py

Em seg., 5 de jul. de 2021 às 05:36, Drazen D. Babic <bab...@gmail.com> escreveu:

Hi, 

Map a button on some form like this:

    item.add_button(item.view_form.find(".form-footer"), 'Load CPU').click(function() {

        item.server('bck');

    });

Paste above code into form Server Module. Do the pip part. Click the button

load_all_cores will not run due to:

AttributeError: 'Item' object has no attribute 'lock'

Tested with v112 and v6...

Of course, this will not solve the issue with running jobs, even if the button works. It is just the matter of time when someone will need this imo.

Good reading:

https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-

[Maxwell Morais]

Dražen, I do use daily ERPNext that is built on Frappeframework.com, internally they use RQ.

And if you check the documentation of RQ they say that:

A worker is a Python process that typically runs in the background and exists solely as a work horse to perform lengthy or blocking tasks that you don’t want to perform inside web processes.

https://python-rq.org/docs/workers/

Also on the main page

To view this discussion on the web visit https://groups.google.com/d/msgid/jam-py/CAMvz_HqvbcoL-tfiqx5XWB_1qpjooty%2BiTWiuuh-_rEqwE5M1A%40mail.gmail.com.

[Drazen D. Babic]

hi Max, 

but that is not the problem :)

rq IS running and this error IS from the rq:

.

.

    raise ValueError('Invalid attribute name: %s' % name)

ValueError: Invalid attribute name: als.journals.email_aliases.example

.

.

What you see here is from Jam: job = queue.enqueue(example, 23)

So the example is a function but that function in RQ ends like als.journals.email_aliases.example

and is clearly saying Invalid attribute name!

I tried calling it with  job = queue.enqueue('example', 23) or  queue.enqueue('task.example'....or....

Tried everything.

D.

[Drazen D. Babic]

Here is a vid, redis with rq with Jam...
https://drive.google.com/file/d/1vs9KLMpbgtPfMSBk-xloBcuniG0Uje2m/view?usp=sharing

[Drazen D. Babic]

Got it to work in Task/Server module and on Button! The error was due to expectation that we can put the Python function in the Server Module. Not sure about that.

To try, follow Miguel general instructions and:

In App folder create file tasks.py (or any_python_file1.py, any_python_file2.... ):

import time

def example(seconds):

    print('Starting task #1')

    for i in range(seconds):

        print(i)

        time.sleep(1)

    print('Task completed')

in Server Module create (see the "import tasks, any_python_file" ? No need for it if created in App folder) :

from redis import Redis

import rq

import tasks, any_python_file

def on_created(task):

    print('Redis started')

    queue = rq.Queue('microblog-tasks', connection=Redis.from_url('redis://127.0.0.1:6379'))

    job = queue.enqueue('tasks.example', 23)

This is all. Instead of tasks.example it can be any_python_file.any_function

or it can be an variable we pass the arguments to from a button.

When the button is clicked, the function executes on the Server as rq job.

Please see the vid:

https://drive.google.com/file/d/1GA_Tl39P5BClX4EHfWS-BZM7uU0auWV9/view?usp=sharing

This concludes running background jobs for me. 

Thanks

[Maxwell Morais]

Cool!

--

You received this message because you are subscribed to the Google Groups "Jam.py Users Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jam-py+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jam-py/4ce617a3-b3cf-48ec-91ba-bf30a1480c2bn%40googlegroups.com.