Performance improvement suggestion

[Tom Clerckx]

In my project I'm dealing with quite a lot (+15000) of files that are uploaded.

During testing and development, I run with the file-watch option enabled so that code is being reloaded when I make changes.

I noticed that due to the amount of files being added to my app folder, the filewatch consumes quite a lot of CPU cycles (CPU usage goes to >90%)

By replacing the python watchgod library by its succesor watchfiles, the CPU dropped back down to only a few percent!

Comparing two py4web runs using the linux time command: one with watchfiles, the other with watchgod, it is clear that the CPU time used by watchfiles is only a fraction of the cpu time used by watchgod

With respect to py4web itself, it only requires changing watchgod to watchfiles in the import:

diff --git a/py4web/core.py b/py4web/core.py

index 4713f460..5a0ac430 100644

--- a/py4web/core.py

+++ b/py4web/core.py

@@ -38,7 +38,7 @@ from collections import OrderedDict

from contextlib import redirect_stderr, redirect_stdout

import portalocker

-from watchgod import awatch

+from watchfiles import awatch

# Optional web servers for speed

try:

In addition watchfiles also supports things such as putting a filter on the filewatches, so you can easily exclude specific directories or filenames with specific extensions from the watch. 

[Massimo]

Hello Tom,

I had to revert 0025dde9892d386f86ae75e8f8fe0309a9bf1429 since it broke CI.

https://github.com/web2py/py4web/actions/workflows/run_test.yaml

I will investigate.

Massimo

[Tom Clerckx]

Hi Massimo, 

I see you reverted 0025dde9892d386f86ae75e8f8fe0309a9bf1429, but the log of the CI tests actually indicated revision c8815289addff2ab6d11c638935b078632b194fb as broken.

If I make a change, how can I run the tests on my side?

Tom.

[Massimo DiPierro]

If you make a PR, it should run tests for you. You can also try run "make test || echo broken"

--
You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/py4web/54b65e16-ae54-427d-af11-73df60779bcan%40googlegroups.com.

[Tom Clerckx]

So the failing test is:

   testargs = ["run", "-d", "demo", dir]
    res = runner.invoke(cli, testargs)

In test_main.py

I think the filewatcher in core.py, which is running as a daemonized thread, is stopped in an ungraceful manner, which may cause watchfiles to 'crash' with a segmentation fault.

Tom.