dashboard patch - issue with large number of files

[Tom Clerckx]

For some projects I have directories with a very large number (order of 30000) of uploaded files.

This causes issues with the _dashboard application, as it loads the directory structure in the browser. Due to the large number of files, it becomes slow and unresponsive.

I made a patch for the _dashboard application that allows one to exclude files and directories from being loaded in the _dashboard.

Excluded files and directories can be specified in a file dashboard_skip.json in the top of the application directory.

Example of a dashboard_skip.json file:

{
    "dirs": [
        "uploads",
        "static/images"
    ],
    "files": [
        "*.md"
    ]
}

[Massimo DiPierro]

I see the problem. I would like to propose a slight different solution (and recycle 90% of your PR). I am going to assume there is a .p4w_ignore at the root of the app with the same syntax as .gitignore as opposed to a custom json format. If present those files will not be visualized. If you agree I can merge your PR and make the change

[Tom Clerckx]

Sounds good!

[Massimo DiPierro]

this is done. Sorry for the delay. Please help me check it.

[Tom Clerckx]

Works for me.

Small change (forgot a dot):

diff --git a/apps/_dashboard/__init__.py b/apps/_dashboard/__init__.py
index df1bf6f1..86becd67 100644
--- a/apps/_dashboard/__init__.py
+++ b/apps/_dashboard/__init__.py
@@ -343,7 +343,7 @@ if MODE in ("demo", "readonly", "full"):
                 name.startswith(".")
                 or name.startswith("#")
                 or name.endswith("~")
-                or name[-4:] in (".pyc", "pyo")
+                or name[-4:] in (".pyc", ".pyo")
                 or name == "__pycache__"
                 or root == "uploads"
             )

[Tom Clerckx]

Few more things:

root uses the full path, so excluding uploads would become slightly different

and: this will exclude all uploads directories

so... maybe uploads should not be in  this evaluation at all (the user should use .py4web_ignore to exclude it)

diff --git a/apps/_dashboard/__init__.py b/apps/_dashboard/__init__.py
index df1bf6f1..bee5786e 100644
--- a/apps/_dashboard/__init__.py
+++ b/apps/_dashboard/__init__.py
@@ -343,9 +343,9 @@ if MODE in ("demo", "readonly", "full"):

                 name.startswith(".")
                 or name.startswith("#")
                 or name.endswith("~")
-                or name[-4:] in (".pyc", "pyo")
+                or name[-4:] in (".pyc", ".pyo")
                 or name == "__pycache__"

-                or root == "uploads"
+                or os.path.basename(root) == "uploads"
             )
 
         if not os.path.exists(top) or not os.path.isdir(top):