_dashboard loading apps from github?

[David Manns]

I was hoping the dashboard might be a really nice way of loading an individual app from Github.

In _dashboard 'installed apps' section the create or upload app button produces a form. 

I specify 'oxcam' as my app name,

click the radiobutton for 'clone from web or git' and  paste in the url for the .zip asset from the desired release on GitHub ("https://github.com/oxcamne/oxcam/archive/refs/tags/v1.1.0.zip"),

click the 'new app' radiobutton and then the Create button.

This created the apps/oxcam directory, but unzipped the zip file into a subdirectory apps/oxcam/oxcam-1.1.0.

What is intended behavior?

Thanks, David

[David Manns]

I observe that one can load the latest source of an app over the web from github by specifying the clone link e.g.: https://github.com/oxcamne/oxcam.git.

However it seems that all github zip files contain their contents within an outer labeling directory, such as oxcam-1.1.0 or oxcam-main and so _dashboard ends up extracting this containing folder to the app directory

[Massimo DiPierro]

We should threat this a s bug. Perhaps the unzip script should detect the location of __init__.py and locate the subfolder to expand. I can implement the fix.

[David Manns]

I think one might usefully recognize github zip files by virtue of the fact that they contain a single zipped directory.

I was thinking that one would 'expandall' into the target directory, and then on recognizing that zip file contains a single directory at the top level, move all contents up into the target directory. Leaving the now empty container would document what was installed and when.

By not deleting the target directory but making the moves replace existing files, the dashboard could be used both for initial install and for updating to new versions, e.g. preserving the database files etc. I'm not sure the New/Replace mode is really needed, or perhaps it should be New/Replace/Update?

[Massimo DiPierro]

I think I have a solution. Mind helping test it?

1) clone master

2) delete apps/_dashboard

3) py4web setup apps # to reinstall _dashboard

4) try it

--
You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/py4web/6d3dec1d-97c1-4c55-b0d5-dc79283bbc7fn%40googlegroups.com.

[David Manns]

With the new _dashboard:

1. The link on the github code page for my app is https://github.com/oxcamne/oxcam.git. _dashboard loads this just fine. However, if one downloads this to the local machine (as oxcam-main.zip) and attempts to load using upload in _dashboard, that doesn't work - it puts a container file in apps/oxcam.

2. If one goes to the releases section of the github page to load a specific tagged version or release, the link is like https://github.com/oxcamne/oxcam/archive/refs/tags/v1.1.0.zip. Loading this from the web (or downloading and uploading from the local machine) again puts a container file in apps/oxcam.

My objective is to have potential users use the released version using the labelled .zip releases.

David

[Massimo DiPierro]

Can you check you have this new code in your apps/_dashboard/__init__.py

# check for subfolders that contain __init__.py

roots = list(

set(

path[:-12]

for path in allfiles

if path.count("/") == 1

and path.endswith("/__init__.py")

)

)

# there can be only one

if len(roots) != 1:

abort(500)

# extract only the subfolder

with tempfile.TemporaryDirectory() as tmpdir:

zfile.extractall(tmpdir)

zfile.close()

shutil.copytree(

os.path.join(tmpdir, roots[0]),

target_dir,

dirs_exist_ok=True,

)

I tested it with  https://github.com/oxcamne/oxcam/archive/refs/tags/v1.1.0.zip. and it seems to work as you suggested.

[David Manns]

No, I didn't have the new _dashboard! I didn't start from scratch, did git pull, but I think perhaps 'make assets' was needed too? Anyway, I'll rebuild from scratch, starting with a clone, and test further in the morning.

[Massimo DiPierro]

Good question. It depends on how you install py4web. But yes, safe to do a make assets before a py4web setup apps.

[David Manns]

I created a new environment, starting with cloning py4web, opening it as a vscode workspace, creating a venv, running make assets and py4web setup apps.

As an aside I noticed that the launch.json distributed needs to be changed from type: "python" to type: "debugpy".

In the create or upload app I filled in the app name 'oxcam', the link to the latest release on github 'https://github.com/oxcamne/oxcam/archive/refs/tags/v1.1.0.zip', selected clone from web and new app and clicked create. I didn't get any message, the installed apps redisplayed with no addition.

With the debugger in note that we reached _dashboard/__init__.py line 515 "roots = list(... but did not reach the next statement at line 524?

I also tried to restore my existing environment. git pull confirms I have the latest repo. I checked out an earlier release and then the current master, so nothing should be missing or out of date in the workspace. Then ran make assets, deleted the _dashboard folder, ran py4web setup apps. It seems that this partially installs _dashboard: _dashboard.utils is missing?

[David Manns]

tried again with clean install. got this error logged:

Traceback (most recent call last): File "/Users/davidmanns/Library/CloudStorage/OneDrive-Personal/Desktop/py4web/py4web/core.py", line 1059, in wrapper ret = func(*func_args, **func_kwargs) File "/Users/davidmanns/Library/CloudStorage/OneDrive-Personal/Desktop/py4web/py4web/core.py", line 1044, in wrapper raise exception File "/Users/davidmanns/Library/CloudStorage/OneDrive-Personal/Desktop/py4web/py4web/core.py", line 1020, in wrapper context["output"] = func(*args, **kwargs) ~~~~^^^^^^^^^^^^^^^^^ File "/Users/davidmanns/Library/CloudStorage/OneDrive-Personal/Desktop/py4web/apps/_dashboard/__init__.py", line 516, in new_app set( ~~~^ path[:-12] ^^^^^^^^^^ ...<2 lines>... and path.endswith("/__init__.py") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/Users/davidmanns/Library/CloudStorage/OneDrive-Personal/Desktop/py4web/apps/_dashboard/__init__.py", line 519, in <genexpr> if path.count("/") == 1 ^^^^^^^^^^ AttributeError: 'ZipInfo' object has no attribute 'count'

id
6

[Massimo DiPierro]

Thanks this was very usefull and I believe the problem is now fixed.

[David Manns]

Tested the new version this morning, it works perfectly to install a new app from a github release such as https://github.com/oxcamne/oxcam/archive/refs/tags/v1.1.0.zip Thank you!

Can we add the ability to update an existing app to a new version using a github zip? Suggest adding an 'update' radiobutton - in that mode prepare_target_dir() would check that target_dir exists, but not delete anything. I believe that extractall will overwrite existing files so extract() should not need to be changed.

[Massimo DiPierro]

Thanks for testing. I can add that tomorrow.

[Massimo DiPierro]

I think I added this as you suggested. Can you please help me test it?

[David Manns]

I tested, and unfortunately modified files are NOT being overwritten, despite what the shutil.copytree() documentation says. I'm running on python 3.13.5. Very strange. Can't find any documented problem to explain this. I've been spinning my wheels!

[David Manns]

What's causing this is in this section of extract():

# extract only the subfolder

with tempfile.TemporaryDirectory() as tmpdir:

zfile.extractall(tmpdir)

zfile.close()

source_dir = tmpdir if roots is None else os.path.join(tmpdir, roots[0])

# make sure we do not override the databases and uploads folders:

for folder in ["databases", "uploads"]:

if os.path.exists(os.path.join(target_dir, folder)):

shutil.rmtree(os.path.join(source_dir, folder))

shutil.copytree(

source_dir,

target_dir,

dirs_exist_ok=True,

)

because we are doing a true update, the databases folder exists, so we get to line 479 to call rmtree. There is no databases folder in source_dir, causing an error and quick exit from extract(). changing to:

for folder in ["databases", "uploads"]:

if os.path.exists(os.path.join(source_dir, folder)):

shutil.rmtree(os.path.join(source_dir, folder))

fixes the problem.

I suppose one could check both source and target, but I don't think that's necessary.

With that problem resolves, seems to work fine!

thanks

[Massimo DiPierro]

Good catch. Will fix tonight.

To view this discussion visit https://groups.google.com/d/msgid/py4web/1e56e31f-4f6f-4e7d-892f-fb28304ccb93n%40googlegroups.com.