ENB: Distribution challenges
Edward K. Ream
This Engineering Notebook post briefly summarizes the challenges in distributing Leo on pypi.
PRs #3834 and #3835 (both drafts) contain the relevant code.
New security restrictions
All files uploaded to pypi and testpypi must have unique filenames and contents.
These new constraints are reasonable. Code repositories such as GitHub and pypi are under continuous, sustained, sophisticated attacks from state-sponsored terrorists. The goal is to insert malware in legitimate software such as Leo.
These new constraints require new scripts. A few hours of experience showed that new scripts are mandatory. I now have private scripts called build-leo, upload-leo, install-leo and uninstall-leo. These scripts will eventually become part of the leo/scripts directory.
Further work
- Automatically create unique filenames using public version ids. The build script might use a private helper file in my home directory.
- Automatically insert a timestamp in leoVersion.py. This timestamp will ensure that the contents of each build will be unique.
The build script (or a helper) will update these ids and timestamps without changing Leo's version number. PR3 uses a version number of 6.7.8.1, but that's likely a temporary expedient.
Bugs
I yanked Leo 6.7.8 from pypi because the Python wheel was dangerous: it sprayed the sub-folders of the leo-editor/leo folder inside Python's site-packages folder!! Happily, `pip uninstall leo` undid the damage!
I am cycling between two bugs. The first creates no "leo" package in the "site-packages" folder; the second damages "site-packages" as described above.
Presumably, pyproject.toml needs only a tweak, but I'm tearing out my hair trying to find it!
Summary
Right now, `pip install leo` installs Leo 6.7.7, not Leo 6.7.8. The bugs described above are release blockers.
It is no longer possible to create or debug pypi distributions without using helper scripts. Those scripts are my next project.
I would greatly appreciate help debugging either PR. My guess is that pyproject.toml needs only a tweak, but finding that tweak has not been easy!!!
Edward
P.S. Both PRs remove setup.py. That's not strictly necessary, but removing setup.py should simplify distribution in the long run.
EKR
Edward K. Ream
This Engineering Notebook post briefly summarizes the challenges in distributing Leo on pypi.
In this kind of challenging project, I look for small gains. Here are today's:
Test locally
Aha/Doh: test my wheel file without using testpypi:
python -m pip install c:\Repos\leo-editor\dist\leo-6.7.8.1-py3-none-any.whl
Use the wheel-inspect package
The following command shows that the 6.7.7 wheel includes a "leo" module:
python -m wheel_inspect dist\leo-6.7.7-3-py3-none-any.whl >c:\Users\Dev\wheel-inspect-6.7.7.txt
A similar command shows that the 6.7.8 wheel does not include the "leo" module. I suspected as much, but now I know for sure.
Summary
Inspecting a wheel shows whether installing from that wheel has any chance of working.
There is no point in uploading a wheel file until a local install works.
The next task: tell the .toml file to include the "leo" module :-)
Onward!
Edward
Edward K. Ream
> In this kind of challenging project, I look for small gains. Here are today's:
And one more. Now that I can test locally, there is no need for a timestamp. Indeed, the .toml file will be different, so the overall hash will be different too.
So the project is getting simpler.
Edward
Thomas Passin
This Engineering Notebook post briefly summarizes the challenges in distributing Leo on pypi.
I would greatly appreciate help debugging either PR. My guess is that pyproject.toml needs only a tweak, but finding that tweak has not been easy!!!
Edward K. Ream
On Monday, March 18, 2024 at 5:43:22 PM UTC-4 Edward K. Ream wrote:I would greatly appreciate help debugging either PR. My guess is that pyproject.toml needs only a tweak, but finding that tweak has not been easy!!!
packages = [
"leo",
If it would help, once everything is working locally for you on Windows I could try it out on several Linux VMs.
Edward K. Ream
This Engineering Notebook post briefly summarizes the challenges in distributing Leo on pypi.