Linux Snapcraft and Packager integrations are broken

[Matt Wilkie]

The Leo Github Webhooks page has two failing integrations: snapcraft.io and packager.io.

From a quick inspection it looks like they're supposed to be for building complete Linux packages to install. Does anyone want to take these over or should we remove the hooks?

-matt

[Chris George]

I will take a run at snapcraft.

My distro has it installed by default.

Chris

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/eec27254-f164-45d6-a681-02a46a09351d%40googlegroups.com.

[Chris George]

From the snapcraft docs:

"Note that your Python project should be using setuptools and you should be able to run python setup.py bdist_wheel without errors. If either of these are not true, please consult the setuptools documentation."

I am at this point and the setup pukes so I assume there is some work still to do with setuptools and setup.py?

Chris

[Chris George]

I found the offending line in the setup.py file.

Before I remove it and commit the change (snapcraft uses a git checkout to build the snap), what does this line do?

cmdclass={'clean': janitor.CleanCommand} # clean more than setuptools, #1055

With it I get the following error:

python3 setup.py bdist_wheel
Creating entry_points for [OS name - system]: posix - Linux
Traceback (most recent call last):
  File "setup.py", line 171, in <module>
    cmdclass={'clean': janitor.CleanCommand} # clean more than setuptools, #1055
NameError: name 'janitor' is not defined

Without it I can build the wheel just fine.

Chris

[Chris George]

Ugh.

I got past that easily enough.

Now I am stuck.

My distribution had an older version of setuptools. My conda environment has a newer. Snapcraft (the snap builder tool) creates its own vm as part of the process but it uses the system setuptools to do so, ignoring the vm.

This causes all kind of hell.

I replaced the system setuptoolswith the same version in the vm (which is new enough to "just work" according to stackoverflow etc.

Except it doesn't.

More googling required.

Chris

[Chris George]

I had the brilliant idea to set up virtualbox and install a clean ubuntu install.

Then I installed pip3 via apt and used it to install everything else I needed, including the latest setuptools.

But when I went to run snapcraft it puked trying to create a vm. Apparently virtualbox doesn't support kvm inside a virtualbox.

Now I am eyeing up an old laptop for a clean install of ubuntu...

Chris

[Matt Wilkie]

On Wednesday, 31 July 2019 05:37:01 UTC-7, Chris George wrote:

I found the offending line in the setup.py file.

Before I remove it and commit the change (snapcraft uses a git checkout to build the snap), what does this line do?

cmdclass={'clean': janitor.CleanCommand} # clean more than setuptools, #1055

This is from setupext-janitor module, which extends `python setup.py clean` with `--all` so that it removes all the files that out of the box clean misses. It's a developer nicety not Leo essential. I'm attached to it because of the time I spent researching what to do, but if it blocks better linux packaging it could be dropped.

With it I get the following error:

python3 setup.py bdist_wheel
Creating entry_points for [OS name - system]: posix - Linux
Traceback (most recent call last):
  File "setup.py", line 171, in <module>
    cmdclass={'clean': janitor.CleanCommand} # clean more than setuptools, #1055
NameError: name 'janitor' is not defined

Without it I can build the wheel just fine.

 

The NameError would be fixed if setupext-janitor is installed (`pip install setupext-janitor`). Maybe all that's needed is to add this to build-backend to `pyproject.toml`? Pip uses the 'requires' section to id and install packages before it tries to parse and run setup.py. Maybe Snapcraft does the same.

-matt

[Matt Wilkie]

The Leo Github Webhooks page has two failing integrations: snapcraft.io and packager.io.

From a quick inspection it looks like they're supposed to be for building complete Linux packages to install. Does anyone want to take these over or should we remove the hooks?

It looks like that with the release of 6.0 in master Packager woke up. The integration hook now reports success in communicating but Packager itself failed the build. I haven't looked at the logs yet: https://packager.io/gh/leo-editor/leo-editor

-matt

[Edward K. Ream]

On Fri, Aug 2, 2019 at 1:46 PM Matt Wilkie <map...@gmail.com> wrote:

It looks like that with the release of 6.0 in master Packager woke up. The integration hook now reports success in communicating but Packager itself failed the build. I haven't looked at the logs yet: https://packager.io/gh/leo-editor/leo-editor

Thanks for looking into this.  BTW, I have no idea what you are talking about, hehe.

Edward

[Chris George]

That was me. I initiated it from the packager side.

I am still working on snap.

I was almost there but needed one file in leo-editor. Unfortunately setup.py, snap, git all have different packaging numbers.

Still at it.

Chris

--

You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAMF8tS24EwtcpvJHAY2M1x%3D1arWEa63DhSf1F0LtRr75L3UkzA%40mail.gmail.com.

[Chris George]

Ugh.

No really. This has been an adventure that I wouldn't wish on my worst enemy.

Snapcraft is, well, a nightmare. Each "compile" of a new snap takes 15 minutes or so. What should be simple is complicated, mainly because packaging python applications is a odg's breakfast.

I am on version 13 of a completed snap that uploads to snapcraft.io but this doesn't count the 50+ full compiles that puked because of this or that missing or wrong.

I am down to a cryptic "QT_5_PRIVATE_API not defined in file libQt5Gui.so.5 with link time reference" in the finished snap that is apparently happening because the Qt5 version in my default environment is different from the Qt5 version required by the libraries in the multipass VM (this is seriously cool software BTW) which is different from the Qt5 version required by Leo's setup.py, all of which matter.

venv, conda, VirtualBox (multipass uses KVM which won't work in a VM), pyenv etc do not work. It wants the versions it wants and so far no combination I can think of will provide that.

So here is the status update.

Today I will leverage the fresh install of my OS (took 15 minutes to reinstall completely after hosing my root partition by filling it completely with broken VM images), and remove the apt Qt version and replace it with  the sudo pip3 install version since --user didn't work any better than venv, conda, or pyenv (also very, very cool software).

I want to meet the guy who invented this programming language/universe and ask him WTF he was thinking when it came to actually preparing a python program for distribution.

Chris

[Edward K. Ream]

On Wed, Aug 7, 2019 at 7:24 AM Chris George <techn...@gmail.com> wrote:

Ugh.

No really. This has been an adventure that I wouldn't wish on my worst enemy.

Wow.  Thanks for this report, all the experiments, and the cartoon.  It takes a special kind of person to persist in the face of such obstacles.

Edward

[Chris George]

Finally.

Like Frankenstein's monster, it lives.

It ain't pretty, but it lives.

I will come back to this in a while. It still needs a .desktop file that lives in the right spot and there are some warts that need trimming. But it works.

Going forward it can be easily built (at least to this point) once the new Leo is available via pip. Eventually I will figure out how to get master running using the git hooks so it will update everytimethere is a new version by itself. I can also push snapshots of devel using a cron job. But that is for later. I am taking a break. My brain hurts.

https://snapcraft.io/leo-editor

The executable from the command prompt is leo-editor.

Chris

[Matt Wilkie]

I will come back to this in a while.

Oh the heroics!

I'm in the hole with you, perversely happy to have company, though I don't wish this on anybody. Enjoy your break properly and don't hurry back.

It still needs a .desktop file that lives in the right spot and there are some warts that need trimming. But it works.

See recent work with "Settings >> Desktop integration". There are buttons that should create and remove the link files on all platforms. No icon support yet; my next research project.

 

https://snapcraft.io/leo-editor

Nice presentation page.

-matt

[Matt Wilkie]

GitHub is advertising an upcoming continuous integration/delivery service called Actions: https://github.com/features/actions. I signed up for the beta, when it opens.

"Whether you want to build a container app, deploy a web service, publish packages to registries, or automate welcoming new users to your open source projects–there’s an action for that"

-matt

[Chris George]

A .deb package to try out.

wget -qO - https://fbs.sh/leo-editor/leo-editor/public-key.gpg | sudo apt-key add -
echo 'deb [arch=amd64] https://fbs.sh/leo-editor/leo-editor/deb stable main' \
  | sudo tee /etc/apt/sources.list.d/leo-editor.list
sudo apt-get update
sudo apt-get install leo-editor

This installs a tightly controlled environment.

Python 3.7.3, PyQt version 5.13.0, ubuntu linux

Chris

[Chris George]

Ooops. Python 3.7.4 not 3.7.3.

[Chris George]

This method (fman.io) lets people install it using a recognized system. It also allows people to upgrade using their familiar package manager.

The fman software also allows packaging for Windows and Mac from the same sources, basically one further step for each out of the twenty or so steps. The downside is that the installers for Windows and Mac must be created on those platforms, neither of which I have access to.

There is apparently a possibility using VirtualBox images that I am looking into next.

Chris

--

You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/ab777eb9-d122-40e6-94dc-eda398898dfb%40googlegroups.com.

[Matt Wilkie]

Interesting! Better link: https://build-system.fman.io/

I might be able to help with setting up a dedicated Windows machine for builds if vbox doesn't work.

-matt

[Matt Wilkie]

We might not be allowed to use FBS because of licensing: "To comply with its license, you need to open source your application itself under the GPL, or buy a commercial PyQt license". This is probably because they solve some of the packaging and distribution problems by bundling PyQt, which is GPL (or commercial). I've asked their licensing department to clarify if this restriction applies to X/MIT projects such as Leo.

-matt

[Chris George]

Good point.

Sigh. Back to snapcraft. :-(

Chris

On Thu, Aug 15, 2019 at 9:21 AM Matt Wilkie <map...@gmail.com> wrote:

We might not be allowed to use FBS because of licensing: "To comply with its license, you need to open source your application itself under the GPL, or buy a commercial PyQt license". This is probably because they solve some of the packaging and distribution problems by bundling PyQt, which is GPL (or commercial). I've asked their licensing department to clarify if this restriction applies to X/MIT projects such as Leo.

-matt

--

You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/10012251-eacc-4d7b-a8e4-7432fadfb028%40googlegroups.com.

[Chris George]

Last night was the second time that multipass hosed my filesystem, so I am real happy with PBS compared to snapcraft.

FBS has the added benefit of being (potentially) multiplatform.

I will push forward with it and give snapcraft a rest for now.

Chris

On Mon, Aug 19, 2019 at 12:06 PM Matt Wilkie <map...@gmail.com> wrote:

We might not be allowed to use FBS because of licensing: "To comply with its license, you need to open source your application itself under the GPL, or buy a commercial PyQt license". This is probably because they solve some of the packaging and distribution problems by bundling PyQt, which is GPL (or commercial). I've asked their licensing department to clarify if this restriction applies to X/MIT projects such as Leo.

FBS author Micheal Herrmann got back to me quickly and granted a specific exemption for Leo, so long as the text of that exemption was added to the LICENSE file. However it only applied to Leo specifically and not to forks. This would mean that other works from Joe Orr, Vitalijie etc. would have to ask for their own expemptions. Not very palatable in my opinion. However, in further discussion we arrived at a much better and cleaner solution: create a new repo under Leo Editor organization that is specific to FBS packaging and distribution. This repo would have GPL license, and thus not need an exemption. Anybody who wants to fork Leo proper can continue to do so as they've always done, ignoring the FBS specific repo.

So Chris yes you can freely explore FBS as a better packaging tool (if you still have appetite for any of this ;-).

I'll add the relevant portions of my discussion with Micheal as a later reply to this thread.

-matt

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/d3fe2de4-d0b6-4862-8c03-ee62671e7fe7%40googlegroups.com.