Bundling dependencies with poetry (or other tools) in an air-locked system

[alonn]

I've recently bumped into this limitation, Which I can't figure a way around, and not sure there is one

building a wheel with poetry, but when trying to include with it different dependencies it seems not to bundle them together but to try to download them when installing the wheel in the target cluster. This is now a problem for me

1. When pushing wheels into an "internet air-locked" clusters

2.  When I have private libraries I depend on,  that have keys in the CI/CD pipeline but can't be installed in the actual deployment target without those keys (which I can't provide there due to the deployment model of this specific environment)

Is there a solution? solution without poetry? no solution at all? I'll be glad for your insights

in the JVM I would use UberJars to solve this but can't find a python equivalent which isn't a whole different tools such as pex (which isn't support in my deployment target - databricks)

[Shai Berger]

No experience with this myself, but I think it was presented in the
group at some point, or maybe on some podcast... anyway, Israeli
project.

https://pypi.org/project/wagon/0.1.2/

[Avraham Serour]

to save everyone two clicks:

" Creates Wheel based archives to allow portable offline installation of Python packages and their dependencies"

sounds like shiv:

https://shiv.readthedocs.io/en/latest/

any difference?

--
You received this message because you are subscribed to the Google Groups "PyWeb-IL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyweb-il+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyweb-il/20220119124733.5c7b4cd3.shai%40platonix.com.

[alonn]

Thanks @shai and @avrham.

Both projects look very interesting, from what I read now both demand a specialized tool to install , e.g I can't 
pip install some_wagon.tar.gz

or

pip install shiv.zip 

Without either some pre existing runtime

So this solves half the challenge: it seems like  good solution for air-locked environment you control and able to preinstall some management tool, but not a good solution for a "PAAS" like environment where you expected to provide a wheel and some automated process takes care of installation 

I hope I understand correctly, but of course, feel free to correct me if I'm wrong

[Meir Kriheli]

There was a project called wheelhouse, not maintained though :-(

https://github.com/level12/wheelhouse

--

You received this message because you are subscribed to the Google Groups "PyWeb-IL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyweb-il+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyweb-il/6debe1ca-58bd-41d7-bc95-17da86fdbd7an%40googlegroups.com.

--

Meir Kriheli
http://meirkriheli.com

[Tal Einat]

 I don't use Poetry, but I do something a bit similar with multi-stage docker builds, here's an overview in case it could be helpful:

1. In development, pin all dependencies and sub-dependencies (I use pip-compile from pip-tools).

2. Save wheels for all dependencies using the `pip wheel` command, e.g. `pip wheel --no-deps --wheel-dir /wheels -r /requirements.txt`.

3. Bundle these wheels, and install them later via e.g. `pip install /wheels/*`.

Seems to me that the last step should work just fine with no network access. After that you can install your own wheel in the same env and all dependencies should be there.

- Tal Einat

On Wed, Jan 19, 2022 at 11:36 AM alonn <alon...@gmail.com> wrote:

--

[Avraham Serour]

I used shiv on a kinda airlocked server.

You only need to preinstall python, the shiv package then is run like an "executable":

> python shiv_package.shiv

But that solves only one part of the problem, you still need to package python and install it.

For my case it was good enough because it means the application updates are smaller (and more frequent than the python installation), it does not need to package python.

You can take a look at pyinstaller, it does package python with your application in an executable.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyweb-il/CALWZvp4FOi7qGmQ6Vq31QXmQ-zAfxEO2D27pQ98i%3Dnwn4b6sKg%40mail.gmail.com.

[alonn]

Thanks everyone for the ideas. While non of those cover my exact use case, I'm sure it would be helpful in other situations (Some of those I've already bumbed into) or be part of the solution I need

I've decided to write a blog post about this, so Tal, Shai, Avraham, Meir please ping me with the relevant medium (or other public page) username so I can give credits :)  

[Avraham Serour]

I remembered now that I never replied, I suggest first of all giving credit to the pyweb group, linking to the meetup page maybe and inviting people to join.

I think the best is linking to my github profile (https://github.com/tovmeod), I don't take enough pictures of my lunch to have an instagram account.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyweb-il/ddb35268-546d-4283-8955-c39f0e8efb53n%40googlegroups.com.

[alonn]

Follow up: The blog post version

https://itnext.io/things-i-wish-python-packaging-learned-from-the-jvm-jars-aed19133c121