Architectures other than x64?

[Vadim]

I'm trying to gauge how hard would to use PyOxidizer for arm and aarch64 Linux targets.  Has anyone here tried anything like that?

[Gregory Szorc]

This is theoretically possible. The major blocker is producing Python distributions that PyOxidizer can consume. If someone were to produce such a distribution, it may _just work_. The best place to track this is at https://github.com/indygreg/python-build-standalone.

PyOxidizer may also need to be taught to better handle cross compiling. It is theoretically possible. It just hasn't been a priority for me yet, as I'm focusing on the major PC architectures first since they are the most popular runtime targets for Python. I would love to eventually support ARM and other architectures. And with macOS going to ARM, I will definitely need to support this in the future! If one cross-compiled architecture domino falls, I anticipate adding others will be relatively easy. Although I anticipate Linux+macOS and Windows may require substantially different solutions. Time will tell.

On Wed, Jul 29, 2020 at 10:06 PM Vadim <vad...@gmail.com> wrote:

I'm trying to gauge how hard would to use PyOxidizer for arm and aarch64 Linux targets.  Has anyone here tried anything like that?

--
You received this message because you are subscribed to the Google Groups "pyoxidizer-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyoxidizer-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyoxidizer-users/f588be43-fb41-453f-a7b8-021e0d522b9cn%40googlegroups.com.

[Gregory Szorc]

On Thu, Sep 17, 2020 at 2:36 PM Vadim Chugunov <vad...@gmail.com> wrote:

Hello Gregory,

On Wed, Sep 16, 2020 at 7:02 PM Gregory Szorc <gregor...@gmail.com> wrote:

This is theoretically possible. The major blocker is producing Python distributions that PyOxidizer can consume. If someone were to produce such a distribution, it may _just work_. The best place to track this is at https://github.com/indygreg/python-build-standalone.

Yes, that was my conclusion too, so I want to give it a try.  Before jumping into this, though, I wanted to pick your thoughts about how to best fit such a change into your project.   If you were to add other arches, where would you start?

I see two ways to go about this: 1) cross-building, 2) native builds on a qemu-emulated OS or, possibly, in qemu-container.   The latter will probably be too slow, so I'm thinking it makes more sense to concentrate on cross-building.    In connection to that, I have these questions:

- Cross-building Python requires building it first for the host architecture, and only then for the cross target.    Not sure where this logic should sit in your scripts.

- Looks like you are bootstrapping your own compiler toolchain, which also needs to be modified to cross-build.   Is there an easy way to use distro-provided toolchains?   I understand, the end result may not be as well-optimized, but it would be much easier for me to get started if I didn't need to figure out how to build cross-toolchains first.

In theory, the existing `cpython-unix` build system could be extended to support cross compilation. And there is some of that for targeting musl Linux. But support is a far cry from what would be eventually required.

For an initial implementation, I'm not too concerned about getting the end-to-end build, LLVM+Clang and all. If you want to take some shortcuts and use an existing Docker image providing a cross-compiler and get a hacked up version of python-build-standalone working in there, that's probably fine: I'll consider it my job to figure out how to plumb everything up end-to-end so we get the deterministic and reproducible builds that python-build-standalone otherwise produces. (The reason I build everything from source is because the Docker image is using an ancient version of Debian for libc symbol version compatibility in built binaries and I want to use a modern LLVM/Clang to get modern compiler features/performance. I can probably modify the build mode to add support for additional arches to facilitate experimentation...)

I would strongly support building the cross compiled target on an x86 host architecture rather than involving a VM or qemu. Since cpython-unix builds everything from source, in theory it is a matter of hooking up the proper flags to `configure` scripts. The double Python build could be a bit annoying. But it should be achievable. The make targets already have the target type in the name. So it should be possible to build multiple targets in a single invocation. We'd likely need to teach the build-cpython.sh script and Python code in build.py about how to plug in a host native Python. But this doesn't seem too bad.

I'm currently on a private email thread with developers of Mu and Beeware. We're attempting to schedule a meeting to discuss our technical solutions to building distributable Python builds so we can learn about others' solutions and potentially consolidate solutions. I'm not sure about Mu. But I know Beeware got Python building for iOS and Android. The code lives at https://github.com/beeware/Python-Apple-support and https://github.com/beeware/Python-Android-support if you are looking for inspiration.