"in-memory-only resources policy active but in-memory extension module importing not supported by this configuration": unable to build single-file executable with code that uses paramiko

[Nathaniel Manista]

Toolchain Enthusiasts-

    I'm currently blocked in an attempt to use PyOxidizer to build a single-file executable with the failure "in-memory-only resources policy active but in-memory extension module importing not supported by this configuration". I've attached a distillation of my application as an sdist that demonstrates the problem; if it's possible today to use PyOxidizer to build a single-file executable from this sdist I'd be delighted to be instructed how.

    From some poking and experimenting that I've done it smells like the root cause may be cryptography's dependence upon cffi (at least one of my dependencies depends upon cryptography). A few open issues in the issue tracker look possibly related (170, 261, 273) but I'm not certain. Also a comment in pull 199 mentions "almost finished getting cryptography working on all platforms", but pull 199 seems to have stalled and... it's not clear that the comment is talking about the fixes for cryptography happening in pull 199 anyway.

    Is it expected that PyOxidizer ought be able to build a single-file executable from my code today? If not is there already an issue tracking the problem(s) with doing so?

    Thanks very much,

-Nathaniel

STEPS TO REPRODUCE:

1) (put the two attached files in the same directory)

2) $ pyoxidizer build

EXPECTED RESULTS:

An executable file will be built and I can pass this one file around to any other user of my platform and it will be runnable for them on their hosts.

OBSERVED RESULTS:

$  pyoxidizer build

resolving 1 targets
resolving target exe
resolving Python distribution Url { url: "https://github.com/indygreg/python-build-standalone/releases/download/20200408/cpython-3.7.7-macos-20200409T0412.tar.zst", sha256: "f312bea46a7d8efecd4df6b22c03f83016775e6bb5944a5701d697e0a52c63b2" }
downloading https://github.com/indygreg/python-build-standalone/releases/download/20200408/cpython-3.7.7-macos-20200409T0412.tar.zst
Python distribution available at /Users/nathaniel/temp-representative/pyoxidizer/./build/python_distributions/cpython-3.7.7-macos-20200409T0412.tar.zst
reading data from Python distribution...
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
installing modified distutils to /var/folders/sf/xg252rpj61j6h32nkg2v9xyh002225/T/pyoxidizer-pip-install.GFUTpaDQPWl0/packages
modifying distutils/_msvccompiler.py for oxidation
modifying distutils/command/build_ext.py for oxidation
modifying distutils/unixccompiler.py for oxidation
pip installing to /var/folders/sf/xg252rpj61j6h32nkg2v9xyh002225/T/pyoxidizer-pip-install.GFUTpaDQPWl0/install
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Processing ./representative-0.0.20200813144015583612.tar.gz
Collecting importlib_resources>=3.0.0
  Using cached importlib_resources-3.0.0-py2.py3-none-any.whl (23 kB)
Collecting paramiko>=2.7.1
  Using cached paramiko-2.7.1-py2.py3-none-any.whl (206 kB)
Collecting requests>=2.24.0
  Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting zeroconf>=0.27.1
  Using cached zeroconf-0.28.0-py3-none-any.whl (50 kB)
Collecting zipp>=0.4; python_version < "3.8"
  Using cached zipp-3.1.0-py3-none-any.whl (4.9 kB)
Collecting cryptography>=2.5
  Using cached cryptography-3.0-cp35-abi3-macosx_10_10_x86_64.whl (1.8 MB)
Collecting pynacl>=1.0.1
  Using cached PyNaCl-1.4.0-cp35-abi3-macosx_10_10_x86_64.whl (380 kB)
Collecting bcrypt>=3.1.3
  Using cached bcrypt-3.1.7-cp34-abi3-macosx_10_6_intel.whl (53 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting ifaddr>=0.1.7
  Using cached ifaddr-0.1.7-py2.py3-none-any.whl (10 kB)
Collecting six>=1.4.1
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting cffi!=1.11.3,>=1.8
  Using cached cffi-1.14.1-cp37-cp37m-macosx_10_9_x86_64.whl (175 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Installing collected packages: zipp, importlib-resources, six, pycparser, cffi, cryptography, pynacl, bcrypt, paramiko, certifi, idna, chardet, urllib3, requests, ifaddr, zeroconf, representative
    Running setup.py install for representative: started
    Running setup.py install for representative: finished with status 'done'
Successfully installed bcrypt-3.1.7 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 cryptography-3.0 idna-2.10 ifaddr-0.1.7 importlib-resources-3.0.0 paramiko-2.7.1 pycparser-2.20 pynacl-1.4.0 representative-0.0.20200813170854227476 requests-2.24.0 six-1.15.0 urllib3-1.25.10 zeroconf-0.28.0 zipp-3.1.0
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
error[PYOXIDIZER_BUILD]: in-memory-only resources policy active but in-memory extension module importing not supported by this configuration
  --> ./pyoxidizer.bzl:18:5
   |
18 | /     exe.add_python_resources(
19 | |         dist.pip_install(["representative-0.0.20200813144015583612.tar.gz"]))
   | |_____________________________________________________________________________^ add_extension_module


error: in-memory-only resources policy active but in-memory extension module importing not supported by this configuration

ENVIRONMENT:

pyoxidizer 0.7.0

my employer's flavor of macOS 10.15.6

ADDITIONAL INFORMATION:

The code runs and behaves as expected with:

pip install representative-0.0.20200813144015583612.tar.gz

python -m representative

[Nathaniel Manista]

On Thu, Aug 13, 2020 at 7:44 PM Nathaniel Manista <nath...@google.com> wrote:

Is it expected that PyOxidizer ought be able to build a single-file executable from my code today? If not is there already an issue tracking the problem(s) with doing so?

Bump on these questions - would it be appropriate to migrate this discussion thread into a filed issue in the issue tracker?

Thanks much,

-Nathaniel

[Martin von Zweigbergk]

I ran into the error many months ago, but I didn't reply earlier because I couldn't remember how I solved the problem anyway :( I think it's appropriate to report it in the issue tracker.

 

Thanks much,

-Nathaniel

[Gregory Szorc]

Loading extension modules from memory is only supported on Windows. And the default configuration and the way the resource location "policy" is defined in the configuration language means people on Linux and macOS will see this error more than they should be.

If you are requesting in-memory loading, PyOxidizer tries to compile the extension from source in such a way that the extension can be linked directly into the binary and not distributed as a shared library (.so file), like Python extensions are typically distributed. This approach is a bit fragile and is likely failing with your package. This is worthy of a PyOxidizer bug report if nothing else so the issue with that package can be tracked.

Unless you absolutely need a single file executable, your best bet right now is to use `resources_policy='prefer-in-memory-fallback-filesystem-relative:<prefix>'` when constructing the `PythonExecutable` Starlark type. I plan to improve the user ergonomics of this behavior in future PyOxidizer releases. (Handling extension modules is a really hard problem!)

--
You received this message because you are subscribed to the Google Groups "pyoxidizer-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyoxidizer-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyoxidizer-users/CAEOYnATuT9_XUAyLi924dcXy7%3DHa6rcFX_soD49vRvMuSdizHg%40mail.gmail.com.