Re: [cython-users] From a Python egg, is it possible to dynamically import a Cython Extension?

[Ralf Schmitt]

Marc <messag...@gmail.com> writes:

> I understand the Cython extensions are specific to the machine architecture
> they are compiled on, so my plan was compile the Cython extension for the
> different architectures, then have the Python project dynamically import
> the appropriate Cython extension based on the current architecture.

here's a how bbfreeze dynamically loads modules:
https://github.com/schmir/bbfreeze/blob/master/bbfreeze/freezer.py#L309

i.e. use imp.load_dynamic

[Stefan Behnel]

Marc, 08.01.2013 19:57:
> My objective is to protect a small piece of logic within a Python project,
> so it cannot be viewed / read.
> My current approach is to try and compile that logic in Cython, then
> reference the resulting extension from a Python project.

>
> I understand the Cython extensions are specific to the machine architecture
> they are compiled on, so my plan was compile the Cython extension for the
> different architectures, then have the Python project dynamically import
> the appropriate Cython extension based on the current architecture.
>

> To ease deployment, I'd like the Python project to be built into a egg
> containing the binary Cython extensions, so it can then run under different
> architectures and dynamically load the appropriate extensions.

Since C extensions (including Cython implemented ones) are normal shared
libraries, it depends on the system library loader if loading from anything
but a real file in the file system can work at all. In almost all cases, it
won't work, so you'd have to first extract your shared library to a
(temporary) file and then load that, with the risk that the runtime
environment might e.g. be configured to not allow loading code from the
temp directory or things like that (happens...).

Might be worth it if your egg contains a substantial number of foreign
architecture libraries and compresses them well. Otherwise, I'd avoid that
additional source of hassle.

Stefan

[Marc]

Thanks Ralf / Stefan,

Just using scripts (I.e. not compiled into an egg), I took your advice I tired to dynamically load the pyd file in my example project, using

sys.modules["foo"] = imp.load_dynamic(__name__, f)

Unfortunately I got a "ImportError: dynamic module does not define init function (initmyextension)"

I'll research this a little more...

[Ralf Schmitt]

Marc <messag...@gmail.com> writes:

> Thanks Ralf / Stefan,
>
> Just using scripts (I.e. not compiled into an egg), I took your advice I
> tired to dynamically load the pyd file in my example project, using
> sys.modules["foo"] = imp.load_dynamic(__name__, f)
>
> Unfortunately I got a "ImportError: dynamic module does not define init
> function (initmyextension)"

you can rename the file on disk to whatever you like, but you must pass
in the right name as first argument to imp.load_dynamic. passing in
__name__ therefore only makes sense if you're calling this from a short
loader script which has the same name as the module to be loaded.

for example you have a cython module foo.bar, normally located in foo/bar.so.
you can now rename foo/bar.so to foo/linux-bar.so and add foo/bar.py and
then use the above code from boo/bar.py to load the .so file.

the nice thing about replacing the .so with this short loader module is,
that you don't have to change other code. imports just work as they did
before.

--
cheers
ralf

[Marc]

Hi Ralf;

Sorry - yes I spotted that issue and corrected it. 

In my originally attached project, if I replace the following in the myproject.myextension.__init__.py

"

f = os.path.join(os.path.dirname(__file__), sys_name, "foo.pyd")

sys.modules["foo"] = imp.load_dynamic("foo", f)

from foo import MyClass

"

Unfortunately, even through this solution works when running from the scripts, when those scripts are compiled into an egg it fails (with the same error as I originally had "ImportError: No module named myextension") 

Thanks,

Marc

On Tuesday, January 8, 2013 6:57:47 PM UTC, Marc wrote:

Hi all;

My objective is to protect a small piece of logic within a Python project, so it cannot be viewed / read.

My current approach is to try and compile that logic in Cython, then reference the resulting extension from a Python project.

I understand the Cython extensions are specific to the machine architecture they are compiled on, so my plan was compile the Cython extension for the different architectures, then have the Python project dynamically import the appropriate Cython extension based on the current architecture.

To ease deployment, I'd like the Python project to be built into a egg containing the binary Cython extensions, so it can then run under different architectures and dynamically load the appropriate extensions.

Current scenario:

I have created an example Cython project, to build the Cython extension under difference architectures - please see attached "MyExtensions.zip"

I have created an example Python project, which houses the compiled extension binaries and dynamically references them. - please see attached "MyProject.zip"

When I run my MyProject as stand alone scripts, dynamically referencing the extensions - everything WORKS! Excellent ....

> cd myproject  

> python myproject\__main__.py  

However, once compiled into an egg I cannot dynamically import the Cython extension.

> python setup.py bdist_egg  

> python dist\myproject-1.0-py2.7.egg

"ImportError: No module named myextension" is given

(note - I use the __main__ trick to automatically execute the egg)

I'm not sure if I'm just approaching this the wrong way or if I'm missing something in the final egg build?

I'm assuming this must be possible, can anyone see where I'm going wrong??

Any help would be very welcome!

[Stefan Behnel]

Melville Ts, 25.02.2013 04:45:
> I suck at the issue for a few days. Now, I find the answer in the document
> of the zipimport module.
>
> It wrotes:
>
>> Any files may be present in the ZIP archive, but only files .py and
>> .py[co] are available for import. ZIP import of dynamic modules (.pyd, .so)
>> is disallowed.
>
> I'm not sure if setuptools(pkg_resources) has a solution, so put the pyd in
> filesystem is a easy way.

It has a "solution" in the sense that it unpacks the shared library into a
user-local directory before it loads it. So unpacking it right away during
the installation is a much better alternative than having each user get
their own redundant copy at import time.

The reason it can't work otherwise is that shared library loading is an OS
specific thing, and operating system shared library loaders know nothing
about zip files. There may be exceptions to that, but I've never come
across any.

Stefan

[Stefan Behnel]

Melville Ts, 25.02.2013 11:33:
> Stefan Behnel:

>> It has a "solution" in the sense that it unpacks the shared library into a
>> user-local directory before it loads it. So unpacking it right away during
>> the installation is a much better alternative than having each user get
>> their own redundant copy at import time.
>

> Yes, I got it worked just now. First, change the code (require setuptools
> installed):
> "
> # f = os.path.join(os.path.dirname(__file__), sys_name, "foo.pyd")

Note that using "__file__" in a source file makes setuptools set the
zip_safe flag to False, which prevents the egg installation (unless
explicitly requested).


> f = pkg_resources.resource_filename(__name__, "foo.pyd")

> sys.modules["foo"] = imp.load_dynamic("foo", f)
> from foo import MyClass
> "

> Then, rename the file extension from zip to egg (important, just rename
> it). And then it works. It's a small trick.

I'm not sure what you're trying to say here. If it's unpacked (by not
installing it as an egg in the first place), this will obviously work. And
if it's packaged in a zip file, then my guess is that the above call to
resource_filename() simply extracts the file if it's in a zip, so you end
up with the same duplication problem as with a normal egg install and a
simple import.

Could you print f in the code above, so see where the library ends up
getting imported from?

Stefan

[Stefan Behnel]

... and to the list ...

Melville Ts, 25.02.2013 13:53:

>> I'm not sure what you're trying to say here. If it's unpacked (by not
>> installing it as an egg in the first place), this will obviously work. And
>> if it's packaged in a zip file, then my guess is that the above call to
>> resource_filename() simply extracts the file if it's in a zip, so you end
>> up with the same duplication problem as with a normal egg install and a
>> simple import.
>>
>

> I think there is a problem with my english. So I modified the Marc's
> project to show how did I get it worked.
> First, make sure setuptools has been installed.
> Then, modify setup.py at line 10:
>
>> - packages = ['', 'myextension.windows', 'myextension.linux',
>> 'myextension.darwin'],
>
> + packages = ['', 'myextension', 'myextension.windows',
>> 'myextension.linux', 'myextension.darwin'],
>
> Last, modify myproject\myextension\windows\__init__.py:
>
>> + import sys
>
> + import imp
>
> + import pkg_resources
>
> + f = pkg_resources.resource_filename(__name__, "foo.pyd")
>
> + print f
>
> + sys.modules["foo"] = imp.load_dynamic("foo", f)
>
>
> Now, we start build the egg with command:
>
>> python setup.py bdist_egg
>>
> Execute the egg with command:
>
>> python dist\myproject-1.0-py2.6.egg
>
> The result of mine:
>
>> D:\Downloads\myproject>python dist\myproject-1.0-py2.6.egg
>
> C:\Users\layzer\AppData\Roaming\Python-Eggs\myproject-1.0-py2.6.egg-tmp\myextens
>
> ion\windows\foo.pyd
>
> Traceback (most recent call last):
>
> File "F:\Develop\Python26\lib\runpy.py", line 122, in _run_module_as_main
>
> "__main__", fname, loader, pkg_name)
>
> File "F:\Develop\Python26\lib\runpy.py", line 34, in _run_code
>
> exec code in run_globals
>
> File "build\bdist.win32\egg\__main__.py", line 1, in <module>
>
> File "build\bdist.win32\egg\myextension\__init__.py", line 8, in <module>
>
> File "build\bdist.win32\egg\myextension\windows\__init__.py", line 6, in
>> <modu
>
> le>
>
> ImportError: Module use of python27.dll conflicts with this version of
>> Python.
>
>
> As you see, the pkg_resources module unzip the egg file automaticly. An
> ImportError raise because of the mismatched python version, and you see it
> worked anyway.

I still don't get it. What is the advantage of going all the way through a
manual import and using pkg_resources, over just saying "import foo" after
a normal installation?

Stefan

[Stefan Behnel]

Melville Ts, 25.02.2013 14:53:

>> I still don't get it. What is the advantage of going all the way through a
>> manual import and using pkg_resources, over just saying "import foo" after
>> a normal installation?
>

> As I known, we cannot import binary module which achived in zip or egg
> file by just saying "import foo". It's disallow by python (see the
> documentation of zipimport module). So I manual import binary module by
> this trick. Did I make a mistake?

If the shared library is unpacked anyway, doing "import foo" will just
work. And unpacking it during the installation is the right thing to do.

Stefan