[ANN] [Hack] Import binary extensions from zipfiles, windows only

[Thomas Heller]

Warning: experimental code!

Overview
========

zipextimporter.py contains the ZipExtImporter class which allows to
load Python binary extension modules contained in a zip.archive,
without unpacking them to the file system.

Call the zipextimporter.install() function to install the import hook,
add a zip-file containing .pyd or .dll extension modules to sys.path,
and import them.

It uses the _memimporter extension which uses code from Joachim
Bauch's MemoryModule library. This library emulates the win32 api
function LoadLibrary.

Sample usage
============

>>> import zipextimporter
>>> zipextimporter.install()
>>> import sys
>>> sys.path.append("lib.zip")
>>> import _socket
>>> _socket.__file__
'c:\\sf\\py2exe\\hacks\\memimp\\lib.zip\\_socket.pyd'
>>> _socket.__loader__
<ZipExtensionImporter at a74480>
>>>

Bugs
====

reload() on already imported extension modules does not work
correctly: It happily loads the extension a second time.

http://starship.python.net/crew/theller/moin.cgi/Hacks_2fZipExtImporter

Cheers,

Thomas

[Michel Claveau - abstraction méta-galactique non triviale en fuite perpétuelle.]

I like this kind of gadget.

[Delaney, Timothy C (Timothy)]

Thomas Heller wrote:

> zipextimporter.py contains the ZipExtImporter class which allows to
> load Python binary extension modules contained in a zip.archive,
> without unpacking them to the file system.

I take it this was what you were talking about the other day when you
mentioned single-file applications produced by py2exe ...

Tim Delaney

[Thomas Heller]

It's a step in this direction only.

Thomas

[PJDM]

I'm trying to make ZopeX3 start faster by zipping up the "zope"
directory. (Because this will be stored on a CD, having less data to
read will make it quicker to start.)

The standard python zipimporter won't work with the ZopeX3 .pyd files,
so zipextimporter came along at just the right time. However, it
doesn't seem to work for me (Python 2.3.4 (#53, May 25 2004, 21:17:02)
[MSC v.1200 32 bit (Intel)] on win32, Windows XP SP2).

Sample script:
----
import zipextimporter
zipextimporter.install()

import sys
print 'Hooks:\n', sys.path_hooks

sys.path.append(r'C:\tmp\zope.zip')
print 'Path:\n', sys.path

# Try an innocuous import first to make sure we have the path correct.
import zope.server
print 'Server file:\n', zope.server.__file__

# Now try a .pyd import.
import zope.thread._zope_thread
print 'Thread file:\n', zope.thread._zope_thread.__file__
----

Output:
----
Hooks:
[<class 'zipextimporter.ZipExtensionImporter'>, <type
'zipimport.zipimporter'>]
Path:
['C:\\tmp', 'C:\\WINDOWS\\system32\\python23.zip', 'C:\\tmp',
'C:\\opt\\Python23\\DLLs', 'C:\\opt\\Python23\\lib',
'C:\\opt\\Python23\\lib\\plat-win', 'C:\\opt\\Python23\\lib\\lib-tk',
'C:\\opt\\Python23', 'C:\\opt\\Python23\\lib\\site-packages',
'C:\\tmp\\zope.zip']
Server file:
C:\tmp\zope.zip\zope\server\__init__.py
Traceback (most recent call last):
File "zei.py", line 15, in ?
import zope.thread._zope_thread
ImportError: No module named _zope_thread
----

The zope.zip file contains zope.thread:
----
$ unzip -l zope.zip |grep zope/thread
0 11-08-04 11:00 zope/thread/
0 07-03-04 04:34 zope/thread/DEPENDENCIES.cfg
65 07-28-04 22:52 zope/thread/SETUP.cfg
994 07-03-04 04:34 zope/thread/tests.py
748 11-08-04 11:00 zope/thread/tests.pyc
748 11-08-04 11:00 zope/thread/tests.pyo
20480 11-07-04 00:54 zope/thread/_zope_thread.pyd
7838 08-17-04 17:20 zope/thread/__init__.py
7808 11-08-04 11:00 zope/thread/__init__.pyc
7808 11-08-04 11:00 zope/thread/__init__.pyo
----

What am I missing?

Thanks.

PJDM

[Thomas Heller]

"PJDM" <Peter...@ap.spherion.com> writes:

zipextimporter, as published, doesn't handle extensions in packages.
The patch I attached at the end, may fix this - I tested it with PIL.

It may work with zope, or may not - depends on how the extensions are
structured. If there are extensions/pyds that link to other
extensions/pyds, then it will most certainly not work.

If it doesn't work, you could still try the py2exe approach:
py2exe bundles the .py modules in a zip file, and also inserts loaders
for extensions into the zip file. The loaders load the .pyd extensions
from the file system.

Thomas

<patch>
Index: zipextimporter.py
===================================================================
RCS file: /cvsroot/py2exe/py2exe/hacks/memimp/zipextimporter.py,v
retrieving revision 1.2
diff -c -r1.2 zipextimporter.py
*** zipextimporter.py 16 Dec 2004 09:29:42 -0000 1.2
--- zipextimporter.py 4 Jan 2005 20:08:08 -0000
***************
*** 50,55 ****
--- 50,56 ----
if result:
return result

+ fullname = fullname.replace(".", "\\")
for s in self._suffixes:
if (fullname + s) in self._files:
return self
***************
*** 60,72 ****
return zipimport.zipimporter.load_module(self, fullname)
except zipimport.ZipImportError:
pass
for s in self._suffixes:
path = fullname + s
if path in self._files:
# XXX should check sys.modules first ? See PEP302 on reload
# XXX maybe in C code...
code = self.get_data(path)
! mod = _memimporter.import_module(code, "init" + fullname)
mod.__file__ = "%s\\%s" % (self.archive, path)
mod.__loader__ = self
if _memimporter.get_verbose_flag():
--- 61,75 ----
return zipimport.zipimporter.load_module(self, fullname)
except zipimport.ZipImportError:
pass
+ initname = fullname.split(".")[-1]
+ fullname = fullname.replace(".", "\\")
for s in self._suffixes:
path = fullname + s
if path in self._files:
# XXX should check sys.modules first ? See PEP302 on reload
# XXX maybe in C code...
code = self.get_data(path)
! mod = _memimporter.import_module(code, "init" + initname)
mod.__file__ = "%s\\%s" % (self.archive, path)
mod.__loader__ = self
if _memimporter.get_verbose_flag():
<patch/>

[PJDM]

Thomas Heller wrote:

> zipextimporter, as published, doesn't handle extensions in packages.
> The patch I attached at the end, may fix this - I tested it with PIL.

Thanks, that works fine.

However, ZopeX3 itself still won't run. As part of the startup, it
attempts to load some files relative to imported classes. Since the
path dives into a zip file, opening a file such as
file://///C|/tmp/zope.zip/zope/app/server/schema.xml doesn't work.
Patching things up to avoid this just leads deeper into the mire, so it
looks like this avenue is closed.

Things would probably be easier if Python had something like Java's
Class.getResourceAsStream(), where a resource is found by the class
loader. This means that is doesn't matter if the class came from a
directory, a JAR file, or a network: if the class loader could load the
class, it could also load the resource from the same place. The
__loader__ attribute from PEP 302 seems to be the equivalent, but the
standard importer doesn't provide it.

maynp@nessus:~> python
Python 2.3.4 (#1, Jul 31 2004, 11:22:19)
[GCC 3.3.1 (SuSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.__loader__

Traceback (most recent call last):

File "<stdin>", line 1, in ?
AttributeError: 'module' object has no attribute '__loader__'
>>>

Oh well.

PJDM