Compile whole python packages into single pyd-files

[Manuel Zabelt]

Hello,

I'd like to compile a set of python packages and modules in to a
single pyd file.
Has anyone done something like this?

Right now, I drilled down my package structure to multiple pyx files
and one master file that imports them all. To do this I had to remove
all interdependencies between my modules, because beeing pyx files
they can not import each other anymore. I put the code snippets that
contained the interdependencies directly into the masterfile.

The point is that all my pyx files do only contain pure python for now
- no Cythen magic in there. That's why, besides compiling these files
into a single pyd, I would really like to still be able to use them as
pure Python files.

Thanks for your help and suggestions.

Greetings
Mmanuel Zabelt

[Robert Bradshaw]

On Nov 13, 2009, at 12:57 AM, Manuel Zabelt wrote:

> Hello,
>
> I'd like to compile a set of python packages and modules in to a
> single pyd file. Has anyone done something like this?

This was brought up before in the context of numpy. You can include
everything
into one gigantic file, but usually there's no advantage.

> Right now, I drilled down my package structure to multiple pyx files
> and one master file that imports them all. To do this I had to remove
> all interdependencies between my modules, because beeing pyx files
> they can not import each other anymore. I put the code snippets that
> contained the interdependencies directly into the masterfile.
>
> The point is that all my pyx files do only contain pure python for now
> - no Cythen magic in there. That's why, besides compiling these files
> into a single pyd, I would really like to still be able to use them as
> pure Python files.
>
> Thanks for your help and suggestions.

I'm not understanding what the problem is--pyx files *can* import
stuff from each other. You shouldn't have to make a master file or
remove dependancies or anything--you should be able to keep things
modular and import things just as in Python.

- Robert

[Lisandro Dalcin]

On Fri, Nov 13, 2009 at 6:57 AM, Manuel Zabelt <gro...@zabelt.net> wrote:
>
> That's why, besides compiling these files
> into a single pyd, I would really like to still be able to use them as
> pure Python files.
>

Can you elaborate a bit more? I fail to see what do you want to do.
From a namespace point of view, a '.pyd' is like as a '.py', i.e. a
module, and you can arange many of them to form a package. But there
is no way (well, I could found a way) to *bundle* the contents of many
".py" files inside a single '.py' file and pretend the original files
are still separated, each one providing its own namespace... The same
applies to compiled '.pyd' files.

Perhaps a better question: Why do you want to have a *single* '.pyd'
file? What's your problem with having many of them?

--
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

[Christopher Barker]

Lisandro Dalcin wrote:
> But there
> is no way (well, I could found a way) to *bundle* the contents of many
> ".py" files inside a single '.py' file and pretend the original files
> are still separated, each one providing its own namespace.

I don't think there is, but you can put them all in a zip file.

> The same applies to compiled '.pyd' files.

I may be misunderstanding what the OP wants, but you CAN create single
.pyd (or .so, or whatever) compiled extension that is a package, with
multiple modules in it. I've done this with hand coded C -- let me know
if you want an example, I could probably dig it up.

no idea how to do it with Cython, but it shouldn't be that hard to
support, just code to write!

I also don't know how important it is, but I found it made things easy
for compiling and distribution (this was before setuptools, however).

-Chris








--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Lisandro Dalcin]

On Fri, Nov 13, 2009 at 6:19 PM, Christopher Barker
<Chris....@noaa.gov> wrote:
> Lisandro Dalcin wrote:
>>
>

AFAIK, using '.pyd' (or '.so' in POSIX) ext modules inside a zip file
requires some special support from setuptools... it is not a native
feature of zipimport. Basically, IIRC, all what is needed is to unpack
the DLL/DSO in a tmp file and next load it.

>> But there
>> is no way (well, I could found a way) to *bundle* the contents of many
>> ".py" files inside a single '.py' file and pretend the original files
>> are still separated, each one providing its own namespace.
>
> I don't think there is, but you can put them all in a zip file.
>
>
>> The same applies to compiled '.pyd' files.
>
> I may be misunderstanding what the OP wants, but you CAN create single .pyd
> (or .so, or whatever) compiled extension that is a package, with multiple
> modules in it.

Yes, but this is not "standard" practice...

>
> I've done this with hand coded C
>

Ah! Of course, I can imagine what are you talking about...

> -- let me know if you want
> an example, I could probably dig it up.
>
> no idea how to do it with Cython, but it shouldn't be that hard to support,
> just code to write!

OK. I'm definitely interested in seeing your example... Could you post a link?

>
> I also don't know how important it is, but I found it made things easy for
> compiling and distribution (this was before setuptools, however).
>

I still fail to see the convenience... Could you elaborate a bit more?

[Christopher Barker]

Lisandro Dalcin wrote:
> AFAIK, using '.pyd' (or '.so' in POSIX) ext modules inside a zip file
> requires some special support from setuptools...

probably -- I was referring to a pure python option.

>> I may be misunderstanding what the OP wants, but you CAN create single .pyd
>> (or .so, or whatever) compiled extension that is a package, with multiple
>> modules in it.
>
> Yes, but this is not "standard" practice...

I know I saw it somewhere before I did it, but I don't think it's common.

> OK. I'm definitely interested in seeing your example... Could you post a link?

It's not on the web anywhere, but I've enclosed the C file. It includes
other files, one for each module, and builds a package with those modules.

This is the key code for the package init, creating a package called
"PACKAGE", and putting module called "module1" in it:

void initPACKAGE(void)
{
PyObject* module;
PyObject* package = Py_InitModule("PACKAGE", noMethods);
if(!package) return;

/* add package attributes, if any */
/*PyModule_AddStringConstant(package, "foo", "bar");*/

/* add a module
module = PyImport_AddModule("PACKAGE.module1");
if(!module) return;
if(PyModule_AddObject(package, "module1", module))
return;
Py_INCREF(module);
initmodule1();

}

>> I also don't know how important it is, but I found it made things easy for
>> compiling and distribution (this was before setuptools, however).
>>
>
> I still fail to see the convenience... Could you elaborate a bit more?

Not really -- it seemed like a good idea at the time, but it's been five
or size years, and i dont' remember teh motivation. I think I found it
easier to distribute a single *.so and *.dll, but with a properly built
setup.y, I'm not sure why I thought it was hard? py2exe may have been a
small motivator -- it has to put dlls alingside, so it was nice to only
have one, but these days I usually use pyexe in the mode where it
doesn't bundle up all the python code anyway.

HTH,

[Stefan Behnel]

Christopher Barker, 13.11.2009 22:50:

> This is the key code for the package init, creating a package called
> "PACKAGE", and putting module called "module1" in it:
>
> void initPACKAGE(void)
> {
> PyObject* module;
> PyObject* package = Py_InitModule("PACKAGE", noMethods);
> if(!package) return;
>
> /* add package attributes, if any */
> /*PyModule_AddStringConstant(package, "foo", "bar");*/
>
> /* add a module
> module = PyImport_AddModule("PACKAGE.module1");
> if(!module) return;
> if(PyModule_AddObject(package, "module1", module))
> return;
> Py_INCREF(module);
> initmodule1();
> }

That looks a lot like what cython_freeze.py does. Shouldn't be that hard to
write a similar script (or extend the existing one) to support extension
'packages' instead of building an executable.

Although I expect this to be used much more rarely than the support for
executables.

Stefan

[Manuel Zabelt]

On 15 Nov., 10:11, Stefan Behnel <stefan...@behnel.de> wrote:
>
> That looks a lot like what cython_freeze.py does. Shouldn't be that hard to
> write a similar script (or extend the existing one) to support extension
> 'packages' instead of building an executable.
>
> Although I expect this to be used much more rarely than the support for
> executables.
>
> Stefan

Thanks alout. Your hint to cython_freeze is greate help for me.
Christopher Barker got me right in the first place and the code he
posted looks alot like, what I need.

For the moment I've got some minor problems getting the cython_freeze
example running under windows, but all in all it looks very
promissing.

Greetings
Manuel