Using Cython for packaging large python project

[Topi Wala]

Hi folks,

I have a fairly large python project with nested directories multiple levels deep with many python files. I have a requirement to now distribute this as a standalone binary, for which I'm using Cython.

I have converted all my python files to shared objects, except the entry-level file {the one I chose was the one specified by the entry_point keyword in my setup}, that I used the --embed option to generate main, and linked against ALL the other so files. GCC does produce a binary for me.

However, on running, I encounter the error:

pkg_resources.DistrbutionNotFound: [executable-name == 0.9.0]

I have this version number in my setup.py. I'd like to keep it for internal consumption [where we'll be using the python files]. 

How do I overcome this? Is there any compiler_directives that I could add to my main entry point python file which will satisfy the runtime?

Thanks,

Topi

[Topi Wala]

I removed the packaging [name/version], and the error went away. However, now I have one more. There are parts in the python code where the 

__file__ is being used.

so, code like:

curdir = os.path.dirname(os.path.realpath(__file__)), for which I get 

NameError: name '__file__' is not defined.

Is there any way around this?

Thanks,

Topi

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stefan Behnel]

Topi Wala schrieb am 01.05.2015 um 18:48:
> I removed the packaging [name/version], and the error went away. However,
> now I have one more. There are parts in the python code where the
>
> __file__ is being used.
>
> so, code like:
>
> curdir = os.path.dirname(os.path.realpath(__file__)), for which I get
>
> NameError: name '__file__' is not defined.
>
> Is there any way around this?

Well, there is not really a "__file__" in your case, is there? If you need
to load resources (and I guess that's the intention here), you'll have to
find a way to load them from the compiled shared library, or maybe embed
them into a code file somehow.

Stefan

[Antony Lee]

This (packaging an entire application as a binary using Cython) seems very interesting.  Any chance you could offer a short write-up of the process?

Thanks in advance,

Antony

[Mark Miller]

Just curious...for this particular application, have you looked into just using something like Pyinstaller with the --onfile flag?

 

I have had great success using it to create stand alone Python programs in Windows, OSX, and Linux. And those Python programs also just happened to rely on cython-created libraries that I developed.

 

-Mark

[Topi Wala]

Anthony, Sure. Once I get past my teething problems. 

Stefan, this kind of code would work in python. So, shouldn't it work once it's cythonified as well.

Try this simple one liner

$cat test.py

print __file__

$python test.py

test.py

$cython --embed -o test.c test.py

$gcc -0s -I /usr/include/python2.7 -o test test.c -lpython2.7 -lpthread -lm -lutil -ldl

$./test

Fails with NameError: name '__file__' is not defined.

Apparently this is a known issue. See issue here:

http://trac.cython.org/ticket/475

I'm wondering if there is a workaround to this.

--Topi

On Fri, May 1, 2015 at 2:08 PM, Antony Lee <anton...@berkeley.edu> wrote:

[Chris Barker]

On Fri, May 1, 2015 at 5:16 PM, Topi Wala <wala...@gmail.com> wrote:

Stefan, this kind of code would work in python. So, shouldn't it work once it's cythonified as well.

no -- it shouldn't. __file__ specifies which file the python code for a module is. Cython creates compiled modules, which are then loaded via the system linking system -- there is no "file" to be found here.

In general, using __file__ is a bit of a hack, specifically designed for working with a traditional pyton install it's not really a language feature per se.

so the short answer is -- find another way to do whatever it is you are doing with __file__.

(and in my mind setuptools pkg_resources is a bit of a hack as well, even for pur python systems, and doesn't work so well with PyInstaller, py2exe, etc...

If you are looking for a way to store non-code resources so you can find them, I like the approach of embedding them in python code -- i.e. as string literals of various sorts.

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Topi Wala]

I've been trying out different tricks, and I think I could do with some direction.

The problem statement is this:

I have a large (~150 python files) multi-package python tree with a directory structure where some of the files share the same name, i.e dirA/file.py & dirB/file.py that needs to get distributed. Ideally, I'd like it to be a big fat ELF binary with all the modules compiled in. 

I tried doing this with cython_freeze, but unfortunately, cython_freeze doesn't seem to work well with directory structures, and results in symbol conflicts due to init_module creation that completely does away with the directory names, and uses the python file names only. Since there are multiple cross-imports everywhere, this would result in a large scale change across multiple files that I'd like to avoid.

Few questions:

(a) What's the best tool to generate the main entry point [i.e. like cython --embed & cython_freeze]

(b) What's the fastest path to get to an ELF binary with minimal surprises? [I can do a bunch of shared objects, if that's the mostly used method]

Thanks

Topi

[Chris Barker]

I'd check out cx-freeze and PyInstaller, and see what works best for you.

I_think_ that neither of these will give you a single binary, but they do give you single "thing" your users can install that looks like a regular old app -- with no need for a python install, etc. And the both support Linux.

Personalty, I've used py2app and py2exe on Windows with good success -- never tried it for Linux.

-CHB

[Tejas Khairnar]

Hello Mark,

You said you can use Pyinstaller with --onefile flag to create an executable.

I am facing an issue with the same. When i run pyinstaller on RHEL it creates and ELF executable for me.

How do I create an executable on RHEL which can run on MAC OSX?

I would highly appreciate any help in this direction.

Thanks,

Tejas.

[Chris Barker]

On Sun, Sep 3, 2017 at 5:37 PM, Tejas Khairnar <tkha...@asu.edu> wrote:

You said you can use Pyinstaller with --onefile flag to create an executable.

I am facing an issue with the same. When i run pyinstaller on RHEL it creates and ELF executable for me.

How do I create an executable on RHEL which can run on MAC OSX?

you can't. You'll need to build the executable on a Mac. pyinstaller will work there, too.

It theory, cross-compilation should be possible, but it would take a lot of work to make it work -- getting the compiler configured correctly would be hard enough, but then you'd need to manipulate the libraries with tools that are only available on the mac..

And all this has nothing to do with Cython per-se. Try the pythonmac list for more help, if you need it.

-CHB

 

To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--