PyInstaller 1.5.1 on AIX

[mgd]

I am trying to use PyInstaller 1.5.1 on AIX 6.1 and have gotten some
of the way. Some input however would be appreciated.

I am using Python 2.6.2 and GCC 4.2.0.

I start out trying to build the bootloader:

--------------
bash-3.00$ python ./waf configure build install
AIX-32bit detected
Checking for program xlc_r or xlc : gcc
<..snip..>
Checking for flags -Wl,--as-needed : yes
'configure' finished successfully (2.711s)
Waf: Entering directory `/data/maconomy/mgd/pyinstaller-1.5.1/source/
build'
[13/16] cc_link: build/debug/common/launch_1.o build/debug/linux/
main_1.o build/debug/linux/getpath_1.o -> build/debug/run_d
ld: 0706-012 The -- flag is not recognized.
ld: 0706-012 The -a flag is not recognized.
ld: 0706-012 The -- flag is not recognized.
ld: 0706-027 The -n flag is ignored.
collect2: ld returned 255 exit status
Waf: Leaving directory `/data/maconomy/mgd/pyinstaller-1.5.1/source/
build'
Build failed: -> task failed (err #1):
{task: cc_link launch_1.o,main_1.o,getpath_1.o -> run_d}
bash-3.00$
--------------

It seems the problem is that the linker does not recognise the option
'--as-needed'.

A small change to source/wscript fixes this:

--------------
--- pyinstaller-1.5.1/source/wscript 2011-09-05 20:17:46.000000000
+0200
+++ pyinstaller-1.5.1.mgd/source/wscript 2011-09-05
20:19:04.000000000 +0200
@@ -187,10 +187,11 @@
conf.env.append_value('CCFLAGS', '/EHa')

# link only with needed libraries
- # XXX: -Wl,--as-needed detected during configure but fails
during mac build
+ # XXX: -Wl,--as-needed detected during configure but fails
during mac and aix builds
if conf.check_cc(ccflags='-Wl,--as-needed',
msg='Checking for flags -Wl,--as-needed') \
- and not myplatform.startswith('darwin'):
+ and not myplatform.startswith('darwin') \
+ and not myplatform.startswith('aix'):
conf.env.append_value('LINKFLAGS', '-Wl,--as-needed')
--------------

Now, the bootloader builds.

However, when running he Configure.py script i run into problems
detecting TCL/TK:

--------------
bash-3.00$ pyinstaller-1.5.1/Configure.py
I: computing EXE_dependencies
I: Finding TCL/TK...
I: Analyzing /usr/bin/python
Traceback (most recent call last):
File "pyinstaller-1.5.1/Configure.py", line 350, in <module>
main(opts.configfile)
File "pyinstaller-1.5.1/Configure.py", line 319, in main
test_TCL_TK(config)
File "pyinstaller-1.5.1/Configure.py", line 110, in test_TCL_TK
mo = re.match(pattern, nm)
UnboundLocalError: local variable 'pattern' referenced before
assignment
bash-3.00$
--------------

It seems the pattern is uninitialised unless the platform is 'win',
'linux' or 'darwin'. Adding a few lines to avoid TCL/TK detection in
other situations fixes this problem:

--------------
--- pyinstaller-1.5.1/Configure.py 2011-07-15 14:46:16.000000000
+0200
+++ pyinstaller-1.5.1.mgd/Configure.py 2011-09-05 20:32:05.000000000
+0200
@@ -91,6 +91,7 @@
saveexcludes = bindepend.excludes
bindepend.excludes = {}

+ pattern = None
if target_platform.startswith("win"):
pattern = r'(?i)tcl(\d\d)\.dll'
elif target_platform.startswith("linux"):
@@ -107,7 +108,9 @@
binaries.extend(bindepend.Dependencies(binaries))
binaries.extend(bindepend.Dependencies([('', sys.executable,
'')]))
for nm, fnm, typ in binaries:
- mo = re.match(pattern, nm)
+ mo = None
+ if pattern:
+ mo = re.match(pattern, nm)
if not mo:
continue
if not target_platform.startswith("darwin"):
--------------

Now, configure runs cleanly.

So now I try to build an executable from a plain and simple "Hello
World" script, that looks like this.

--------------
#!/usr/bin/python
print "Hello PyInstaller!"
--------------

To build it, I run:

--------------
bash-3.00$ pyinstaller-1.5.1/Makespec.py test.py
wrote /data/maconomy/mgd/test.spec
now run Build.py to build the executable
bash-3.00$ pyinstaller-1.5.1/Build.py test.spec
<..snip..>
--------------

Trying to run the executable results in:

--------------
bash-3.00$ cd dist/test/
bash-3.00$ ./test
Error loading Python lib './libpython2.6.so.1.0': 0509-022
Cannot load module ..
0509-026 System error: A file or directory in the path name
does not exist.
--------------

According to 'ldd' my executable does not depend on something called
'libpython'

--------------
bash-3.00$ ldd test
test needs:
/opt/freeware/lib/libz.a(libz.so.1)
/usr/lib/libdl.a(shr.o)
/usr/lib/libc.a(shr.o)
/usr/lib/librtl.a(shr.o)
/usr/lib/libc.a(shr_64.o)
/unix
/usr/lib/libcrypt.a(shr.o)
/usr/lib/libcrypt.a(shr_64.o)
bash-3.00$
--------------

so I guess it is something the generated binary loads dynamically.
However, there is no 'libpython<anything>.so' on my AIX box, only
something called libpython2.6.a and I am not sure whether this is a
static library or a shared library. But simply copying it to my
PyInstaller generated folder and renaming it to libpython2.6.so.1.0
does not help.

--------------
Error loading Python lib './libpython2.6.so.1.0': 0509-022
Cannot load module ./libpython2.6.so.1.0.
0509-103 The module has an invalid magic number.
--------------

Now, I am a bit stuck.

Any help or pointers are highly appreciated.

Best regards,
Martin Gamwell Dawids

P.S. I am aware of the open ticket requesting PyInstaller on AIX
http://www.pyinstaller.org/ticket/343

[Martin Zibricky]

mgd píše v Po 05. 09. 2011 v 05:49 -0700:

> Trying to run the executable results in:
>
> --------------
> bash-3.00$ cd dist/test/
> bash-3.00$ ./test
> Error loading Python lib './libpython2.6.so.1.0': 0509-022
> Cannot load module ..
> 0509-026 System error: A file or directory in the path name
> does not exist.
> --------------
>
> According to 'ldd' my executable does not depend on something called
> 'libpython'

The executable should not depend directly on libpython. (it is loaded
dinamically).

>
> so I guess it is something the generated binary loads dynamically.
> However, there is no 'libpython<anything>.so' on my AIX box, only
> something called libpython2.6.a and I am not sure whether this is a
> static library or a shared library. But simply copying it to my
> PyInstaller generated folder and renaming it to libpython2.6.so.1.0
> does not help.

libpython2.6.a is not a dynamic library and it can't be used with
pyinstaller.

>
> --------------
> Error loading Python lib './libpython2.6.so.1.0': 0509-022
> Cannot load module ./libpython2.6.so.1.0.
> 0509-103 The module has an invalid magic number.
> --------------
>
> Now, I am a bit stuck.
>
> Any help or pointers are highly appreciated.

PyInstaller is not made to work with statically linked python. The
bootloader is searching for any libpython*.so in your dist folder.

That's also the reason why pyinstaller does not work with activestate
python - activestate python is also statically linked.

I don't know much about C/C++ (bootloader code) but you would have to
add python interpreter to dist folder and modify bootloader to somehow
use python interpreter to run the python code with it.

Maybe you could try link the libpython.a statically when compiling the
bootloader.

[mgd]

Hi Martin,

Thank you for your help.

> Maybe you could try link the libpython.a statically when compiling the
> bootloader.

I made two changes:

1) In source/wscript there is some code that ensures that we do not
link with libpython:

# Do not link with libpython and libpthread.
for lib in conf.env.LIB_PYEMBED:
if lib.startswith('python') or lib.startswith('pthread'):
conf.env.LIB_PYEMBED.remove(lib)

By adding an extra test on the platform we ensure libpython is
statically linked on AIX:

--- pyinstaller-1.5.1.orig/source/wscript 2010-11-12
17:12:34.000000000 +0100
+++ pyinstaller-1.5.1/source/wscript 2011-09-07 17:19:57.000000000
+0200
@@ -93,9 +93,9 @@
conf.check_tool('python')
conf.check_python_headers()

- # Do not link with libpython and libpthread.
+ # Do not link with libpython (except on AIX) and libpthread.
for lib in conf.env.LIB_PYEMBED:
- if lib.startswith('python') or lib.startswith('pthread'):
+ if lib.startswith('pthread') or (lib.startswith('python')
and not myplatform.startswith('aix')):
conf.env.LIB_PYEMBED.remove(lib)

2) In the bootloader file source/common/launch.c the Python DLL/Shared
Lib is loaded dynamically in function loadPython(). By adding a simple
#elif we prevent loading the shared library on AIX:

--- pyinstaller-1.5.1.orig/source/common/launch.c 2011-07-05
00:17:42.000000000 +0200
+++ pyinstaller-1.5.1/source/common/launch.c 2011-09-07
17:37:13.000000000 +0200
@@ -495,7 +495,11 @@
}

mapNames(dll);
- #else
+
+ #elif defined(_AIX)
+ /* On AIX we link statically with libpython so don't try to load
any shared library. */
+
+ #else /* neither WIN32 nor AIX */

/* Determine the path */
#ifdef __APPLE__


Now, I can build the boot loader

bash-3.00$ ./waf distclean
'distclean' finished successfully (0.977s)
bash-3.00$ ./waf configure build install
AIX-32bit detected
<..snip..>
'install' finished successfully (0.191s)
bash-3.00$

I can build my executable:

bash-3.00$ rm -rf dist/ build/
bash-3.00$ pyinstaller-1.5.1/Configure.py
<..snip..>

bash-3.00$ pyinstaller-1.5.1/Makespec.py test.py
wrote /data/maconomy/mgd/test.spec
now run Build.py to build the executable
bash-3.00$ pyinstaller-1.5.1/Build.py test.spec
<..snip..>

The generated binary no longer fails (trying to load a non-existent
shared library). However, nothing happens apart from a non-zero exit
code:

bash-3.00$ dist/test/test
bash-3.00$ echo $?
255
bash-3.00$

Running the original Python script, however, works as expected.

bash-3.00$ ./test.py
Hello PyInstaller!
bash-3.00$


Any hints on where to look in the boot loader code will be highly
appreciated.

/Martin

[Martin Zibricky]

mgd píše v St 07. 09. 2011 v 02:17 -0700:

> Any hints on where to look in the boot loader code will be highly
> appreciated.

I'm by no means expert on bootloader code, but you should find place in
the code where the python code is actually executed. See how it works
and how it could be changed to run python code with statically linked
python.

[mgd]

Hi Martin,

> I'm by no means expert on bootloader code, but you should find place in
> the code where the python code is actually executed. See how it works
> and how it could be changed to run python code with statically linked
> python.

It took some debugging (past fork/exec calls) to find out why the
bootloader did not work. I found out that the child process crashed
with a segmentation fault when accessing undefined (NULL) Python
library symbols.

In the bootloader code, all Python symbols are referenced via PI_
prefixed variables which are pointers to the real Python library
symbols. E.g.
* 'PI_Py_NoSiteFlag' is a pointer to the Python library variable
'Py_NoSiteFlag'
* 'PI_Py_SetProgramName' is a pointer to the Python function
'Py_SetProgramName'

These symbols are declared, defined and bound like this using macros
defined in the file 'common/launch.h':
* The macros EXTDECLPROC and EXTDECLVAR are used to declare these
'PI_' symbols.
* The macros DECLPROC and DECLVAR are named a bit misleading but are
used to define the same symbols and assign them the value NULL.
* The macros GETPROCOPT, GETPROC and GETVAR are used to bind the
Python library symbols to the 'PI_' prefixed symbols.

On Windows and on Linux the binding of the symbols are done by loading
the Python library symbols dynamically in the code generated by the
GET* macros.

I have made some small changes to all the macros above in the AIX/
static-link case so the 'PI_' symbols are simply assigned the
addresses of the statically linked Python lib symbols.

And it seems to work. At least I can run a small test script that
prints to the console and writes a text file.

So far so good. :-)

Now, I have to do a little clean-up, look at TCL/TK detection (if it
makes sense on AIX) and run the tests.

/Martin

[Martin Zibricky]

mgd píše v Út 13. 09. 2011 v 05:24 -0700:

> So far so good. :-)

That sounds great.

Maybe your changes would make it possible to use ActiveState python with
pyinstaller. I think it is also statically linked.

[Martin Dawids]

> Now, I have to do a little clean-up, look at TCL/TK detection (if it
> makes sense on AIX) and run the tests.

I ran the tests like this:

bash-3.00$ buildtests/runtests.py

The results were:

{'failed': [],
'passed': ['test1',
'test2',
'test5',
'test6',
'test7',
'test8',
'test9',
'test10',
'test11',
'test12',
'test13',
'test14',
'test-email',
'test-encoders',
'test_f_option',
'test_filename',
'test-celementtree',
'test-nestedlaunch0',
'test-nestedlaunch1',
'test-relative-import',
'test-relative-import2',
'test-relative-import3',
'test_error_during_import',
'test_getfilesystemencoding'],
'skipped': ['test15',
'test-wx',
'test-numpy',
'test-pycrypto',
'test-zipimport1',
'test-zipimport2',
'test-ctypes-cdll-c',
'test-ctypes-cdll-c2']}

Nothing failed, so that's good.

I am not sure why some tests were skipped and whether that is expected or not?

/Martin

Please consider the environment before printing.
</pre>

<div style="FONT-FAMILY: arial;color:#525759;font-size:9pt; margin-top:15px">This e-mail and any attachments are intended only for the named recipient(s) and may contain information that is legally privileged, confidential, or exempt from disclosure under applicable law. This message may be logged for archival purposes, may be reviewed by parties at Deltek other than those named in the message header, and may not necessarily constitute an official representation of Deltek. If you have received this message in error, or are not the named recipient(s), you may not retain copy or use this e-mail or any attachment for any purpose or disclose all or any part of the contents to any other person. Any such dissemination, distribution or copying of this e-mail or its attachments is strictly prohibited. Please immediately notify the sender and permanently delete this e-mail and any attachment from your computer.</div>

[Martin Zibricky]

Martin Dawids píše v Út 13. 09. 2011 v 13:27 +0000:

>
> I am not sure why some tests were skipped and whether that is expected
> or not?

This is expected. For some tests they are needed some additional
modules. If these modules are not present then those tests are skipped.
Like: test-numpy - you need 'numpy' module.

Could you please attach your patches to the following ticket?

http://www.pyinstaller.org/ticket/343

If you are happy with your changes could you please try backport your
changes for the svn version?

Thanks in advance.

[Martin Gamwell Dawids]

Hi Martin

> Could you please attach your patches to the following ticket?
>
> http://www.pyinstaller.org/ticket/343

I am working on making the patch nice and clean.
This includes trying to add TCL/TK detection for AIX in
'Configure.py'. I am using the same pattern as for Linux:

+ elif target_platform.startswith("linux") or
target_platform.startswith("aix"):
+ pattern = r'libtcl(\d\.\d)?\.so'

However, I find nothing as the only binary analysed in' /usr/bin/
python'. However, on my AIX installation there is a TCL/TK library in
'/usr/lib/libtcl8.4.so'.

Any idea how to fix that?
(Is it because the installed Python was not built with TCL/TK
support?)

> If you are happy with your changes could you please try backport your
> changes for the svn version?

I will try that when the patch is nice.

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v St 14. 09. 2011 v 02:10 -0700:

>
> Any idea how to fix that?
> (Is it because the installed Python was not built with TCL/TK
> support?)

Test if tcl/tk is available in python. Run the following import in
python console:

>>> import Tkinter

If it does not work then python was not built with tcl/tk or this
support is not installed (I don't know how installing sw on aix works.).

And I think that in this case you don't need to bother with it.

[Martin Gamwell Dawids]

> Test if tcl/tk is available in python. Run the following import in
> python console:
>
> >>> import Tkinter

bash-3.00$ python
Python 2.6.2 (r262:71600, Sep 2 2009, 00:55:24) [C] on aix5
Type "help", "copyright", "credits" or "license" for more
information.
>>> import Tkinter

Traceback (most recent call last):

File "<stdin>", line 1, in <module>
ImportError: No module named Tkinter
>>>

I guess that is the problem.

> If it does not work then python was not built with tcl/tk or this
> support is not installed (I don't know how installing sw on aix works.).
>
> And I think that in this case you don't need to bother with it.

Ok, I will leave the new robust detection code so Configure.py does
not crash on AIX.

/Martin

[Martin Gamwell Dawids]

> Could you please attach your patches to the following ticket?
>
> http://www.pyinstaller.org/ticket/343

Just attached a patch that works with PyInstaller 1.5.1 on AIX 6.1
with Python 6.2 and GCC 4.2.0

> If you are happy with your changes could you please try backport your
> changes for the svn version?

Working on it.

[Martin Zibricky]

Martin Gamwell Dawids píše v St 14. 09. 2011 v 08:28 -0700:

>
> Just attached a patch that works with PyInstaller 1.5.1 on AIX 6.1
> with Python 6.2 and GCC 4.2.0

Thanks.

I hope to look at it soon.

[Martin Gamwell Dawids]

Hi Martin,

I am almost done backporting to trunk and will submit a patch tomorrow
if all goes well.

> > Just attached a patch that works with PyInstaller 1.5.1 on AIX 6.1
> > with Python 6.2 and GCC 4.2.0
>
> Thanks.
>
> I hope to look at it soon.

The patch I submitted was generated using a version of diff on AIX
that does not work exactly like GNU diff and I cannot apply the patch
on my Mac. Therefore, I will generate a new patch file for PyInstaller
1.5.1 and submit that tomorrow as well.

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Čt 15. 09. 2011 v 08:02 -0700:

> I am almost done backporting to trunk and will submit a patch tomorrow
> if all goes well.

I hope to look at it over the weekend.

[Martin Gamwell Dawids]

> I am almost done backporting to trunk and will submit a patch tomorrow
> if all goes well.

I just uploaded an updated patch for 1.5.1 (same contents, but
generated using 'diff' on Mac as 'diff on AIX worked strange and its
output did not work with 'patch' neither on my Mac nor on AIX).

I also uploaded a backport patch for trunk.

Both are tested on AIX 6.1, Python 2.6.2, GCC 4.2.0.

I ran the tests in 'buildtests'

On 1.5.1 the results were:

On trunk the results were:

{'failed': ['import/test_zipimport',
'import/test_relative_import',
'multipackage/test_multipackage1',
'multipackage/test_multipackage2',
'multipackage/test_multipackage3',
'multipackage/test_multipackage4',
'multipackage/test_multipackage5',
'basic/test_absolute_ld_library_path'],
'passed': ['basic/test_1',
'basic/test_2',
'basic/test_5',
'basic/test_6',
'basic/test_7',
'basic/test_8',
'basic/test_9',
'basic/test_10',
'basic/test_11',
'basic/test_12',
'basic/test_13',
'basic/test_14',
'basic/test_16',
'basic/test_email',
'basic/test_encoders',
'basic/test_f_option',
'basic/test_filename',
'basic/test_celementtree',
'basic/test_nestedlaunch0',
'basic/test_nestedlaunch1',
'import/test_relative_import2',
'import/test_relative_import3',
'import/test_error_during_import',
'basic/test_getfilesystemencoding'],
'skipped': ['basic/test_ctypes',
'libraries/test_wx',
'libraries/test_numpy',
'import/test_zipimport2',
'libraries/test_enchant',
'libraries/test_pycrypto',
'import/test_ctypes_cdll_c',
'libraries/test_sqlalchemy',
'import/test_ctypes_cdll_c2']}

I guess the latter is not perfectly fine.

Any idea what to do about the failing tests on trunk?

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Pá 16. 09. 2011 v 05:41 -0700:

>
> Any idea what to do about the failing tests on trunk?

Look at them if they contain platform specific code like:

if is_lin
elif is_win
elif is_mac

[Martin Zibricky]

Martin Gamwell Dawids píše v Pá 16. 09. 2011 v 05:41 -0700:

> Any idea what to do about the failing tests on trunk?

in the case of multipackage look at files

trunk/buildtests/multipackage/*.toc

I think we could remove the pattern

'(?i).*python.*'

since libpython is not available when it is linked statically.

[Martin Gamwell Dawids]

> in the case of multipackage look at files
>
> trunk/buildtests/multipackage/*.toc
>
> I think we could remove the pattern
>
> '(?i).*python.*'
>
> since libpython is not available when it is linked statically.

I guess you are right even though I do not understand the tests to the
full extent.

Removing the pattern
'(?i).*multipackage1_B\.exe:.*python.*'
from the file
"test_multipackage1.toc"

and the pattern
'(?i).*python.*'
from the file
"multipackage1_B.toc"

fixes the "multipackage/test_multipackage1.py" test.


I guess that the tests are matching the patterns in
trunk/buildtests/multipackage/<testname>.toc
against
trunk/buildtests/multipackage/dist/<testname>.toc
Right?

Can we remove the pattern in a way so it is still there for
dynamically linked platforms or should I simply remove it?

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Pá 16. 09. 2011 v 07:44 -0700:

>
> Can we remove the pattern in a way so it is still there for
> dynamically linked platforms or should I simply remove it?

We could simply remove it. Because if bootloader is linked dynamically
and libpython is missing then other test cases would fail. So we do not
need to check it in multipackage tests.

[Martin Gamwell Dawids]

Hi Martin

Regarding the multipackage tests you wrote (about the patterns
matching dynamic Python library):

> We could simply remove it. Because if bootloader is linked dynamically
> and libpython is missing then other test cases would fail. So we do not
> need to check it in multipackage tests.

Ok, that fixed all the multipackage tests.


The other tests that failed were:
1. import/test_zipimport
2. import/test_relative_import
3. basic/test_absolute_ld_library_path


1) import/test_zipimport:
The problem was that the package 'pkg_resources' was missing on my AIX
installation.
Installing setuptools from http://pypi.python.org/pypi/setuptools
fixed the problem.

However, not import/test_zipimport2 fails which it did not before.
Strange.
I write more about that below.


2) import/test_relative_import:
This test fails on AIX bu also on my Mac like this:

#################### EXECUTING TEST import/test_relative_import
####################

RUNNING: dist/test_relative_import/test_relative_import.exe

Traceback (most recent call last):

File "<string>", line 2, in <module>
File "../PyInstaller/iu.py", line 424, in importHook
File "../PyInstaller/iu.py", line 514, in doimport
File "<..snip..>/buildtests/import/build/pyi.darwin/
test_relative_import/outPYZ1.pyz/relimp.B.C", line 3, in <module>
File "../PyInstaller/iu.py", line 443, in importHook
ImportError: No module named
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! TEST import/
test_relative_import FAILED !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
{'failed': ['import/test_relative_import'], 'passed': [], 'skipped':
[]}

My trunk dir is checked out from revision 1585.

Does the test run cleanly on your machine?


3) basic/test_absolute_ld_library_path:
This test does not really check what it says in the comments:

# LD_LIBRARY_PATH set by bootloader should not contain ./

However, the assert simply says:

libpath =
os.path.normpath(os.path.abspath(os.path.dirname(sys.executable)))
libpath += '/''
<..snip..>
assert libpath == os.environ.get('LD_LIBRARY_PATH')

In my environment LD_LIBRARY_PATH was already set (but not to "./").
This caused the test to fail. Unsetting the LD_LIBRARY_PATH var before
running the test made it pass.

Any good ideas how to improve the test?
Really, the test should simply look for the value of 'libpath' in the
LD_LIBRARY_PATH. It cannot assume the var is not set beforehand. Or we
should add a comment to the test which says we assume that
LD_LIBRARY_PATH is not set before the test runs and explains how to
unset it (e.g. from bash "unset LD_LIBRARY_PATH").

What do you think?


4) import/test_zipimport2
Now the test status for trunk on my AIX box is:

{'failed': ['import/test_zipimport2', 'import/test_relative_import'],

'passed': ['basic/test_1',
'basic/test_2',
'basic/test_5',
'basic/test_6',
'basic/test_7',
'basic/test_8',
'basic/test_9',
'basic/test_10',
'basic/test_11',
'basic/test_12',
'basic/test_13',
'basic/test_14',
'basic/test_16',
'basic/test_email',
'basic/test_encoders',
'basic/test_f_option',
'basic/test_filename',

'import/test_zipimport',

'basic/test_celementtree',
'basic/test_nestedlaunch0',
'basic/test_nestedlaunch1',
'import/test_relative_import2',
'import/test_relative_import3',
'import/test_error_during_import',

'multipackage/test_multipackage1',
'multipackage/test_multipackage2',
'multipackage/test_multipackage3',
'multipackage/test_multipackage4',
'multipackage/test_multipackage5',

'basic/test_getfilesystemencoding',
'basic/test_absolute_ld_library_path'],

'skipped': ['basic/test_ctypes',
'libraries/test_wx',
'libraries/test_numpy',

'libraries/test_enchant',
'libraries/test_pycrypto',
'import/test_ctypes_cdll_c',
'libraries/test_sqlalchemy',
'import/test_ctypes_cdll_c2']}

You can see, now a new test has started failing: import/
test_zipimport2
(As I also wrote above.)
The output is:

now importing setuptools.dist

Traceback (most recent call last):

File "<string>", line 42, in <module>
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 424, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 514, in doimport
exec co in mod.__dict__
File "/tmp/_MEI4Rynaa/eggs/setuptools-0.6c11-py2.6.egg/setuptools/
__init__.py", line 2, in <module>
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 424, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 514, in doimport
exec co in mod.__dict__
File "/tmp/_MEI4Rynaa/eggs/setuptools-0.6c11-py2.6.egg/setuptools/
extension.py", line 2, in <module>
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 424, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 514, in doimport
exec co in mod.__dict__
File "/tmp/_MEI4Rynaa/eggs/setuptools-0.6c11-py2.6.egg/setuptools/
dist.py", line 5, in <module>
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 424, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 514, in doimport
exec co in mod.__dict__
File "/tmp/_MEI4Rynaa/eggs/setuptools-0.6c11-py2.6.egg/setuptools/
command/install.py", line 2, in <module>
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 424, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 514, in doimport
exec co in mod.__dict__
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched-mgd/
buildtests/import/build/pyi.aix5/test_zipimport2/outPYZ1.pyz/
distutils.command.install", line 21, in <module>
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 424, in importHook
mod = _self_doimport(nm, ctx, fqname)
File "<workdir>/pyinstaller-patches/pyinstaller-trunk-patched/
PyInstaller/iu.py", line 514, in doimport
exec co in mod.__dict__
File "/tmp/_MEI4Rynaa/eggs/setuptools-0.6c11-py2.6.egg/site.py",
line 73, in <module>
File "/tmp/_MEI4Rynaa/eggs/setuptools-0.6c11-py2.6.egg/site.py",
line 22, in __boot
File "/tmp/_MEI4Rynaa/eggs/setuptools-0.6c11-py2.6.egg/site.py",
line 73, in <module>
File "/tmp/_MEI4Rynaa/eggs/setuptools-0.6c11-py2.6.egg/site.py",
line 38, in __boot
ImportError: Couldn't find the real 'site' module
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! TEST import/test_zipimport2
FAILED !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
{'failed': ['import/test_zipimport2'], 'passed': [], 'skipped': []}

I don't really understand why this test did not fail before. It is
also importing the 'pkg_resources' package which I have just
installed, as you have read above.

Running the test Python code outside PyInstaller works fine:

$ python import/test_zipimport2.py
<..snip..>
-----------
now importing pkg_resources
-----------
now importing setuptools.dist
-----------
now importing setuptools.command
$

Any ideas?

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Po 19. 09. 2011 v 05:13 -0700:

> My trunk dir is checked out from revision 1585.
>
> Does the test run cleanly on your machine?

No, it does not. On my linux box it's the same (No module 'named'). I
also wonder why it is failing.

Thanks

[Martin Gamwell Dawids]

Hi Martin,

1) import/test_relative_import

> > My trunk dir is checked out from revision 1585.
>
> > Does the test run cleanly on your machine?
>
> No, it does not. On my linux box it's the same (No module 'named'). I
> also wonder why it is failing.

Ok, I will leave this test as it is.

2) basic/test_absolute_ld_library_path

I will add a comment to the test telling users to try to unset
LD_LIBRARY_PATH before running the test if they see it failing.

3) import/test_zipimport2

Do you have a good idea what to do about this test that now fails for
me?
Or should I ignore it since it has to do with installation of
setuptools?
(By the way, this test passes on my Mac.)

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Po 19. 09. 2011 v 05:40 -0700:

>
> 2) basic/test_absolute_ld_library_path
>
> I will add a comment to the test telling users to try to unset
> LD_LIBRARY_PATH before running the test if they see it failing.

That sounds good.

>
> 3) import/test_zipimport2
>
> Do you have a good idea what to do about this test that now fails for
> me?
> Or should I ignore it since it has to do with installation of
> setuptools?

I will add it to the list of skipped when setuptools is not installed.

[Martin Gamwell Dawids]

Great.

I just uploaded a patch to the test files on trunk.

/Martin

[Martin Gamwell Dawids]

Well...

While everything seems to work on the AIX box where I have Python
installed, moving the produced binary to another AIX box without
Python seems to cause problems.

Apparently the libpython2.6.a is _not_ a static library, so when I
move the binary to another AIX box it cannot find the Python library.

Inspecting the libpython2.6.a file reveals it is a shared library:

bash-3.00$ dump -ov /opt/freeware/lib/libpython2.6.a

/opt/freeware/lib/libpython2.6.a[libpython2.6.so]:

***Object Module Header***
# Sections Symbol Ptr # Symbols Opt Hdr Len Flags
4 0x001da39e 16588 72 0x3002
Flags=( EXEC DYNLOAD SHROBJ DEP_SYSTEM )
Timestamp = "02 sep 00:55:27 2009"
Magic = 0x1df (32-bit XCOFF)
<..snip..>

Also, the produced binary (bootloader) depends on dynamic libraries:

bash-3.00$ ldd test
test needs:
/opt/freeware/lib/libz.a(libz.so.1)

/opt/freeware/lib/libpython2.6.a(libpython2.6.so)
/usr/lib/libc.a(shr.o)
/usr/lib/librtl.a(shr.o)
/usr/lib/libc.a(shr_64.o)
/usr/lib/libpthreads.a(shr_xpg5.o)
/usr/lib/libpthreads.a(shr_comm.o)
/unix
/usr/lib/libcrypt.a(shr.o)
/usr/lib/libcrypt.a(shr_64.o)

So I guess I need to dive deeper into linking the bootloader
correctly... :-(

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v St 21. 09. 2011 v 06:39 -0700:

> Apparently the libpython2.6.a is _not_ a static library, so when I
> move the binary to another AIX box it cannot find the Python library.
>
> Inspecting the libpython2.6.a file reveals it is a shared library:
>
> bash-3.00$ dump -ov /opt/freeware/lib/libpython2.6.a

does it mean that AIX uses *.a extension for shared libraries name?

>
> /opt/freeware/lib/libpython2.6.a[libpython2.6.so]:
>
> ***Object Module Header***
> # Sections Symbol Ptr # Symbols Opt Hdr Len Flags
> 4 0x001da39e 16588 72 0x3002
> Flags=( EXEC DYNLOAD SHROBJ DEP_SYSTEM )
> Timestamp = "02 sep 00:55:27 2009"
> Magic = 0x1df (32-bit XCOFF)
> <..snip..>
>
> Also, the produced binary (bootloader) depends on dynamic libraries:

The bootloader should depend on some standard dynamic libraries. That is
normal. Only the libpython should not be there.

Is the /opt/freeware/lib/libz.a available by default on every AIX
installation?

>
> bash-3.00$ ldd test
> test needs:
> /opt/freeware/lib/libz.a(libz.so.1)
> /opt/freeware/lib/libpython2.6.a(libpython2.6.so)
> /usr/lib/libc.a(shr.o)
> /usr/lib/librtl.a(shr.o)
> /usr/lib/libc.a(shr_64.o)
> /usr/lib/libpthreads.a(shr_xpg5.o)
> /usr/lib/libpthreads.a(shr_comm.o)
> /unix
> /usr/lib/libcrypt.a(shr.o)
> /usr/lib/libcrypt.a(shr_64.o)
>

> So I guess I need to dive deeper into linking the bootloader
> correctly... :-(

If the python library is dynamic library then you could just try the
same approach what's pyinstaller doing for linux:

Copy the libpython2.6.a to dist folder and update the bootloader to be
also looking for libpython with *.a extension.

Does environment variable LD_LIBRARY_PATH work on AIX?

[Martin Zibricky]

Martin Gamwell Dawids píše v St 21. 09. 2011 v 06:39 -0700:

> So I guess I need to dive deeper into linking the bootloader
> correctly... :-(

Maybe we should try to update the wscript to tell gcc to try linking
libpython statically.

Gcc has some options how to do that.

[Martin Gamwell Dawids]

Hi Martin,

> does it mean that AIX uses *.a extension for shared libraries name?

Yes, sort of. On AIX the correct term is a shared object which can
reside inside a library archive (*.a). When linking with the *.a file
the binary becomes dependent on the shared objects in the library and
the library must be available at runtime.

On AIX a shared library is a library (names something.a) which
contains one or more shared objects (but it can also contain non-
shared objects aka static object files.)

You can read more here (which is where I just got my limited knowledge
from):
http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/index.jsp?topic=%2Fcom.ibm.vacpp7a.doc%2Fgetstart%2Foverview%2Fport_aix_obj_lib.htm

> > Also, the produced binary (bootloader) depends on dynamic libraries:
>
> The bootloader should depend on some standard dynamic libraries. That is
> normal. Only the libpython should not be there.

Ok.

> Is the /opt/freeware/lib/libz.a available by default on every AIX
> installation?

Not sure, but I guess not. So this also poses problems when moving the
produced binary to another machine. Furthermore, I might have the
problem (not sure about that) that when building the bootloader with
GCC 4.x on one machine it does not run on another machine
(incompatible library versions).

> If the python library is dynamic library then you could just try the
> same approach what's pyinstaller doing for linux:
>
> Copy the libpython2.6.a to dist folder and update the bootloader to be
> also looking for libpython with *.a extension.

I had the same thought.
Originally (before modifying the bootloader) I simple moved
libpython2.6.a to the dist folder and renamed it to libpython2.6.so
(or whatever the bootloader asked for), but the bootloader failed to
load the library.

So I could try to do what you suggest but I am afraid it will not
work. But I will give it a try.

> Does environment variable LD_LIBRARY_PATH work on AIX?

Yes, I believe so. (Had to change it to make some of the tests pass in
the test suite for trunk.)

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v St 21. 09. 2011 v 06:39 -0700:

> So I guess I need to dive deeper into linking the bootloader
> correctly... :-(

If the libpython would be statically linked with bootloader then the
size of bootloader equals size(bootloader) + size(libpython26.a)

>
>

[Martin Gamwell Dawids]

> Maybe we should try to update the wscript to tell gcc to try linking
> libpython statically.
>
> Gcc has some options how to do that.

But doesn't that require that a static version of libpython is
available?

[Martin Gamwell Dawids]

> If the libpython would be statically linked with bootloader then the
> size of bootloader equals  size(bootloader) + size(libpython26.a)

Agreed.

[Martin Zibricky]

Martin Gamwell Dawids píše v St 21. 09. 2011 v 07:14 -0700:

> Not sure, but I guess not. So this also poses problems when moving the
> produced binary to another machine. Furthermore, I might have the
> problem (not sure about that) that when building the bootloader with
> GCC 4.x on one machine it does not run on another machine
> (incompatible library versions).

then the libz.a should be also present in the dist folder.

It seems to me that you will have to modify in bootloader source the
usage of dlopen() function. According to

http://stromberg.dnsalias.org/~strombrg/AIX-shared-libs.html

You will have to use dlopen like

dlopen('libpython2.6.a(libpython2.6.so)')

Explicitly saing what file should be imported from the *.a archive.

[Martin Gamwell Dawids]

> then the libz.a should be also present in the dist folder.

Maybe, yes.

> It seems to me that you will have to modify in bootloader source the
> usage of dlopen() function. According to
>
> http://stromberg.dnsalias.org/~strombrg/AIX-shared-libs.html
>
> You will have to use dlopen like
>
> dlopen('libpython2.6.a(libpython2.6.so)')
>
> Explicitly saing what file should be imported from the *.a archive.

Thank you for the link.
And then I guess I will need the libpython2.6.a in the dist folder.

However, it is funny that the bootloader (as I have changed it for
now) links the libpython2.6.a library which means:
* The python symbols are directly available in the bootloader (where I
assign them to the PI_ pointers)
* The bootloader loads the shared object from libpython2.6.a which it
finds in its default position.

So actually just moving libpython2.6.a to the dist folder solves this
part of the problem.

Do you know why the boot loader explicitly loads symbols from the
shared libs (using dlopen and dlsym) instead of simply linking to a
library? It seems kind of complicated.

I corresponds to a situation where you did not link to the std C lib
(which would still be loaded dynamically) but instead loaded the
library using dlopen and dlsym.

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v St 21. 09. 2011 v 08:33 -0700:

>
> Do you know why the boot loader explicitly loads symbols from the
> shared libs (using dlopen and dlsym) instead of simply linking to a
> library? It seems kind of complicated.

That's a hook how bootloader works.

If the libpython would be simply linking to this library you then need
libpython available in fixed location -
like: /opt/freeware/lib/libpython2.6.a

But for it to work you need libpython available at the default location
before running your binary created by pyinstaller.

Using dlopen for libpython solves this issue that you don't need
libpython in default location before executing your app - you are thus
able to run your app on system without python installed.

[Martin Zibricky]

Martin Gamwell Dawids píše v St 21. 09. 2011 v 08:33 -0700:

> then the libz.a should be also present in the dist folder.
>
> Maybe, yes.

If the libz.a is not available as static for bootloader we should then
try to use the bundled libz used for windows platform.

[Martin Gamwell Dawids]

Hi Martin,

> If the libpython would be simply linking to this library you then need
> libpython available in fixed location -
> like: /opt/freeware/lib/libpython2.6.a
>
> But for it to work you need libpython available at the default location
> before running your binary created by pyinstaller.

Not sure if you are right about this.
When I try to run the program on a machine with no Python
installation, it is missing "libpython2.6.a":

bash-3.00$ ./test
exec(): 0509-036 Cannot load program ./test because of the following
errors:
0509-150 Dependent module libpython2.6.a(libpython2.6.so)
could not be loaded.
0509-022 Cannot load module libpython2.6.a(libpython2.6.so).
0509-026 System error: A file or directory in the path name
does not exist.

Simply copying the file "libpython2.6.a" to the dist folder (for this
test I am using --onedir) changes the behaviour:

bash-3.00$ ./test
exec(): 0509-036 Cannot load program ./test because of the following
errors:
0509-130 Symbol resolution failed for MonitorBlockingSessions
because:
0509-136 Symbol mkdtemp (number 41) is not exported from
dependent module /usr/lib/libc.a(shr.o).
0509-192 Examine .loader section symbols with the
'dump -Tv' command.

So now it picks up the Python lib but runs into the next error (which
seems to be a too old version of the "libc".)

Also, it should be possible to change where an executable looks for
"statically" linked dynamic libs by using the linker option "-
blibpath". (However, it seems not to be necessary in this case.)

But...
I think I understand why the shared Python lib is not linked to the
bootloader but is loaded programmatically using "dlopen": in order to
work with different versions of Python, the code in the bootloader
cannot assume that all the possible symbols are available. If it did,
the linker would fail with older versions of Python where e.g.
"Py_IncRef" and "Py_DecRef" are not available.

Therefore, the library is loaded programmatically using "dlopen". Then
these two symbols are loaded optionally which you can see in "source/
common/launch.c" in function "int mapNames(HMODULE dll):
(Look for GETPROCOPT(...) below.)

int mapNames(HMODULE dll)
{
/* Get all of the entry points that we are interested in */
GETVAR(dll, Py_FrozenFlag);
<..snip..>
GETPROCOPT(dll, Py_IncRef);
GETPROCOPT(dll, Py_DecRef);
GETPROC(dll, PyImport_ExecCodeModule);
<..snip..>

if (!PI_Py_IncRef) PI_Py_IncRef = _EmulatedIncRef;
if (!PI_Py_DecRef) PI_Py_DecRef = _EmulatedDecRef;

return 0;
}

I think my approach now will be:
* Get the newest code from branch/1.5 (revision 1594 at the time of
writing).
* Apply my patches to this code (I know there will be some conflicts.)
* Keep the code I wrote for statically linking Python. It could be
used in the future maybe for
linking ActiveState Python.
* Keep the code you have added to the 1.5 branch in change 1586
regarding static linking of Python.
* Remove my comments regarding AIX in connection with static linking.
* In launch.c function "int loadPython()" add code for AIX to load the
shared lib correctly as
"libpython<ver>.a(libpythonx<ver>.so)".

After that I will have to address the problems with libc and libz.

Does it sound reasonable?

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Čt 22. 09. 2011 v 02:49 -0700:

> Not sure if you are right about this.
> When I try to run the program on a machine with no Python
> installation, it is missing "libpython2.6.a":

Remember that you changed the wscript file to always link the bootloader
with libpython on aix.

[Martin Zibricky]

Martin Gamwell Dawids píše v Čt 22. 09. 2011 v 02:49 -0700:

> I think my approach now will be:
> * Get the newest code from branch/1.5 (revision 1594 at the time of
> writing).
> * Apply my patches to this code (I know there will be some conflicts.)

I only commited the changes

> * Keep the code I wrote for statically linking Python. It could be
> used in the future maybe for
> linking ActiveState Python.

Could you submit the patch with code for statical linking to a new
ticket? like 'support statical linking of python library'?

> * Keep the code you have added to the 1.5 branch in change 1586
> regarding static linking of Python.
> * Remove my comments regarding AIX in connection with static linking.
> * In launch.c function "int loadPython()" add code for AIX to load the
> shared lib correctly as
> "libpython<ver>.a(libpythonx<ver>.so)".

I think we would need to modify the bindepend.py for resolving binary
dependencies for AIX. There should be a function which uses 'ldd'
command to do this. Maybe it should be modified for aix if id does not
work.

>
> After that I will have to address the problems with libc and libz.

You don't need to bother with libc.(in case you do not use your own
version). Every binary in system is dependent on system libc.

Let me know if should link libz statically with loader for aix. It
should be trivial to make such changes to wscript.

>
> Does it sound reasonable?

[Martin Gamwell Dawids]

> > I think my approach now will be:
> > * Get the newest code from branch/1.5 (revision 1594 at the time of
> > writing).
> > * Apply my patches to this code (I know there will be some conflicts.)
>
> I only commited the changes

Do you want a patch against 1.5.1 or against revision 1594 of branches/
1.5?

I have a patch against branches/1.5 ready:
* It builds upon what you committed.
* It removes the default static linking for AIX.
* It adds my changes to launch.c/.h

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Čt 22. 09. 2011 v 04:41 -0700:

> revision 1594 of branches/
> 1.5?

If possible against ./branches/1.5

[Martin Gamwell Dawids]

> If possible against ./branches/1.5

I have it ready so if you would approve the new ticket I have created
(Support statical linking of Python library) I will attach the patch.

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Čt 22. 09. 2011 v 05:13 -0700:

> I have it ready so if you would approve the new ticket I have created
> (Support statical linking of Python library) I will attach the patch.

Done

[Martin Gamwell Dawids]

> > I have it ready so if you would approve the new ticket I have created
> > (Support statical linking of Python library) I will attach the patch.
>
> Done

I have attached the patch.

[Martin Gamwell Dawids]

> It seems to me that you will have to modify in bootloader source the
> usage of dlopen() function. According to
>
> http://stromberg.dnsalias.org/~strombrg/AIX-shared-libs.html
>
> You will have to use dlopen like
>
> dlopen('libpython2.6.a(libpython2.6.so)')
>
> Explicitly saing what file should be imported from the *.a archive.

Working on it, but "dlopen()" is not happy:

Loading '/data/maconomy/mgd/pyinstaller-patches/2nd/gcc/
libpython2.6.a(libpython2.6.so)'
0509-022 Cannot load module /data/maconomy/mgd/pyinstaller-
patches/2nd/gcc/libpython2.6.a(libpython2.6.so).

0509-026 System error: A file or directory in the path name
does not exist.

Also tried a with "[]" around the shared object name and appending ".
1" to the so.

I guess I have to Google around a bit.

/Martin

[Martin Gamwell Dawids]

> Working on it, but "dlopen()" is not happy:
>
> Loading '/data/maconomy/mgd/pyinstaller-patches/2nd/gcc/
> libpython2.6.a(libpython2.6.so)'
>         0509-022 Cannot load module /data/maconomy/mgd/pyinstaller-
> patches/2nd/gcc/libpython2.6.a(libpython2.6.so).
>         0509-026 System error: A file or directory in the path name
> does not exist.

It is possible to extract the shared object from the archive like
this:

bash-3.00$ ar -v -x libpython2.6.a libpython2.6.so

Calling
dlopen("libpython2.6.so", ...)
no longer fails.

[Martin Gamwell Dawids]

Ok, here it is:

Loading a shared object from a library archive on AIX:
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.basetechref/doc/basetrf1/dlopen.htm

Use the flag RTLD_MEMBER when calling dlopen().
This causes dlopen() to pass L_LOADMEMBER when calling on to load().

And then you can call dlopen() like this:

dlopen("<path>/libpython2.6.a(libpython2.6.so)", RTLD_MEMBER | ...);

:-) Martin

[Martin Zibricky]

Hi Martin,

any progress with that?

Martin Gamwell Dawids píše v Čt 22. 09. 2011 v 08:38 -0700:

[Martin Gamwell Dawids]

Hi Martin,

> Hi Martin,
>
> any progress with that?

Sorry, but I have been very busy the past week. Hope it clears out
soon so I can continue working on the patch.

/Martin

[Martin Gamwell Dawids]

Hi Martin,

Today, I had time to look into the AIX patch again. The launcher now
loads the shared libpython2.6.a dynamically on AIX. Also, I have
changed Build.py and bindepend.py in order to get the shared lib
copied to the distribution folder.

So everything works on the AIX machine where I work with pyinstaller.
This machine runs AIX 6.1.

I then moved the produced executables (both packaged as one file and
as a dir) to another AIX machine running AIX 5.2. On this machine the
generated binaries do not run:

-bash-3.00$ dist/test/test
exec(): 0509-036 Cannot load program dist/test/test because of the
following errors:
0509-130 Symbol resolution failed for test because:

0509-136 Symbol mkdtemp (number 41) is not exported from
dependent module /usr/lib/libc.a(shr.o).
0509-192 Examine .loader section symbols with the
'dump -Tv' command.

As you can see, the function 'mkdtemp' is missing from 'libc'.
Inspecting 'libc' on the two machines reveals that this is indeed a
difference:

AIX 5.2 machine:
--------
-bash-3.00$ dump -Tv /usr/lib/libc.a
<..snip..>
[1203] 0x000440dc .data EXP DS SECdef [noIMid]
ftw64
[1204] 0x000440e8 .data EXP DS SECdef [noIMid]
mkstemp64
[1205] 0x000440f4 .data EXP DS SECdef [noIMid]
mkstemp
<..snip..>

AIX 6.1 machine:
--------
<..snip..>
[1448] 0x0006cc90 .data EXP DS SECdef [noIMid]
ftw64
[1449] 0x0006cc9c .data EXP DS SECdef [noIMid]
mkdtemp <--- HERE IT IS
[1450] 0x0006cca8 .data EXP DS SECdef [noIMid]
mkstemp64
[1451] 0x0006ccb4 .data EXP DS SECdef [noIMid]
mkstemp
<..snip..>

Any ideas how to deal with this problem?

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v St 19. 10. 2011 v 08:53 -0700:

> Any ideas how to deal with this problem?

It is similar to solaris support. Patch for solaris is merged in trunk
and in 1.5 branch and it includes hook for the mkdtemp function.

Maybe it is similar for AIX. Look at

http://www.pyinstaller.org/changeset/1610

[Martin Gamwell Dawids]

> It is similar to solaris support. Patch for solaris is merged in trunk
> and in 1.5 branch and it includes hook for the mkdtemp function.
>
> Maybe it is similar for AIX. Look at
>
> http://www.pyinstaller.org/changeset/1610

Thanks for the pointer. It got me past that hurdle.

Now, I package the application as "onedir" and "onefile".

Executing both gives me this error:

-bash-3.00$ test-dir/test
mod is NULL - structTraceback (most recent call last):
File "/opt/freeware/lib/python2.6/struct.py", line 1, in <module>
ImportError: No such file or directory
mod is NULL - archiveTraceback (most recent call last):
File "/data/maconomy/mgd/pyinstaller-patches/2nd/pyinstaller/
archive.py", line 42, in <module>
ImportError: No module named struct
Traceback (most recent call last):
File "<string>", line 25, in <module>
ImportError: No module named archive
-bash-3.00$

(The paths in the error messages above are paths from the machine
where the application was built using PyInstaller. These paths does
not exist on the machine where I try to execute the application.)

If I "cd" into the "onedir" application and then run the application
from here, I get:

-bash-3.00$ cd test-dir
-bash-3.00$ ./test
exec(): 0509-036 Cannot load program ./test because of the following
errors:
0509-130 Symbol resolution failed for ./libc.a(shr.o) because:
0509-136 Symbol acct_wpar (number 24) is not exported from
dependent module /unix.
<..snip..>

It seems the application is packaged with the versions of 'libc' and
'libpthreads' from the build machine which obviously is not correct.
If I delete those two, the application works:

-bash-3.00$ rm libc.a
-bash-3.00$ rm libpthreads.a
-bash-3.00$ ./test
Hello PyInstaller!
-bash-3.00$

Backing up one dir (cd ..) and running the command (test-dir/test)
still gives the error with importing "struct" and "archive" modules.

So it seems I have (at least) two problems left to resolve:

1) I should prevent libc.a and libpthreads.a from being packaged along
with the application. I guess it is the regexes in bindepend.py which
does not work for AIX, as they incorrectly assume all shared libs ends
with ".so".

2) Python cannot find packages to import unless the current working
dir contains the shared objects. Any ideas how to fix this?

Thank you for your invaluable help so far!

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Čt 20. 10. 2011 v 02:51 -0700:

> Backing up one dir (cd ..) and running the command (test-dir/test)
> still gives the error with importing "struct" and "archive" modules.

It seems like the bootloader is working now.

>
> So it seems I have (at least) two problems left to resolve:
>
> 1) I should prevent libc.a and libpthreads.a from being packaged along
> with the application. I guess it is the regexes in bindepend.py which
> does not work for AIX, as they incorrectly assume all shared libs ends
> with ".so".

yes, that's correct. Maybe there are more libraries with .a which should
be also ignored.

>
> 2) Python cannot find packages to import unless the current working
> dir contains the shared objects. Any ideas how to fix this?

You should try to use ArchiveViewer.py to inspect created binaries what
python modules get included.

Look for linux specific python code which is not adjusted for AIX.

Add some print statements to the code where python module dependencies
are analyzed.

look at tools 'ldd' and 'objdump' if these are available on your AIX
devel. machine and look how these are used in pyinstaller if their
output is properly parsed by pyinstaller.
(maybe their output could is formatted differently and thus is not
properly parsed by pyinstaller)

>
> Thank you for your invaluable help so far!

Some hints:
- i would recommend focusing on --onedir mode only for now (onefile is
harder)
- use option --debug for ./Makespec.py to get more debug messages
- in iu.py at the top in debug(msg) function enable debugging to see at
runtime what modules are loaded
- do not worry about trunk, I can do the backport of AIX support

[Martin Gamwell Dawids]

> It seems like the bootloader is working now.

That was my conclusion too. :-)

> > So it seems I have (at least) two problems left to resolve:
>
> > 1) I should prevent libc.a and libpthreads.a from being packaged along
> > with the application. I guess it is the regexes in bindepend.py which
> > does not work for AIX, as they incorrectly assume all shared libs ends
> > with ".so".
>
> yes, that's correct. Maybe there are more libraries with .a which should
> be also ignored.

Yes, I have been experimenting and actually I could remove all of them
(except of course libpython). I will have to test it on a clean AIX
installation with no devel tools installed. (I hope I can find such a
(virtual) machine here.)

> > 2) Python cannot find packages to import unless the current working
> > dir contains the shared objects. Any ideas how to fix this?

Sorry, I phrased that incorrectly. I believe PyInstaller has correctly
packed all the Python so libs into the application dir. When my
current working dir (cwd) is in the folder of the packaged --onedir
application everything works. E.g.

-bash-3.00$ cd test

-bash-3.00$ ./test
Hello PyInstaller!
-bash-3.00$

However, if I go up one level, it fails:

-bash-3.00$ cd ..
-bash-3.00$ test/test

mod is NULL - structTraceback (most recent call last):
File "/opt/freeware/lib/python2.6/struct.py", line 1, in <module>
ImportError: No such file or directory
mod is NULL - archiveTraceback (most recent call last):
File "/data/maconomy/mgd/pyinstaller-patches/2nd/pyinstaller/
archive.py", line 42, in <module>
ImportError: No module named struct
Traceback (most recent call last):
File "<string>", line 25, in <module>
ImportError: No module named archive
-bash-3.00$

It seems that either the search path is not set correctly _or_ the
current working dir is not set correctly before loading Python. I
guess this is a problem in the boot loader. Do you know where to look?

> Look for linux specific python code which is not adjusted for AIX.
>
> Add some print statements to the code where python module dependencies
> are analyzed.
>
> look at tools 'ldd' and 'objdump' if these are available on your AIX
> devel. machine and look how these are used in pyinstaller if their
> output is properly parsed by pyinstaller.
> (maybe their output could is formatted differently and thus is not
> properly parsed by pyinstaller)

I already did that and also changed code in 'Build.py' and
'bindepend.py' to accomodate for differences in naming of shared libs
(.s vs .so) and differences in 'ldd' output on AIX. On Linux each
'ldd' output line is something like this "libname => libpath
(address)" whereas on AIX is is "libpath(sharedobj)". So I am pretty
sure PyInstaller picks up all libraries (and forgets to exclude some
like 'libc').

> Some hints:
> - i would recommend focusing on --onedir mode only for now (onefile is
> harder)
> - use option --debug  for ./Makespec.py to get more debug messages
> - in iu.py  at the top in debug(msg) function enable debugging to see at
> runtime what modules are loaded

Thanks for the tips!

> - do not worry about trunk, I can do the backport of AIX support

Thank you. However, I already did it (with my previous patch), so if I
have time, I might do it or you, as I can also test if it works.

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Čt 20. 10. 2011 v 04:44 -0700:

> It seems that either the search path is not set correctly _or_ the
> current working dir is not set correctly before loading Python. I
> guess this is a problem in the boot loader. Do you know where to look?

Boot loader sets LD_LIBRARY_PATH for linux. However I found out that for
AIX it should set LIBPATH environment variable.

http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html

Look in the bootloader code for place where LD_LIBRARY_PATH is set and
try to use LIBPATH for AIX.

[Martin Gamwell Dawids]

> Boot loader sets LD_LIBRARY_PATH for linux. However I found out that for
> AIX it should set LIBPATH environment variable.

Brilliant!
Just changed it and now the bootloader works.

Now, I am trying to set up a clean AIX installation to check which
libs I should exclude (other than libc and libpthreads).

/Martin

[Martin Gamwell Dawids]

Hi Martin,

> Now, I am trying to set up a clean AIX installation to check which
> libs I should exclude (other than libc and libpthreads).

A little AIX info.

AIX 5.2
-----------
I have an old machine running AIX 5.2, but it is not clean as it has
developer tools (e.g. make, gcc, g++, etc.) installed.
This is the 5.2 machine I have been testing on so far.
I do not have compatible hardware where I can install AIX 5.2.
Support for AIX 5.2 ended on April 30, 2009.

AIX 5.3
-----------
I have an old machine running AIX 5.3, but it is not clean as it has
developer tools (e.g. make, gcc, g++, etc.) installed.
I haven't tested on this machine (yet).
I do not have compatible hardware where I can install AIX 5.3.
AIX 5.3 is still supported, but support ends 30th April 2012.

AIX 6.1 (released November 9, 2007)
-----------
As I write, I am doing a clean install of AIX 6.1 on a virtual server.
This way I can be sure that everything needed is packaged by
PyInstaller as this machine will have nothing but the standard
libraries installed.
End of support for AIX 6.1 is not announced yet (AFAIK).

AIX 7.1 (released Sept 10, 2010)
-----------
I don't have access to AIX 7.1


Suggestion:
-----------------
The only reason I had to include the 'mkdtemp' hack on AIX was since
this function was not yet in 'libc' in AIX 5.2 (and maybe also not in
5.3). However, the hack does no harm. Therefore, I suggest that I
leave this code in as it makes PyInstaller generated programs run on
AIX 5.2.

With regards to finding out which libs needs to be redistributed, I
will test this on my clean AIX 6.1 installation. I cannot test this
reliably on 5.2 og 5.3 as I cannot get to a clean install of these AIX
versions.

So what we will end up with is an AIX version of PyInstaller that
works with 6.1 where it is tested. It will probably also run on 5.2
and 5.3.

Any suggestions?

Regards,
Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Čt 20. 10. 2011 v 07:53 -0700:

> So what we will end up with is an AIX version of PyInstaller that
> works with 6.1 where it is tested. It will probably also run on 5.2
> and 5.3.
>
> Any suggestions?

Pyinstaller will probably work on 5.x. But the apps created on 6.1 might
not work on 5.x. I think that apps created on 6.1 should work on 7.

[Martin Gamwell Dawids]

I am working on extending the exclude list 'excludes' in bindepend.py.
It strikes me that this list is the union of all libs that should be
excluded on all platforms.

I think of splitting the list up so depending on platform, different
things are added to the list. DLLs, for example, are only relevant on
Windows and Cygwin. GNU libs (glibc) are standard on Linux, but not on
AIX.

Of course, by changing the list I run the risk of breaking something.
However, without doing something, I will exclude too many libs on AIX.

To be (more) sure not to break anything, I could start out by:
* Excluding all DLLs currently in the list when on Windows and Cygwin.
* Excluding all lib*.so currently in the list when on any non-AIX
platforms.
* Create my hand tailored exclude list of lib*.a when on AIX.

A more gentle approach could be:
* Leave the exclude list as is for all non-AIX platform (and let
somebody else do the cleanup here later).
* Create my hand tailored exclude list of lib*.a when on AIX.
Doing this I am sure not to make anything worse than today on non-AIX
platforms.

What would you prefer?

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Čt 20. 10. 2011 v 10:12 -0700:

> A more gentle approach could be:
> * Leave the exclude list as is for all non-AIX platform (and let
> somebody else do the cleanup here later).
> * Create my hand tailored exclude list of lib*.a when on AIX.
> Doing this I am sure not to make anything worse than today on non-AIX
> platforms.
>
> What would you prefer?

Just create an AIX specific exclude list. Just keep minimum changes to
make it work on aix. Since it is a stable branch.

In svn trunk these exclude lists are already splitted for platforms.

[Martin Gamwell Dawids]

> Just create an AIX specific exclude list. Just keep minimum changes to
> make it work on aix. Since it is a stable branch.

You are right.

> In svn trunk these exclude lists are already splitted for platforms.

Good.

I have now made an AIX-specific exclude list. Of course I cannot be
sure it contains everything it should but I have put these libs in the
exclude list:

* All the libs that I used to get along when running PyInstaller on
AIX
but which I have verified are already on my clean AIX 6.1 machine.

* All the libs already in the list for Linux which I (after
replacing .so with .a)
have verified are already on my clean AIX 6.1 machine.

* the glibc libraries are not on the AIX exclude list as GNU C
libraries
are not installed on a clean AIX installation.

I have tested with a minor program and it works in both --one-dir and
--one-file mode. The program is built on AIX 6.1 and is subsequently
tested on AIX 6.1 (build machine and cleanly installed machine), on
AIX 5.3 (devel machine) and on AIX 5.2 (devel machine).

I have also run the test suite on the AIX 6.1 build machine. Results:

{'failed': ['test-zipimport2'],
'passed': ['test1',
'test2',
'test5',
'test6',
'test7',
'test8',
'test9',
'test10',
'test11',
'test12',
'test13',
'test14',
'test-email',
'test-encoders',
'test_f_option',
'test_filename',
'test-zipimport1',
'test-celementtree',
'test-nestedlaunch0',
'test-nestedlaunch1',
'test-relative-import',
'test-relative-import2',
'test-relative-import3',
'test_error_during_import',
'test_getfilesystemencoding'],
'skipped': ['test15',
'test-wx',
'test-numpy',
'test-pycrypto',
'test-ctypes-cdll-c',
'test-ctypes-cdll-c2']}

Like earlier in this thread some tests are skipped, but one test
fails: 'test-zipimport2'
Running the test file from the command line works fine (i.e. executing
"python test-zipimport2.py").

It looks like the test cannot find the site package "setuptools".
However, as previously discussed, this should be installed (or the
test would be skipped). On my machine I have found these files:
/opt/freeware/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg
/opt/freeware/lib/python2.6/site-packages/setuptools.pth

The last lines of the stack trace says:

File "/data/maconomy/mgd/pyinstaller-patches/2nd/pyinstaller/iu.py",
line 521, in doimport
exec co in mod.__dict__
File "/tmp/_MEIPzajUa/eggs/setuptools-0.6c11-py2.6.egg/site.py",
line 73, in <module>
File "/tmp/_MEIPzajUa/eggs/setuptools-0.6c11-py2.6.egg/site.py",
line 22, in __boot
File "/tmp/_MEIPzajUa/eggs/setuptools-0.6c11-py2.6.egg/site.py",
line 73, in <module>
File "/tmp/_MEIPzajUa/eggs/setuptools-0.6c11-py2.6.egg/site.py",
line 38, in __boot
ImportError: Couldn't find the real 'site' module

Should I look further into why this test fails, and do you have any
ideas where to look?

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Čt 20. 10. 2011 v 11:50 -0700:

> Should I look further into why this test fails, and do you have any
> ideas where to look?

Do not bother with test. I fixed it in svn so to fix it for 1.5 should
be easy for me.

[Martin Gamwell Dawids]

> Do not bother with test. I fixed it in svn so to fix it for 1.5 should
> be easy for me.

Ok, fine. That means that my patch is ready. :-)

I will attach a new patch to the AIX ticket. The patch is against the
1.5 branch (from which I checked out to revision 1697) and it applies
cleanly. I would be happy to backport it to trunk, but if you could
look through the 1.5 patch first, then I will wait for your comments
before starting to avoid double work.

Could you delete the old incorrect patches 'pyinstaller-1.5.1-
aix.patch' and 'pyinstaller-trunk-aix.patch' from the AIX ticket?
http://www.pyinstaller.org/ticket/343
Please leave the 'pyinstaller-trunk-aix-test.patch' patch as this is
still relevant (but might need an update).

/Martin

[Martin Gamwell Dawids]

I just attached the patch to the ticket:
http://www.pyinstaller.org/attachment/ticket/343/pyinstaller-1.5-rev1697-aix-support.patch

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Čt 20. 10. 2011 v 12:07 -0700:

> I would be happy to backport it to trunk, but if you could
> look through the 1.5 patch first, then I will wait for your comments
> before starting to avoid double work.

I commited your patch into 1.5 branch.
Thanks

Tests worked for me on linux.

Just a note to C code:

for windows compatibility we try to avoid using mixed declaration and
code, like:

uint32_t pyvers_major;
uint32_t pyvers_minor;

pyvers_major = ntohl(f_cookie.pyvers) / 10;
pyvers_minor = ntohl(f_cookie.pyvers) % 10;

const char* dllPathPrefix = f_workpath ? f_workpath : f_homepath;

That was only change I made. Otherwise it seems ok. Could you please
test the code directly from 1.5 branch if it still woks?

If you want to port AIX to trunk I would appreciate that. The code
changed recently. I made some refactoring. For code specific to
Linux/Solaris/AIX there is a variable 'is_unix'.

Do you think you could write some text for the manual? I would welcome
some info about:
- what and how to install tools on AIX necessary for pyinstaller to work
- how to compile bootloader on AIX (in case there is something special
to be done by users).

[Martin Gamwell Dawids]

> I commited your patch into 1.5 branch.
> Thanks

Great!

> Just a note to C code:
>
> for windows compatibility we try to avoid using mixed declaration and
> code, like:

Sorry. I am normally aware of that and keep it out of pure C code as
it is only allowed in C++.

Is ths line ok?

int dlopenMode = RTLD_NOW | RTLD_GLOBAL;

It comes before other declarations but is a constant initialization.

> That was only change I made. Otherwise it seems ok. Could you please
> test the code directly from 1.5 branch if it still woks?

I will and report back here when done.

> If you want to port AIX to trunk I would appreciate that. The code
> changed recently. I made some refactoring. For code specific to
> Linux/Solaris/AIX there is a variable  'is_unix'.

That sounds good. I also remember having seen other variables like
this to avoid constantly duplicated (and possible inconsistent) calls
to 'sys.platform.startswith(<something>)'.

> Do you think you could write some text for the manual? I would welcome
> some info about:
> - what and how to install tools on AIX necessary for pyinstaller to work
> - how to compile bootloader on AIX (in case there is something special
> to be done by users).

I am not sure I will have time this week. I don't think I did anything
special (apart from installing setuptools). But I use GCC for
compiling the bootloader, and installing GCC on AIX can sometimes be a
pain (my colleagues says). I know we have been building GCC from
sources and that is not a simple task.

In any case, I will read the existing manual and see if adding AIX
specific information/instructions make sense.

Cheers,
Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Pá 21. 10. 2011 v 02:17 -0700:

> Is ths line ok?
>
> int dlopenMode = RTLD_NOW | RTLD_GLOBAL;
>
> It comes before other declarations but is a constant initialization.

Well, this line broke compilation on windows. I moved it to unix
specific part. Otherwise it is ok. I fixed it for 1.5.

> For code specific to
> > Linux/Solaris/AIX there is a variable 'is_unix'.
>
> That sounds good. I also remember having seen other variables like
> this to avoid constantly duplicated (and possible inconsistent) calls
> to 'sys.platform.startswith(<something>)'.

There is a bunch of them:

from PyInstaller import is_win, is_darwin, is_unix, is_solar, is_aix

Could you try compile bootloader with xlc? I suppose it is the default
compiler.

Is the /opt/freeware/lib standard path on aix for freeware libs?

[Martin Gamwell Dawids]

> > Is ths line ok?
>
> >     int dlopenMode = RTLD_NOW | RTLD_GLOBAL;
>
> > It comes before other declarations but is a constant initialization.
>
> Well, this line broke compilation on windows. I moved it to unix
> specific part. Otherwise it is ok. I fixed it for 1.5.

Ok, sounds fine.

BTW: Which Windows C compiler do you use?

> There is a bunch of them:
>
> from PyInstaller import is_win, is_darwin, is_unix, is_solar, is_aix

I'm aware of that. :-)

> Could you try compile bootloader with xlc? I suppose it is the default
> compiler.

I am not sure. It is not installed on any of our AIX boxes. And before
trying to install it (if I get the time) I would like to port the AIX
support to trunk.

> Is the /opt/freeware/lib standard path on aix for freeware libs?

I guess so. It is on all our AIX boxes including the one I installed
yesterday (containing e.g. libz.a and libbz2.a).

Earlier today (before your final small fix to the bootloader code), I
checked out the latest code from the 1.5 branch and:
* Compiled bootloader on AIX 6.1
* Ran testsuite (same results, test-zipimport2 failing and some
skipped tests)
* Wrapped a python program as onedir and onefile and ran it on
- AIX 6.1 (build machine)
- AIX 6.1 (freshly installed)
- AIX 5.3
- AIX 5.2
Everything worked fine.

I also tried to build the bootloader on AIX 5.2 and 5.3, but these
machines have Python 2.2 installed, and waf wouldn't run with Python
2.2. So instead I took the bootloader compiled on AIX 6.1 and moved to
AIX 5.2. The I tried to invoke PyInstaller to build a small test
program, but Build.py fails:

Traceback (most recent call last):

File "pyinstaller/Build.py", line 1500, in ?
main(args[0], configfilename=opts.configfile)
File "pyinstaller/Build.py", line 1478, in main
build(specfile)
File "pyinstaller/Build.py", line 1435, in build
execfile(spec)
File "test.spec", line 3, in ?
pathex=['/data2/mgd/pyinstaller-1.5'])
File "pyinstaller/Build.py", line 347, in __init__
self.__postinit__()
File "pyinstaller/Build.py", line 298, in __postinit__
self.assemble()
File "pyinstaller/Build.py", line 472, in assemble
self.fixMissingPythonLib(binaries)
File "pyinstaller/Build.py", line 527, in fixMissingPythonLib
if typ == 'BINARY' and name in fnm:
TypeError: 'in <string>' requires character as left operand

Seems like this line:

525 for (nm, fnm, typ) in binaries:
526 for name in names:
--> 527 if typ == 'BINARY' and name in fnm:
528 # lib found
529 return

is not valid Python 2.2 code. If PyInstaller is still supposed to
support Python 2.2, maybe this should be fixed?

/Martin

[Martin Zibricky]

> BTW: Which Windows C compiler do you use?

MS Visual studio 2008 express.

It should be possible to compile bootloader with mingw but mingw
bootloader is not well tested and the msvcrt library is not statically
linked with the binary as it is with msvc.

Yes, that should be fixed for 1.5.2.

[Martin Gamwell Dawids]

> > Seems like this line:
>
> >     525         for (nm, fnm, typ) in binaries:
> >     526             for name in names:
> > --> 527                 if typ == 'BINARY' and name in fnm:
> >     528                     # lib found
> >     529                     return
>
> > is not valid Python 2.2 code. If PyInstaller is still supposed to
> > support Python 2.2, maybe this should be fixed?
>
> Yes, that should be fixed for 1.5.2.

Working on it.

I tried to change the offending line to:

if typ == 'BINARY' and fnm.find(name) != -1:

and that removed the problem. However, PyInstaller cannot find the
Python library as the 'python' exe is maybe statically linked.
Therefore, 'ldd' does not give information about the Python lib, and
on this machine, the lib was in an entirely different position:

/usr/local/lib/python2.2/config/libpython2.2.a

I'm working on adding an additional parameter to Build.py to specify a
custom search path for the Python lib.

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Pá 21. 10. 2011 v 07:15 -0700:

> > Could you try compile bootloader with xlc? I suppose it is the
> default
> > compiler.
>
> I am not sure. It is not installed on any of our AIX boxes. And before
> trying to install it (if I get the time) I would like to port the AIX
> support to trunk.

What compiler is in general more used in the aix world? Gcc or xlc?
What is supposed to be better?

[Martin Gamwell Dawids]

> What compiler is in general more used in the aix world?  Gcc or xlc?
> What is supposed to be better?

Good question. :-)

I don't know anybody else developing for AIX. I mainly code on Linux,
Windows and in Java and Scala on the JVM.

We always use gcc in order to use the same compiler across all our
*NIX platforms (Linux, Solaris and AIX), and I also use gcc a bit on
my Mac.

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Pá 21. 10. 2011 v 07:55 -0700:

> /usr/local/lib/python2.2/config/libpython2.2.a
>
> I'm working on adding an additional parameter to Build.py to specify a
> custom search path for the Python lib.

what about looking for command 'python-config'?

you could get some info from it like:

python-config --prefix

[Martin Gamwell Dawids]

> what about looking for command 'python-config'?
>
> you could get some info from it like:
>
> python-config --prefix

I cannot find that cmmand on my system.

I got the option working (I know you might not be so fond of an extra
option) and bumped into the next problem. However, it looks like I'm
pretty close to get it working.

/Martin

[Martin Gamwell Dawids]

> I got the option working (I know you might not be so fond of an extra
> option) and bumped into the next problem. However, it looks like I'm
> pretty close to get it working.

Hmm, the problem is that Python 2.2 (or at least my installation) does
not have the 'platform' module. So trying to import it in function
'_bootloader_file' in 'Build.py' fails. Then the code resorts to using
'os.name' which returns 'posix' which cannot really be used to find
the name of the bootloader.

Using a call to 'uname -a' fixed it, so now I can package an
application on my AIX 5.2 box.

But... (there is always a next problem) ...the libpython2.2.a is a
static library!

-bash-3.00$ dist/test/test
Error loading Python lib 'dist/test/
libpython2.2.a(libpython2.2.so)':
0509-026 System error: Cannot run a file that does not have a valid
format.
0509-022 Cannot load module dist/test/
libpython2.2.a(libpython2.2.so).
0509-152 Member libpython2.2.so is not found in archive

Inspecting the library reveals it has _no_ '.so' members (shared
objects) but only '.o' members.

So I guess to make it work, I need to brush off the static linking
code I wrote some time ago.

I think I will call it the day for now, and have a look at it after
the weekend.

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Pá 21. 10. 2011 v 08:42 -0700:

> Inspecting the library reveals it has _no_ '.so' members (shared
> objects) but only '.o' members.
>
> So I guess to make it work, I need to brush off the static linking
> code I wrote some time ago.

You could also look at the wscript file if it needs some changes to tell
gcc link libpython static.

>
> I think I will call it the day for now, and have a look at it after
> the weekend.

Or we could just state:

"Pyinstaller works on AIX 6.1, created executables should work on
5.2/5.3. PyInstaller does not work on 5.x.

[Martin Gamwell Dawids]

> You could also look at the wscript file if it needs some changes to tell
> gcc link libpython static.

I know you did some work. But I will still need my changes in the
bootloader. Otherwise, the bootloader cannot get to the symbols in the
library.

However, that leads to a bigger problem, for as long as I cannot build
the bootloader (as waf requires something more than Python 2.2), I
cannot produce a bootloader that links in a static libpython2.2. (A
catch 22 situation.)

The only other solution I see is to distribute the libpython2.6.a
(shared lib) from the machine where the bootloader is built, along
with the bootloader. But maybe it is too much work to make it work for
Python 2.2 on AIX.

> > I think I will call it the day for now, and have a look at it after
> > the weekend.
>
> Or we could just state:
>
> "Pyinstaller works on AIX 6.1, created executables should work on
> 5.2/5.3. PyInstaller does not work on 5.x.

Something like that, but this is really not a 5.x problem, but rather
a Python 2.2 problem on AIX as:
* libpython2.2.a is static on AIX (at least on my machine)
* I cannot build a bootloader on Python 2.2 (because of waf).

Maybe the option I added could be altered to give the full path to the
dynamic lib I want the bootloader to load.
(I'll look into that on Monday.)

Anyways, the fix mentioned above, changing line 527 in Build.py from:

if typ == 'BINARY' and name in fnm:

to

if typ == 'BINARY' and fnm.find(name) != -1:

should be applied. Otherwise, PyInstaller is potentially broken on
Python 2.2 on any *NIX platform.

/Martin

[Martin Gamwell Dawids]

> The only other solution I see is to distribute the libpython2.6.a
> (shared lib) from the machine where the bootloader is built, along
> with the bootloader. But maybe it is too much work to make it work for
> Python 2.2 on AIX.

Just tried to hack a little. Using a binary editor, I changed the
integer (pyvers) in the bootloader which is used to determine the
python version and thus the name of the libpython to load. Then a
replaced the libpython2.2.a with the libpython2.6.a from the AIX 6.1
machine. Now, the bootloader ran and loaded the python lib, but the
program never wrote anything to stdout (as I would expect).

So I don't think I will spend more time on neither Python 2.2 nor AIX
5.2 (desupported in 2009) nor AIX 5.3 (to be desupported in April
2012).

Couldn't we write something like:

"PyInstaller works on AIX. The boot loader builds with GCC and is
tested with GCC version 4.2.0 on AIX 6.1. Python executables created
using PyInstaller on AIX 6.1 should work on AIX 5.2/5.3. PyInstaller
will not work with statically linked Python libraries which has been
encountered in Python 2.2 installations on AIX 5.x."

Have a nice weekend.

/Martin

[Martin Zibricky]

Martin Gamwell Dawids píše v Pá 21. 10. 2011 v 09:36 -0700:

> Couldn't we write something like:
>
> "PyInstaller works on AIX. The boot loader builds with GCC and is
> tested with GCC version 4.2.0 on AIX 6.1. Python executables created
> using PyInstaller on AIX 6.1 should work on AIX 5.2/5.3. PyInstaller
> will not work with statically linked Python libraries which has been
> encountered in Python 2.2 installations on AIX 5.x."
>
> Have a nice weekend.

That sounds good.

Thanks again for your work.

[Martin Zibricky]

Martin Gamwell Dawids píše v Pá 21. 10. 2011 v 09:01 -0700:

> * I cannot build a bootloader on Python 2.2 (because of waf).

That's the decision we made. We suppose that precompiled bootloader
would be used.

[Martin Zibricky]

Martin Gamwell Dawids píše v Pá 21. 10. 2011 v 09:01 -0700:

>
> > You could also look at the wscript file if it needs some changes to
> tell
> > gcc link libpython static.
>
> I know you did some work. But I will still need my changes in the
> bootloader. Otherwise, the bootloader cannot get to the symbols in the
> library.

I only added an option to wscript to enable symbols for static linking.

But there seems to be more to do. We need to tell gcc to link static
version of python library. I was trying to do that but without success.
Or maybe i didn't use correctly python with static lib.

[Martin Zibricky]

Martin Gamwell Dawids píše v Pá 21. 10. 2011 v 09:01 -0700:

> Anyways, the fix mentioned above, changing line 527 in Build.py from:
>
> if typ == 'BINARY' and name in fnm:
>
> to
>
> if typ == 'BINARY' and fnm.find(name) != -1:
>
> should be applied. Otherwise, PyInstaller is potentially broken on
> Python 2.2 on any *NIX platform.

thanks, fix commited.

[Martin Gamwell Dawids]

Yes, you probably needs a static version of libpython, but apart from
that, I believe I have the code in place. (The patch attached to
ticket #420 is not complete, but I think I have the bits in place.)

I will try to update the incomplete patch on ticket #420 later this
week, if you are interested.

/Martin

[Martin Gamwell Dawids]

On Oct 20, 9:43 pm, Martin Gamwell Dawids wrote:
> I just attached the patch to the ticket:http://www.pyinstaller.org/attachment/ticket/343/pyinstaller-1.5-rev1...

I just discovered that there is a flaw in my patch in file
'bindepend.py'. The incorrect patch hunk looks like this:

@@ -741,6 +770,8 @@
`name`must include the prefix, e.g. ``libpython2.4.so``
"""
- assert sys.platform == 'linux2' or
sys.platform.startswith('sunos'), \
- "Current implementation for Linux and Solaris (sunos) only"
+ "Current implementation for Linux, Solaris (sunos) and AIX only"
+ assert (sys.platform == 'linux2' or
+ sys.platform.startswith('sunos') or
+ sys.platform.startswith('aix'))

lib = None

I guess I somehow messed up the assert. The resulting lines should
look like this:

assert (sys.platform == 'linux2' or
sys.platform.startswith('sunos') or
sys.platform.startswith('aix')), \
"Current implementation for Linux, Solaris (sunos) and AIX only"

Could you fix this in branches/1.5?

Regards,
Martin

[Martin Gamwell Dawids]

Hi Martin,

> Let me know if should link libz statically with loader for aix. It

> should be trivial to make such changes to wscript.

Funny that this first struck me 3 years later. :-)

I have been trying to modify wscript like this:

        if sys.platform.startswith('freebsd'):

            ...

        elif sys.platform.startswith('aix'):

            libs = ['dl', 'm']

            staticlibs = ['z']

        else:

            libs = ['dl', 'z', 'm']  # 'z' - zlib, 'm' - math,

        for key, value in targets.items():

            bld(

                features = 'cc cprogram',

                source = bld.path.ant_glob('linux/*.c common/*.c'),

                target = value,

                install_path = install_path,

                lib = libs,

                staticlib = staticlibs,   <========== NEW LINE

                env = bld.env_of_name(key).copy(),

                includes = ['common', 'linux'],

            )

However, unfortunately it does not provide the correct GCC options:

/usr/bin/gcc debugw/linux/getpath_4.o ...  -o .../bootloader/build/debugw/runw_d -Wl,-brtl -Wl,-Bstatic -lz -ldl -lm

This gives me this line:

ld: 0706-027 The -B static flag is ignored.

Any ideas?