OS X import bug?
Manoj Plakal
Hello,
I've just noticed a weird problem with importing modules
on MacOS X 10.1 and Python 2.2.
Say I have two modules foo and bar, both of which
are built with a common file baz.c with some variables
defined in it.
I make a setup.py which specifies that module foo
is built from foomodule.c and baz.c, and module bar
is built from barmodule.c and baz.c.
On Red Hat Linux 7.2 with Python 2.2, this builds fine and I
can import both foo and bar into the same interpreter session.
On OS X 10.1 with Python 2.2, this builds fine but I can
only import *ONE* of foo or bar into the same interpreter
session. E.g., if I import foo first and then bar,
Python craps out saying "Failure linking new module".
Similary for bar first and then foo.
This is both with a version built from the 2.2 source
release, as well as a binary version from Fink
(fink.sourceforge.net).
I've attached the files which can be used to reproduce
this: setup.py, foomodule.c, barmodule.c, baz.c. In
case the attachments don't work, you can grab them
off the web: http://yumpee.org/python/setup.py
http://yumpee.org/python/foomodule.c
http://yumpee.org/python/barmodule.c
http://yumpee.org/python/baz.c
Is this a bug? Undefined behavior? Feature?
Is it a fluke that it works on Linux?
If someone else with OS X can reproduce this error,
(and people on other OSes can confirm that
it's working), that would be great ...
Manoj
Martin von Loewis
> On OS X 10.1 with Python 2.2, this builds fine but I can
> only import *ONE* of foo or bar into the same interpreter
> session. E.g., if I import foo first and then bar,
> Python craps out saying "Failure linking new module".
> Similary for bar first and then foo.
I believe on OS X 10.1, you need to link with -flat_namespace
> Is this a bug? Undefined behavior? Feature?
> Is it a fluke that it works on Linux?
I'd say it is a feature that it works on Linux, since it allows
independently-developed shared libraries to have symbols in common.
From a viewpoint of the C programming language, it is an error to have
multiple definitions of the same thing. The question here is what
exactly constitutes a program: a common view is that a
dynamically-loaded shared library is not part of the program, and thus
does not fall under the one-definition-rule. Apparently, Apple does
not share this view.
Regards,
Martin
Manoj Plakal
> Manoj Plakal <tera...@yumpee.org> writes:
>>On OS X 10.1 with Python 2.2, this builds fine but I can
>>only import *ONE* of foo or bar into the same interpreter
>>session. E.g., if I import foo first and then bar,
>>Python craps out saying "Failure linking new module".
>>Similary for bar first and then foo.
>>
>
> I believe on OS X 10.1, you need to link with -flat_namespace
>
Python distutils does use "-flat_namespace"
in addition to other options. Here's a complete log
of a build and import of the code I sent earlier.
Manoj
eve(430):test% python setup.py build
running build
running build_ext
building 'foo' extension
cc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -no-cpp-precomp -fno-common -dynamic
-I/Library/Frameworks/Python.framework/Versions/2.2/include/python2.2 -c foomodule.c -o
build/temp.darwin-1.4-Power Macintosh-2.2/foomodule.o
foomodule.c:26: warning: function declaration isn't a prototype
cc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -no-cpp-precomp -fno-common -dynamic
-I/Library/Frameworks/Python.framework/Versions/2.2/include/python2.2 -c baz.c -o
build/temp.darwin-1.4-Power Macintosh-2.2/baz.o
cc -Wl,-F. -Wl,-flat_namespace,-U,_environ -bundle -framework Python build/temp.darwin-1.4-Power
Macintosh-2.2/foomodule.o build/temp.darwin-1.4-Power Macintosh-2.2/baz.o -o
build/lib.darwin-1.4-Power Macintosh-2.2/foo.so
building 'bar' extension
cc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -no-cpp-precomp -fno-common -dynamic
-I/Library/Frameworks/Python.framework/Versions/2.2/include/python2.2 -c barmodule.c -o
build/temp.darwin-1.4-Power Macintosh-2.2/barmodule.o
barmodule.c:26: warning: function declaration isn't a prototype
skipping baz.c (build/temp.darwin-1.4-Power Macintosh-2.2/baz.o up-to-date)
cc -Wl,-F. -Wl,-flat_namespace,-U,_environ -bundle -framework Python build/temp.darwin-1.4-Power
Macintosh-2.2/barmodule.o build/temp.darwin-1.4-Power Macintosh-2.2/baz.o -o
build/lib.darwin-1.4-Power Macintosh-2.2/bar.so
eve(431):test% python
Python 2.2+ (#1, Jan 17 2002, 10:33:30)
[GCC 2.95.2 19991024 (release)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import foo
var_for_foo = 1
>>> import bar
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ImportError: Failure linking new module
>>>
--EOF
Martin von Loewis
> Python distutils does use "-flat_namespace"
> in addition to other options. Here's a complete log
> of a build and import of the code I sent earlier.
[...]
> ImportError: Failure linking new module
I see. That appears to originate from dynload_next.c, without any
further analysis of the error. It would be valuable to let the OS
explain what it thinks went wrong. It appears that Apple found that,
as well, and introduced NSLinkEditError in OS X DR2, see
http://developer.apple.com/techpubs/macosx/ReleaseNotes/CompilerTools.html
So I'd suggest that you add appropriate calls to that function, to get
a more informative message than "failure linking new module". Once you
are done, please post your code to SF (and perhaps follow-up here to
tell us what you found).
Good luck,
Martin
Steven Majewski
On Thu, 24 Jan 2002, Manoj Plakal wrote:
> Python 2.2+ (#1, Jan 17 2002, 10:33:30)
> [GCC 2.95.2 19991024 (release)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import foo
> var_for_foo = 1
> >>> import bar
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> ImportError: Failure linking new module
> >>>
I believe whenever I had a problem with duplicate symbols, the error
message actually said something about duplicate symbols.
The 'Failure linking new module' is the error I got from the
the 'PyObject_DelItemString' problem:
http://sourceforge.net/tracker/?group_id=5470&atid=105470&func=detail&aid=498915
(But the symptom of working for one import but not the other doesn't
sound right.)
This bug was in 2.2. I don't know if it's fixed in the 2.2+ you seem
to be running, but it's worth checking.
Below is a patch that adds a call to NSLinkEditError, which will
give you a more explicit error message about what's wrong. It's
how I discovered which symbol was the problem with that bug:
>>> import pyobjc
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ImportError: dyld: /usr/local/src/Python-2.2/python.exe Undefined symbols:
_PyObject_DelItemString
Failure linking new module
>>>
-- Steve
*** dynload_next.c.0 Fri Jan 18 13:50:06 2002
--- dynload_next.c Fri Jan 18 14:08:13 2002
***************
*** 150,157 ****
if (errString == NULL) {
newModule = NSLinkModule(image, pathname,
NSLINKMODULE_OPTION_BINDNOW|NSLINKMODULE_OPTION_RETURN_ON_ERROR);
! if (!newModule)
! errString = "Failure linking new module";
}
if (errString != NULL) {
PyErr_SetString(PyExc_ImportError, errString);
--- 150,164 ----
if (errString == NULL) {
newModule = NSLinkModule(image, pathname,
NSLINKMODULE_OPTION_BINDNOW|NSLINKMODULE_OPTION_RETURN_ON_ERROR);
! if (!newModule) { // sdm7g
! int errNo;
! char *fileName, *moreErrorStr;
! NSLinkEditErrors c;
! errString = "Failure linking new module";
! NSLinkEditError( &c, &errNo, &fileName, &moreErrorStr );
! errString = strcat( fileName, errString );
! errString = strcat( moreErrorStr, errString );
! } // sdm7g
}
if (errString != NULL) {
PyErr_SetString(PyExc_ImportError, errString);
Manoj Plakal
Steven,
Thanks for the patch. I applied it to the
2.2 tree that I had checked out from CVS
and re-built foo and bar and tried
importing them again.
eve(1418):test% python
Python 2.2+ (#2, Jan 24 2002, 20:36:33)
[GCC 2.95.2 19991024 (release)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import foo
var_for_foo = 1
>>> import bar
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ImportError: dyld: python multiple definitions of symbol _var_for_bar
foo.so definition of _var_for_bar
bar.so definition of _var_for_bar
Failure linking new module
>>>
var_for_foo and var_for_bar are defined in baz.c which
is included in both modules foo and bar. The modules
were compiled with a simple distutils setup.py file
(which I had included in my earlier post) which
builds the two modules with -flat_namespace.
The same source code works fine on Red Hat Linux 7.2
with Python 2.1 and Win98 with Python 2.2b2.
So my question now is: is this really a bug
in the OS X implementation, or is this how
Python is supposed to work (and the Linux
and Win32 dynamic loading implementations
just happen to allow this without errors)?
If I am to file a bug at Sourceforge, I'd
like to know exactly what the bug is :)
Manoj
Steven Majewski wrote:
>
> Below is a patch that adds a call to NSLinkEditError, which will
> give you a more explicit error message about what's wrong. It's
> how I discovered which symbol was the problem with that bug:
>
>
Manoj Plakal
Is there some OS X linker option one can give
while building the extension such that
the symbols in baz.c are exposed to
foomodule.c while building module foo,
but these baz symbols are not exposed
outside module foo? Would the
"two-level namespace" as opposed
to the flat namespace be of use here?
Manoj
Martin von Loewis
> Is there some OS X linker option one can give
> while building the extension such that
> the symbols in baz.c are exposed to
> foomodule.c while building module foo,
> but these baz symbols are not exposed
> outside module foo? Would the
> "two-level namespace" as opposed
> to the flat namespace be of use here?
Might be. On Linux (and other SysV systems), Python uses RTLD_LOCAL
when loading extensions. This means essentially that one extension
cannot see any symbols of a different extension. This is by design:
Python extensions ought to have separate namespaces.
I'm not even sure why flat_namespace is used on OS X. I understood it
to avoid the problem of duplicate symbols, whereas it appears that it
is precisely *not* the intent of this approach, and that two-level
namespaces are meant to be used for applications like Python.
On Windows, there are only two-level namespaces; each symbol is
referenced by DLL name and symbol name. Unfortunately, since the main
application does not have a DLL name, you need a pythonXY.dll to
provide the core Python symbols to each extension.
Regards,
Martin
Pete Shinners
fixed this himself, but i don't believe yet submitted anything. in any
event i'm told it's a quick-fix, but here's the problem for you OSX
maintainers...
in my pygame package i have a "time" module. this is an extension
library. this has not been a problem from python-1.5.2 and up on every
different platform. users can easily choose between the stdlib time and
the pygame time
import time #std
import pygame.time #mine
problem is, on OSX the command "import time" is actually importing my
time module, from inside my package (i'm not using PTH files or anything
either).
of course, my time module doesn't have the same functions as the
standard one. so when standard modules (like random) say "import time"
they are getting the wrong module and raise a quick exception.
i really hope this can get fixed for python-2.2.1. :]
Steven Majewski
On Fri, 25 Jan 2002, Pete Shinners wrote:
> there's more bugs in the OSX module loader. one of my OSX porters has
> fixed this himself, but i don't believe yet submitted anything. in any
> event i'm told it's a quick-fix, but here's the problem for you OSX
> maintainers...
>
> in my pygame package i have a "time" module. this is an extension
> library. this has not been a problem from python-1.5.2 and up on every
> different platform. users can easily choose between the stdlib time and
> the pygame time
>
> import time #std
> import pygame.time #mine
>
> problem is, on OSX the command "import time" is actually importing my
> time module, from inside my package (i'm not using PTH files or anything
> either).
Actually, the behaviour I see when I try to reproduce this is that
whichever module you import first get's used for both, so in the
example above, pygame.time would refer to the standard time module;
swapping the order of the imports gives me the state you describe,
where both refer to your timemodule. Is that what you see ?
The -v seems to show that it's finding the right module.
I think what's happening is that once 'inittime' is bound, it
stays bound. (I'm surprised that there's no error message, though!)
I tried swapping the calls to NSLookupAndBindSymbol() to calls to
NSLookupSymbolInModule() [ which is what the dlcompat libs seem to
use instead ] but it didn't change the behaviour:
#define NSLookupAndBindSymbol( f ) NSLookupSymbolInModule( newmodule, f )
> of course, my time module doesn't have the same functions as the
> standard one. so when standard modules (like random) say "import time"
> they are getting the wrong module and raise a quick exception.
>
> i really hope this can get fixed for python-2.2.1. :]
>
But I'll take a closer look at this.
-- Steve