ImportError: ........__intel_security_cookie

[Alex van Houten]

Hi,

Tried to compile cython modules using icc on Windows OS, see my previous post,
to investigate speed benefits. I figured that this investigation might be
slightly easier on a Linux OS, because it would only require setting the
CC="icc" environment variable, allegedly, while still using distutils.
So I installed the Intel compiler on my Fedora 14 system and compilation of my
module dose.pyx was successfull, albeit some warnings:
"
python setup.py build_ext --inplace
running build_ext
cythoning dose.pyx to dose.c
building 'dose' extension
icc -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE
-fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE
-fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c dose.c -o
build/temp.linux-x86_64-2.7/dose.o -O3
icc: command line warning #10156: ignoring option '-fp'; no argument required
icc: command line warning #10006: ignoring unknown option '-fwrapv'
icc: command line warning #10156: ignoring option '-fp'; no argument required
icc: command line warning #10006: ignoring unknown option '-fwrapv'
icc: command line warning #10120: overriding '-O2' with '-O3'
gcc -pthread -shared build/temp.linux-x86_64-2.7/dose.o -L/usr/lib64 -lpython2.7
-o some_path/dose.so
"

But when I ran the python program that calls dose.so I ran into this error:
"
Traceback (most recent call last):
File "some_python_script.py", line 6, in <module>
import dose
ImportError: some_path/dose.so: undefined symbol: __intel_security_cookie
"

Any clue?

Thnx,
Alex

[Robert Bradshaw]

Sounds like you forgot to link a library in, but I've never used icc
myself so I don't know. I'd try asking on an ICC forum.

- Robert

[Alex van Houten]

Robert Bradshaw <robertwb <at> math.washington.edu> writes:

>
> Sounds like you forgot to link a library in, but I've never used icc
> myself so I don't know. I'd try asking on an ICC forum.
>

The shared library is built using gcc, not icc,
could that have raised this error?

Cheers,
Alex

[Lisandro Dalcin]

Try this:

CC=icc LDSHARED="icc -shared" python setup.py build

--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

[Alex van Houten]

Lisandro Dalcin <dalcinl <at> gmail.com> writes:

> Try this:
>
> CC=icc LDSHARED="icc -shared" python setup.py build
>

Thanks, it got me one step further:

python some_python_script.py

Traceback (most recent call last):
File "some_python_script.py", line 6, in <module>
import dose

ImportError: libimf.so: cannot open shared object file:
No such file or directory

Cheers,
Alex

[Alex van Houten]

Alex van Houten <sparrow2867 <at> yahoo.com> writes:

> python some_python_script.py
> Traceback (most recent call last):
> File "some_python_script.py", line 6, in <module>
> import dose
> ImportError: libimf.so: cannot open shared object file:
> No such file or directory
>

This link might help:
https://www.nbcr.net/pub/wiki
/index.php?title=MyMPI_Setup#The_libimf.so_Error_Message

Alex

[Alex van Houten]

Yep. Solved.
export LD_LIBRARY_PATH=path_to_libimf.so
And icc compilation worked!!!
The python script that loads the module ran flawlessly.

Conclusion: No speed benefits from icc, but not any slower either.
Contrary to the common conclusion that icc should be able to speed things up,
sometimes by
a factor ~2.
(I have an Intel Core I7 920 processor)

Some side notes.
icc ignored the "-fp" and "-fwrapv" options.
Not sure what this meant.

When compiling with gcc I get:
"gcc -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2

-fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
-D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
-D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c dose.c -o
build/temp.linux-x86_64-2.7/dose.o -O3"

Apparently it uses the suboptimal -O2 by default, I however added
extra_compile_args=["-O3"] to setup.py.
But when I remove this option from setup.py, there is no slowing down.
Weird.
If I just compile "manually" using
cython dose.pyx
gcc -fno-strict-aliasing -O3 ....
gcc -shared ....
the speed is still the same.

Cheers,
Alex.

[Alex van Houten]

Alex van Houten <sparrow2867 <at> yahoo.com> writes:

> Conclusion: No speed benefits from icc, but not any slower either.
> Contrary to the common conclusion that icc should be able to speed things up,
> sometimes by
> a factor ~2.
> (I have an Intel Core I7 920 processor)
>
> Some side notes.
> icc ignored the "-fp" and "-fwrapv" options.
> Not sure what this meant.
>

I googled
intel compiler optimization
and tried the options from the
[PDF]
Quick-Reference Guide to Optimization with Intel® Compilers
but that did not give any speed benefits beyond the 2% level.
---> gcc is competitive in this case.

Cheers,
Alex

[Lisandro Dalcin]

Can you try to hack things and remove the -fno-strict-aliasing flag?
However, note that things could get easily broken without that flag
(unless you run Python 3)

Also, not sure about this (after all you are building with -m64), but
perhaps using -mtune=native instead of -mtune=generic could make any
difference?

Finally, remember that the GIL could prevent you for running
thread-based (or OpenMP) compiler auto-generated parallelism. Do you
have your code online to take a look?

[Alex van Houten]

Lisandro Dalcin <dalcinl <at> gmail.com> writes:

> Can you try to hack things and remove the -fno-strict-aliasing flag?
> However, note that things could get easily broken without that flag
> (unless you run Python 3)
>
> Also, not sure about this (after all you are building with -m64), but
> perhaps using -mtune=native instead of -mtune=generic could make any
> difference?
>
> Finally, remember that the GIL could prevent you for running
> thread-based (or OpenMP) compiler auto-generated parallelism. Do you
> have your code online to take a look?
>

Ok.
I removed that flag and used -mtune=native:
icc -O3 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=native -D_GNU_SOURCE -fPIC -fwrapv
-DNDEBUG -O3 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=native -D_GNU_SOURCE

-fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c dose.c -o
build/temp.linux-x86_64-2.7/dose.o

icc: command line warning #10156: ignoring option '-fp'; no argument required

icc: command line warning #10159: invalid argument for option '-m'

icc: command line warning #10006: ignoring unknown option '-fwrapv'
icc: command line warning #10156: ignoring option '-fp'; no argument required

icc: command line warning #10159: invalid argument for option '-m'

icc: command line warning #10006: ignoring unknown option '-fwrapv'

icc -shared build/temp.linux-x86_64-2.7/dose.o -L/usr/lib64 -lpython2.7 -o
/some_path/dose.so

No speedup. Not slower either.
Same with -mtune=generic.
The heavy computations are "with nogil".
Cheers,
Alex.