mpi4py profiling

[sven...@xlab.si]

Hi

I am trying to profile an application developed with mpi4py.

In the mpi4py repository there is a directory 'demo/vampirtrace'.
When I run:
  $ make run
in that directory all the programs (cpilog.py, ring.py and threads.py) run without errors.
However, there are no trace files generated.

How do I get the trace files?

I am running this on my desktop machine under Arch Linux with the MPICH mpi implementation,
and mpi4py installed using pip.

[Lisandro Dalcin]

You need a VampirTrace built against your MPI, and the `vtcc` compiler
wrapper in your $PATH (or export
VTCC="/path/to/vampirtrace/bin/vtcc"), and then you install mpi4py
(clear your pip cache first, otherwise you will get the old build
installed).

> --
> You received this message because you are subscribed to the Google Groups
> "mpi4py" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mpi4py+un...@googlegroups.com.
> To post to this group, send email to mpi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/mpi4py.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mpi4py/cd6b639f-654a-4a77-a2df-c6506c328d38%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 0109
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

[sven...@xlab.si]

I have done everything as described and still do not get any trace files.

I have `vtcc` in my $PATH and have also tried setting $VTCC, but it did not help.
`vtcc` works for a simple C MPI application and there I get the trace files.

For mpi4py the installation with pip finishes without warnings and when I run
  $ mpiexec -n 4 python ring.py
in the demo/vampirtrace/ directory
the program executes normally, but there are no trace files generated.

I have installed mpi4py in a python venv, might this be the problem?

[Lisandro Dalcin]

On 28 March 2018 at 09:21, <sven...@xlab.si> wrote:
> I have done everything as described and still do not get any trace files.
>
> I have `vtcc` in my $PATH and have also tried setting $VTCC, but it did not
> help.
> `vtcc` works for a simple C MPI application and there I get the trace files.
>
> For mpi4py the installation with pip finishes without warnings and when I
> run
> $ mpiexec -n 4 python ring.py
> in the demo/vampirtrace/ directory
> the program executes normally, but there are no trace files generated.
>

Oh, then maybe there is a build issue with some recent VampirTrace
version, I have not tried since for long time ago.

What's the VampirTrace version you are using?


At this point, I think it is best if you build mpi4py by downloading
the source tarball, or maybe a git repo clone.
Then, could you please send me the output of these commands:

$ rm -rf build/
$ python setup.py build_clib

>
> I have installed mpi4py in a python venv, might this be the problem?
>

Unlikely, given the way profiling works (dlopen of a shared library)
in mpi4py. Did you clear the pip cache before installing?

[sven...@xlab.si]

I am using VampirTrace 5.14.4 (latest from https://tu-dresden.de/zih/forschung/projekte/vampirtrace)

I have now built and installed mpi4py by cloning it from bitbucket.
The trace files are still not generated.

The output of

  $ rm -rf build/
  $ python setup.py build_clib

is attached in build_clib.out

[Lisandro Dalcin]

I just tried building and installing latest VampirTrace from sources,
then build and install mpi4py from a git clone (branch maint), and
everything works as expected on my side.

[Lisandro Dalcin]

On 28 March 2018 at 10:52, <sven...@xlab.si> wrote:
> I am using VampirTrace 5.14.4 (latest from
> https://tu-dresden.de/zih/forschung/projekte/vampirtrace)
>

Same one I used.

> I have now built and installed mpi4py by cloning it from bitbucket.
> The trace files are still not generated.
>

Damn, not sure what's going on in your side.

> The output of
> $ rm -rf build/
> $ python setup.py build_clib
> is attached in build_clib.out
>

The build log looks just fine, the VT libs are found and things seems
to build correctly.

In the mpi4py source directory, what's the output of the following?

$ ldd build/lib.linux-x86_64-3.6/mpi4py/lib-pmpi/libvt.so
$ ldd build/lib.linux-x86_64-3.6/mpi4py/lib-pmpi/libvt-mpi.so

[justin....@gmail.com]

I tried the same build steps as Sven, on ubuntu16.04, and also did not get the traces. mpi4py installed with pip (I will try from source afterwards):

root@ff2-mpich-vt-test:~/devel/mpi4py/demo/vampirtrace# ll `find / -iname libvt.so.0`
lrwxrwxrwx 1 root root 14 Mar 28 07:50 /root/devel/VampirTrace-5.14.4/vtlib/.libs/libvt.so.0 -> libvt.so.0.0.0*
lrwxrwxrwx 1 root root 14 Mar 28 07:51 /usr/local/lib/libvt.so.0 -> libvt.so.0.0.0*

find / -iname libvt.so
/root/devel/VampirTrace-5.14.4/vtlib/.libs/libvt.so
/root/devel/virtualenvs/venv-mpi4py/lib/python3.5/site-packages/mpi4py/lib-pmpi/libvt.so
/usr/local/lib/libvt.so
root@ff2-mpich-vt-test:~/devel/mpi4py/demo/vampirtrace# ldd /root/devel/virtualenvs/venv-mpi4py/lib/python3.5/site-packages/mpi4py/lib-pmpi/libvt.so
    linux-vdso.so.1 =>  (0x00007ffe8c1e7000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f67fa255000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f67f9e8b000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f67fa674000)
root@ff2-mpich-vt-test:~/devel/mpi4py/demo/vampirtrace# ldd /root/devel/virtualenvs/venv-mpi4py/lib/python3.5/site-packages/mpi4py/lib-pmpi/libvt-mpi.so
    linux-vdso.so.1 =>  (0x00007fff2f9e2000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7d4a6c3000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7d4a2f9000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f7d4aae2000)

libopen-trace-format.so.1 or libotfaux.so.0 are no referred. With C hello-world, I had to "LD_LIBRARY_PATH=/usr/local/lib/ mpiexec -n 1 ./a.out" to get libvt-mpi.so.0.

[Lisandro Dalcin]

No, this is not right, the mpi4py/lib-pmpi/libvt-mpi.so shared object
should be linked to the VT libraries.

The mpi4py/lib-pmpi/*.so are basically dummy libraries, they are there
just to pull in all the VT and MPI libraries in advance when you call
`mpi4py.profile()`, this call just dlopen() the mpi4py/lib-pmpi/*.so
DSOs to dynamically inject the MPI profiling interface, this way you
do not have to recompile mpi4py to get traces.

Please try to build from sources, attach the build log, and run ldd on
the mpi4py/lib-pmpi/*.so DSOs within the build/ directory.

> libopen-trace-format.so.1 or libotfaux.so.0 are no referred. With C
> hello-world, I had to "LD_LIBRARY_PATH=/usr/local/lib/ mpiexec -n 1 ./a.out"
> to get libvt-mpi.so.0.
>

If mpi4py is properly built, you should not need LD_LIBRARY_PATH

[justin....@gmail.com]

(venv-mpi4py) root@ff2-mpich-vt-test:~/devel/mpi4py#
pip uninstall mpi4py
rm build -fr
python setup.py build_clib | tee build_clib.log
(venv-mpi4py) root@ff2-mpich-vt-test:~/devel/mpi4py# ldd build/lib.linux-x86_64-3.5/mpi4py/lib-pmpi/libvt-mpi.so
    linux-vdso.so.1 =>  (0x00007ffdd0381000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4e80d9e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4e809d4000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f4e811bd000)
(venv-mpi4py) root@ff2-mpich-vt-test:~/devel/mpi4py#

The a.out was a plain C example.

[sven...@xlab.si]

I have found the issue.

When looking at the output of
  python setup.py build_clib
I found the line that links libvf-mpi.so (line 38 in build_clib.out).

Among the flags passed to the linker there is the
  --as-needed
flag (present twice in the same command).

The --as-needed flag causes the linker to link only needed libraries.
Since src/vt-mpi.c is a dummy file, the VampirTrace libraries are not needed.

I then built libvt-mpi.so manually and removed the --as-needed flags.
Output of ldd on the resulting libvt-mpi.so is:
    linux-vdso.so.1 (0x00007ffd2a7cd000)
    libvt-mpi.so.0 => /usr/local/lib/libvt-mpi.so.0 (0x00007f15d13a6000)
    libmpi.so.12 => /opt/mpich/lib/libmpi.so.12 (0x00007f15d0f07000)
    libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f15d0ce9000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007f15d0932000)
    libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f15d072e000)
    libvt-mpi-unify.so.0 => /usr/local/lib/libvt-mpi-unify.so.0 (0x00007f15d04a6000)
    libotfaux.so.0 => /usr/local/lib/libotfaux.so.0 (0x00007f15d029e000)
    libopen-trace-format.so.1 => /usr/local/lib/libopen-trace-format.so.1 (0x00007f15d0057000)
    libz.so.1 => /usr/lib/libz.so.1 (0x00007f15cfe40000)
    librt.so.1 => /usr/lib/librt.so.1 (0x00007f15cfc38000)
    libgfortran.so.4 => /usr/lib/libgfortran.so.4 (0x00007f15cf85c000)
    libm.so.6 => /usr/lib/libm.so.6 (0x00007f15cf510000)
    libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f15cf2f9000)
    libquadmath.so.0 => /usr/lib/libquadmath.so.0 (0x00007f15cf0b9000)
    /usr/lib64/ld-linux-x86-64.so.2 (0x00007f15d1832000)
    libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f15ced32000)

I overwrote libvt-mpi.so in the install directory with the manually built one.

Now the demo/vampirtrace/ring.py example works and generates the tracefiles.

I don't know where the --as-needed flags come from. Is this added by Python distutils or by gcc?
I am using Python 3.6.4 and gcc 7.3.1.
I will try to figure it out and prepare a patch.

[Lisandro Dalcin]

On 28 March 2018 at 12:51, <justin....@gmail.com> wrote:
> (venv-mpi4py) root@ff2-mpich-vt-test:~/devel/mpi4py#
> pip uninstall mpi4py
> rm build -fr
> python setup.py build_clib | tee build_clib.log
> (venv-mpi4py) root@ff2-mpich-vt-test:~/devel/mpi4py# ldd
> build/lib.linux-x86_64-3.5/mpi4py/lib-pmpi/libvt-mpi.so
> linux-vdso.so.1 => (0x00007ffdd0381000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007f4e80d9e000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4e809d4000)
> /lib64/ld-linux-x86-64.so.2 (0x00007f4e811bd000)
> (venv-mpi4py) root@ff2-mpich-vt-test:~/devel/mpi4py#
>

OK, there are two issues here:

1) For some reason, the linker is not adding the VT libs, I guess that
somehow the linker is using by default the --as-needed flag. This
should be relatively easy to fix with `export
LDFLAGS=-Wl,--no-as-needed`

2) The option -Bsymbolic-functions will may prevent the dynamic
injection of the MPI profiling library at runtime.

So, in short, this should work:

export LDFLAGS="-Wl,--no-as-needed -Bno-symbolic-functions"
rm -rf build
python setup.py install

[Lisandro Dalcin]

On 28 March 2018 at 15:04, <sven...@xlab.si> wrote:
> I have found the issue.
>
> When looking at the output of
> python setup.py build_clib
> I found the line that links libvf-mpi.so (line 38 in build_clib.out).
>
> Among the flags passed to the linker there is the
> --as-needed
> flag (present twice in the same command).
>
> The --as-needed flag causes the linker to link only needed libraries.
> Since src/vt-mpi.c is a dummy file, the VampirTrace libraries are not
> needed.
>

You nailed it indeed.

>
> I don't know where the --as-needed flags come from. Is this added by Python
> distutils or by gcc?
> I am using Python 3.6.4 and gcc 7.3.1.

Not sure.

>
> I will try to figure it out and prepare a patch.
>

Can you confirm the -Bsymbolic-functions flag is not preventing things
from working as expected?

[sven...@xlab.si]

When I run
  export LDFLAGS='-Bno-symbolic-functions'
  rm -rf build
  python setup.py build

The needed libraries are still not linked.
It seems --as-needed really is the culprit.

[Lisandro Dalcin]

On 28 March 2018 at 16:30, <sven...@xlab.si> wrote:
> When I run
> export LDFLAGS='-Bno-symbolic-functions'
> rm -rf build
> python setup.py build
>
> The needed libraries are still not linked.
> It seems --as-needed really is the culprit.
>

Yes, for sure you need --no-as-needed for building mpi4py's VT dummy libraries.

But -Bno-symbolic-functions may be needed for the MPI.so extension module.

I think the -Bsymbolic flag does not affect MPI profiling but I'm not
sure, so better to double check.

[Yury V. Zaytsev]

On Wed, 28 Mar 2018, Lisandro Dalcin wrote:

>> I don't know where the --as-needed flags come from. Is this added by Python
>> distutils or by gcc?
>> I am using Python 3.6.4 and gcc 7.3.1.
>
> Not sure.

If I remember correctly, this flag was added to the Ubuntu toolchain from
Ubuntu 14.04 on or somewhere around that time. The motivation was to
dramatically improve application startup times by not linking libraries
from which no symbols are used.

I'm not sure whether the decision was really backed by benchmarks, but the
consensus was that it is really worth it. Anyways, it did cause many
problems down the line for pathological cases where for example plugins
were backlinked to the library that was supposed to load them to be able
to access its symbols instead of defining a proper API, etc. or weird
cases like yours, when the library was actually meant to be preloaded and
authors were relying on dynamic linking instead of LD_PRELOAD.

Of course, from there on, it propagates to the rest of the system. If one
uses system Python form Debian package, then distutils pick up the
compiler and linker flags with which the package was compiled...

--
Sincerely yours,
Yury V. Zaytsev

[Lisandro Dalcin]

On 28 March 2018 at 16:44, Yury V. Zaytsev <yu...@shurup.com> wrote:
> On Wed, 28 Mar 2018, Lisandro Dalcin wrote:
>
>>> I don't know where the --as-needed flags come from. Is this added by
>>> Python
>>> distutils or by gcc?
>>> I am using Python 3.6.4 and gcc 7.3.1.
>>
>>
>> Not sure.
>
>
> If I remember correctly, this flag was added to the Ubuntu toolchain from
> Ubuntu 14.04 on or somewhere around that time. The motivation was to
> dramatically improve application startup times by not linking libraries from
> which no symbols are used.
>

I agree with the rationale.

What I don't like is that you don't get at least a warning for you to notice.

> Of course, from there on, it propagates to the rest of the system. If one
> uses system Python form Debian package, then distutils pick up the compiler
> and linker flags with which the package was compiled...
>

Sure, I think I'll just add a fix in mpi4py's build system for Linux.
I should look at macOS as well.

[sven...@xlab.si]

Oh, I misunderstood because the -Bsymbolic-functions flag was never present on my system.
It is present in Justin's build_clib.log.

Now I ran
  export LDFLAGS='-Wl,--no-as-needed -Bsymbolic-functions'

  rm -rf build
  python setup.py build

  python setup.py install
and everything works as expected (demo/vampirtrace/ring.py generates the trace files).

Is this what you wanted me to test?

[Lisandro Dalcin]

On 28 March 2018 at 16:53, <sven...@xlab.si> wrote:
> Oh, I misunderstood because the -Bsymbolic-functions flag was never present
> on my system.
> It is present in Justin's build_clib.log.
>
> Now I ran
> export LDFLAGS='-Wl,--no-as-needed -Bsymbolic-functions'
> rm -rf build
> python setup.py build
> python setup.py install
> and everything works as expected (demo/vampirtrace/ring.py generates the
> trace files).
>
> Is this what you wanted me to test?
>

Yes, good to know it works. Now I'll start working on a fix for this issue.

[Lisandro Dalcin]

On 28 March 2018 at 17:27, Lisandro Dalcin <dal...@gmail.com> wrote:
> On 28 March 2018 at 16:53, <sven...@xlab.si> wrote:
>> Oh, I misunderstood because the -Bsymbolic-functions flag was never present
>> on my system.
>> It is present in Justin's build_clib.log.
>>
>> Now I ran
>> export LDFLAGS='-Wl,--no-as-needed -Bsymbolic-functions'
>> rm -rf build
>> python setup.py build
>> python setup.py install
>> and everything works as expected (demo/vampirtrace/ring.py generates the
>> trace files).
>>
>> Is this what you wanted me to test?
>>
>
> Yes, good to know it works. Now I'll start working on a fix for this issue.
>

Tentative fix in this branch:

https://bitbucket.org/mpi4py/mpi4py/branch/dalcinl/profile

Folks, could any of you give it a quick try and confirm whether it
works on Ubuntu (too lazy to fire up a docker and try it myself!) ?

[justin....@gmail.com]

My commands:
  515  git checkout dalcinl/profile
  516  rm -rf build/
  517  pip uninstall mpi4py
  518  python setup.py install

Then
mpiexec -n 1 -env VT_VERBOSE 2 python ring.py
No extra verbose output from VT, and no VT output files were created.

small part of build output:
checking for library 'vt-mpi' ...
/usr/bin/mpicc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c _configtest.c -o _configtest.o
/usr/bin/mpicc -pthread _configtest.o -L/usr/local/lib -lvt-mpi -o _configtest
success!
removing: _configtest.c _configtest.o _configtest
building 'vt-mpi' dylib library
/usr/bin/mpicc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c src/lib-pmpi/vt-mpi.c -o build/temp.linux-x86_64-3.5/src/lib-pmpi/vt-mpi.o
/usr/bin/mpicc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/src/lib-pmpi/vt-mpi.o -L/usr/local/lib -Wl,--enable-new-dtags,-R/usr/local/lib -lvt-mpi -o build/lib.linux-x86_64-3.5/mpi4py/lib-pmpi/libvt-mpi.so -Wl,--no-as-needed
checking for library 'vt-hyb' ...

Looking at /proc/PID/maps, "python ring.py" loads /root/devel/virtualenvs/venv-mpi4py/lib/python3.5/site-packages/mpi4py/lib-pmpi/libvt-mpi.so only (from cat /proc/25059/maps | grep vt)

Also building mpi4py with Sven's cmdline:
LD_FLAGS='-Wl,--no-as-needed -Bsymbolic-functions' python setup.py install
That didn't help me (but I'm using ubutnu 16.04, and he is on archLinux ver?).

mpiexec -n 1 -env VT_VERBOSE 2 -env LD_PRELOAD=/usr/local/lib/libvt-mpi.so.0.0.0 python ring.py
  loads also /usr/local/lib/libvt-mpi.so.0.0.0 and /usr/local/lib/libvt-mpi-unify.so.0.0.0
But I guess that last line/trick doesn't help to get real sollution :/

[justin....@gmail.com]

Should be LDFLAGS, not LD_FLAGS. Tnx, Sven :)
So this does help me, too.

[sven...@xlab.si]

Tentative fix in this branch:

https://bitbucket.org/mpi4py/mpi4py/branch/dalcinl/profile

Folks, could any of you give it a quick try and confirm whether it
works on Ubuntu (too lazy to fire up a docker and try it myself!) ?

The fix did not work for me.

I believe the problem is that extra_link_args are being appended to the end of the build command.
But the '-Wl,--no-as-needed' needs to appear before the object files in the command.
The reason that $LDFLAGS works is that the flags are put in the right position.

I have made a pull request (https://bitbucket.org/mpi4py/mpi4py/pull-requests/12/build-c-libraries-on-linux-with-linker/diff) with my fix where I changed the build_library() function in conf/mpidistutils.py so that '-Wl,--no-as-needed' is passed to extra_preargs (instead of extra_postargs).

[Lisandro Dalcin]

On 29 March 2018 at 12:56, <sven...@xlab.si> wrote:
>
>> Tentative fix in this branch:
>>
>> https://bitbucket.org/mpi4py/mpi4py/branch/dalcinl/profile
>>
>> Folks, could any of you give it a quick try and confirm whether it
>> works on Ubuntu (too lazy to fire up a docker and try it myself!) ?
>
>
> The fix did not work for me.
>
> I believe the problem is that extra_link_args are being appended to the end
> of the build command.
> But the '-Wl,--no-as-needed' needs to appear before the object files in the
> command.
> The reason that $LDFLAGS works is that the flags are put in the right
> position.
>

Yes, indeed.

> I have made a pull request
> (https://bitbucket.org/mpi4py/mpi4py/pull-requests/12/build-c-libraries-on-linux-with-linker/diff)
> with my fix where I changed the build_library() function in
> conf/mpidistutils.py so that '-Wl,--no-as-needed' is passed to extra_preargs
> (instead of extra_postargs).
>

Thanks! Merged to branches master and maint.