Installing Dedalus on cluster Stampede2

[Ashna Aggarwal]

Hello Dedalus Users, 

I'm a PhD student at UCLA working with Jon Aurnou. 

Having trouble installing Dedalus on Stampede2. After communicating with the TACC help desk, I was told to build dedalus from source on Stampede2 because users are advised against building their own conda environments. 

I loaded necessary fftw3 and MPI modules (mpvapich2/2.3.4 is compatible with fftw3/3.3.8 on this cluster, list of loaded modules is below)

1) intel/18.0.2   2) git/2.24.1   3) autotools/1.1   4) cmake/3.16.1   5) xalt/2.10.2   6) TACC   7) mvapich2/2.3.4   8) python3/3.7.0   9) fftw3/3.3.8

and set  the environment variables to the paths using 

export FFTW_INCLUDE_PATH = -I/opt/apps/intel18/impi18_0/fftw3/3.3.8/include
export FFTW_LIBRARY_PATH = -Wl,-rpath,/opt/apps/intel18/impi18_0/fftw3/3.3.8/lib-L/opt/apps/intel18/impi18_0/fftw3/3.3.8/lib-lfftw3_mpi-lfftw3
export MPI_INCLUDE_PATH = -I/opt/apps/intel18/mvapich2/2.3.4/include
export MPI_LIBRARY_PATH = -L/opt/apps/intel18/mvapich2/2.3.4/lib

Running 

pip3 install dedalus --user

gives

Collecting dedalus
  Using cached dedalus-2.2006.tar.gz (123 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: pytest in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from dedalus) (4.2.0)
Collecting pytest-benchmark
  Using cached pytest_benchmark-3.2.3-py2.py3-none-any.whl (49 kB)
Requirement already satisfied: numpy in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages/numpy-1.16.1-py3.7-linux-x86_64.egg (from dedalus) (1.16.1)
Collecting pytest-cov
  Using cached pytest_cov-2.11.1-py2.py3-none-any.whl (20 kB)
Requirement already satisfied: matplotlib in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from dedalus) (3.0.2)
Requirement already satisfied: mpi4py>=2.0.0 in /opt/apps/intel18/impi18_0/python3/3.7.0/lib/python3.7/site-packages (from dedalus) (3.0.0)
Collecting docopt
  Using cached docopt-0.6.2.tar.gz (25 kB)
Requirement already satisfied: scipy>=0.13.0 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages/scipy-1.2.0-py3.7-linux-x86_64.egg (from dedalus) (1.2.0)
Collecting h5py>=2.10.0
  Using cached h5py-3.2.1-cp37-cp37m-manylinux1_x86_64.whl (4.1 MB)
Collecting cached-property
  Using cached cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
Requirement already satisfied: cycler>=0.10 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from matplotlib->dedalus) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from matplotlib->dedalus) (1.0.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from matplotlib->dedalus) (2.3.1)
Requirement already satisfied: python-dateutil>=2.1 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from matplotlib->dedalus) (2.8.0)
Requirement already satisfied: six in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from cycler>=0.10->matplotlib->dedalus) (1.12.0)
Requirement already satisfied: setuptools in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib->dedalus) (40.8.0)
Requirement already satisfied: attrs>=17.4.0 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from pytest->dedalus) (18.2.0)
Requirement already satisfied: py>=1.5.0 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from pytest->dedalus) (1.7.0)
Requirement already satisfied: pluggy>=0.7 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from pytest->dedalus) (0.8.1)
Requirement already satisfied: atomicwrites>=1.0 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from pytest->dedalus) (1.3.0)
Requirement already satisfied: more-itertools>=4.0.0 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from pytest->dedalus) (5.0.0)
Collecting py-cpuinfo
  Using cached py-cpuinfo-7.0.0.tar.gz (95 kB)
Collecting coverage>=5.2.1
  Using cached coverage-5.5-cp37-cp37m-manylinux2010_x86_64.whl (242 kB)
Collecting pytest
  Using cached pytest-6.2.2-py3-none-any.whl (280 kB)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting packaging
  Using cached packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting attrs>=17.4.0
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting importlib-metadata>=0.12
  Using cached importlib_metadata-3.7.2-py3-none-any.whl (11 kB)
Collecting pluggy>=0.7
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting iniconfig
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting py>=1.5.0
  Using cached py-1.10.0-py2.py3-none-any.whl (97 kB)
Collecting zipp>=0.5
  Using cached zipp-3.4.1-py3-none-any.whl (5.2 kB)
Collecting typing-extensions>=3.6.4
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Using legacy 'setup.py install' for docopt, since package 'wheel' is not installed.
Using legacy 'setup.py install' for py-cpuinfo, since package 'wheel' is not installed.
Building wheels for collected packages: dedalus
  Building wheel for dedalus (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/apps/intel18/python3/3.7.0/bin/python3.7 /home1/06953/aaggarwa/.local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpgxvnd824
       cwd: /tmp/pip-install-da42c6ra/dedalus_cd82225e25d54fe2b6dfe28eae1abc51
  Complete output (93 lines):
  Looking for mpi include path
    Found env var MPI_INCLUDE_PATH = -I/opt/apps/intel18/mvapich2/2.3.4/include
  Looking for fftw include path
    Found env var FFTW_INCLUDE_PATH = -I/opt/apps/intel18/impi18_0/fftw3/3.3.8/include
  Looking for fftw library path
    Found env var FFTW_LIBRARY_PATH = -Wl,-rpath,/opt/apps/intel18/impi18_0/fftw3/3.3.8/lib-L/opt/apps/intel18/impi18_0/fftw3/3.3.8/lib-lfftw3_mpi-lfftw3
  Looking for mpi library path
    Found env var MPI_LIBRARY_PATH = -L/opt/apps/intel18/mvapich2/2.3.4/lib
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/dedalus
  copying dedalus/__init__.py -> build/lib.linux-x86_64-3.7/dedalus
  copying dedalus/__main__.py -> build/lib.linux-x86_64-3.7/dedalus
  copying dedalus/public.py -> build/lib.linux-x86_64-3.7/dedalus
  copying dedalus/dev.py -> build/lib.linux-x86_64-3.7/dedalus
  creating build/lib.linux-x86_64-3.7/dedalus/tests
  copying dedalus/tests/test_bvp.py -> build/lib.linux-x86_64-3.7/dedalus/tests
  copying dedalus/tests/__init__.py -> build/lib.linux-x86_64-3.7/dedalus/tests
  copying dedalus/tests/test_ivp.py -> build/lib.linux-x86_64-3.7/dedalus/tests
  copying dedalus/tests/test_nlbvp.py -> build/lib.linux-x86_64-3.7/dedalus/tests
  copying dedalus/tests/test_evp.py -> build/lib.linux-x86_64-3.7/dedalus/tests
  copying dedalus/tests/test_output.py -> build/lib.linux-x86_64-3.7/dedalus/tests
  copying dedalus/tests/test_matsolvers.py -> build/lib.linux-x86_64-3.7/dedalus/tests
  creating build/lib.linux-x86_64-3.7/dedalus/libraries
  copying dedalus/libraries/__init__.py -> build/lib.linux-x86_64-3.7/dedalus/libraries
  copying dedalus/libraries/matsolvers.py -> build/lib.linux-x86_64-3.7/dedalus/libraries
  creating build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/metadata.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/solvers.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/problems.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/__init__.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/system.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/field.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/pencil.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/basis.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/timesteppers_new.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/distributor.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/evaluator.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/future.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/operators.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/domain.py -> build/lib.linux-x86_64-3.7/dedalus/core
  copying dedalus/core/timesteppers.py -> build/lib.linux-x86_64-3.7/dedalus/core
  creating build/lib.linux-x86_64-3.7/dedalus/extras
  copying dedalus/extras/plot_tools.py -> build/lib.linux-x86_64-3.7/dedalus/extras
  copying dedalus/extras/atmospheres.py -> build/lib.linux-x86_64-3.7/dedalus/extras
  copying dedalus/extras/__init__.py -> build/lib.linux-x86_64-3.7/dedalus/extras
  copying dedalus/extras/flow_tools.py -> build/lib.linux-x86_64-3.7/dedalus/extras
  creating build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/general.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/__init__.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/parallel.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/parsing.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/progress.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/dispatch.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/cache.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/logging.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/config.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/sparse.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/plot_op.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/array.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/post.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  copying dedalus/tools/exceptions.py -> build/lib.linux-x86_64-3.7/dedalus/tools
  creating build/lib.linux-x86_64-3.7/dedalus/tests/special_functions
  copying dedalus/tests/special_functions/__init__.py -> build/lib.linux-x86_64-3.7/dedalus/tests/special_functions
  copying dedalus/tests/special_functions/airy.py -> build/lib.linux-x86_64-3.7/dedalus/tests/special_functions
  copying dedalus/tests/special_functions/bessel.py -> build/lib.linux-x86_64-3.7/dedalus/tests/special_functions
  creating build/lib.linux-x86_64-3.7/dedalus/libraries/fftw
  copying dedalus/libraries/fftw/__init__.py -> build/lib.linux-x86_64-3.7/dedalus/libraries/fftw
  copying dedalus/dedalus.cfg -> build/lib.linux-x86_64-3.7/dedalus
  copying dedalus/examples.tar.gz -> build/lib.linux-x86_64-3.7/dedalus
  running build_ext
  building 'dedalus.libraries.fftw.fftw_wrappers' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/dedalus
  creating build/temp.linux-x86_64-3.7/dedalus/libraries
  creating build/temp.linux-x86_64-3.7/dedalus/libraries/fftw
  mpicc -fno-strict-aliasing -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -Wformat -Wformat-security -D_FORTIFY_SOURCE=2 -fstack-protector -fwrapv -fpic -O3 -fPIC -Idedalus/libraries/fftw/ -I/tmp/pip-build-env-u_coywgd/overlay/lib/python3.7/site-packages/numpy/core/include -I/tmp/pip-build-env-u_coywgd/overlay/lib/python3.7/site-packages/mpi4py/include -I-I/opt/apps/intel18/mvapich2/2.3.4/include -I-I/opt/apps/intel18/impi18_0/fftw3/3.3.8/include -I/opt/apps/intel18/python3/3.7.0/include/python3.7m -c dedalus/libraries/fftw/fftw_wrappers.c -o build/temp.linux-x86_64-3.7/dedalus/libraries/fftw/fftw_wrappers.o -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -Wno-unused-function
  In file included from /tmp/pip-build-env-u_coywgd/overlay/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h(12),
                   from /tmp/pip-build-env-u_coywgd/overlay/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h(4),
                   from dedalus/libraries/fftw/fftw_wrappers.c(653):
  /tmp/pip-build-env-u_coywgd/overlay/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h(84): warning #2650: attributes ignored here
                        NPY_CHAR NPY_ATTR_DEPRECATE("Use NPY_STRING"),
                                 ^

  dedalus/libraries/fftw/fftw_wrappers.c(664): catastrophic error: cannot open source file "fftw3.h"
    #include "fftw3.h"
                      ^

  compilation aborted for dedalus/libraries/fftw/fftw_wrappers.c (code 4)
  error: command 'mpicc' failed with exit status 4
  ----------------------------------------
  ERROR: Failed building wheel for dedalus
Failed to build dedalus
ERROR: Could not build wheels for dedalus which use PEP 517 and cannot be installed directly

It looks like fftw3 is not being linked to MPI somehow - I've tried installing my own FFTW3 following the steps here for an older version of Stampede, but alas to no avail.  https://dedalus-project.readthedocs.io/en/latest/machines/stampede/stampede.html

Would love help on this issue. Thanks so much. 

-Ashna

[Geoffrey Vasil]

Hi Ashna, 

I’ve had possibly similar issues with the module system. The problem is that FFTW needs to be linked to MPI properly when the modules are installed. In the case I was dealing with, the only solution was to as the help desk to install a new module with the two linked. The consultants are usually surprised that this is a thing, And after a a bit of  back-and-forth, they get things installed and everything works. 

But this is only one guess. Give emailing the help desk another try and see what they say. 

Best, 

Geoff 

--
You received this message because you are subscribed to the Google Groups "Dedalus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/a50796af-dc5f-4840-af9d-1c00156ac9fbn%40googlegroups.com.

[Jeffrey S. Oishi]

Hi Ashna,

Those environment variables should not contain compiler flags, just bare paths. 

For example, try 

export FFTW_INCLUDE_PATH = /opt/apps/intel18/impi18_0/fftw3/3.3.8/include

And so on.

Jeff

--

[Ashna Aggarwal]

Hello Geoff and Jeff,

Thank you for your replies. It looked like the issue was indeed how I was setting the environment variables. Running export FFTW_INCLUDE_PATH = /opt/apps/intel18/impi18_0/fftw3/3.3.8/include, etc allowed me to install dedalus on Stampede2. 

However, I've run into another issue (I also don't mean to bother people here; I am happy to contact one of you separately if need be).  

After installing dedalus  

pip3 install dedalus

Requirement already satisfied: dedalus in ./.local/lib/python3.7/site-packages (2.2006)

Requirement already satisfied: pytest-cov in ./.local/lib/python3.7/site-packages (from dedalus) (2.11.1)

Collecting h5py>=2.10.0

  Using cached h5py-3.2.1-cp37-cp37m-manylinux1_x86_64.whl (4.1 MB)

Requirement already satisfied: pytest-benchmark in ./.local/lib/python3.7/site-packages (from dedalus) (3.2.3)

Requirement already satisfied: pytest in ./.local/lib/python3.7/site-packages (from dedalus) (6.2.2)

Requirement already satisfied: numpy in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages/numpy-1.16.1-py3.7-linux-x86_64.egg (from dedalus) (1.16.1)

Requirement already satisfied: matplotlib in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from dedalus) (3.0.2)

Requirement already satisfied: docopt in ./.local/lib/python3.7/site-packages (from dedalus) (0.6.2)

Requirement already satisfied: mpi4py>=2.0.0 in /opt/apps/intel18/impi18_0/python3/3.7.0/lib/python3.7/site-packages (from dedalus) (3.0.0)

Requirement already satisfied: scipy>=0.13.0 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages/scipy-1.2.0-py3.7-linux-x86_64.egg (from dedalus) (1.2.0)

Requirement already satisfied: cached-property in ./.local/lib/python3.7/site-packages (from h5py>=2.10.0->dedalus) (1.5.2)

Requirement already satisfied: cycler>=0.10 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from matplotlib->dedalus) (0.10.0)

Requirement already satisfied: kiwisolver>=1.0.1 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from matplotlib->dedalus) (1.0.1)

Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from matplotlib->dedalus) (2.3.1)

Requirement already satisfied: python-dateutil>=2.1 in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from matplotlib->dedalus) (2.8.0)

Requirement already satisfied: six in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from cycler>=0.10->matplotlib->dedalus) (1.12.0)

Requirement already satisfied: setuptools in /opt/apps/intel18/python3/3.7.0/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib->dedalus) (40.8.0)

Requirement already satisfied: packaging in ./.local/lib/python3.7/site-packages (from pytest->dedalus) (20.9)

Requirement already satisfied: py>=1.8.2 in ./.local/lib/python3.7/site-packages (from pytest->dedalus) (1.10.0)

Requirement already satisfied: importlib-metadata>=0.12 in ./.local/lib/python3.7/site-packages (from pytest->dedalus) (3.7.3)

Requirement already satisfied: pluggy<1.0.0a1,>=0.12 in ./.local/lib/python3.7/site-packages (from pytest->dedalus) (0.13.1)

Requirement already satisfied: attrs>=19.2.0 in ./.local/lib/python3.7/site-packages (from pytest->dedalus) (20.3.0)

Requirement already satisfied: toml in ./.local/lib/python3.7/site-packages (from pytest->dedalus) (0.10.2)

Requirement already satisfied: iniconfig in ./.local/lib/python3.7/site-packages (from pytest->dedalus) (1.1.1)

Requirement already satisfied: zipp>=0.5 in ./.local/lib/python3.7/site-packages (from importlib-metadata>=0.12->pytest->dedalus) (3.4.1)

Requirement already satisfied: typing-extensions>=3.6.4 in ./.local/lib/python3.7/site-packages (from importlib-metadata>=0.12->pytest->dedalus) (3.7.4.3)

Requirement already satisfied: py-cpuinfo in ./.local/lib/python3.7/site-packages (from pytest-benchmark->dedalus) (7.0.0)

Requirement already satisfied: coverage>=5.2.1 in ./.local/lib/python3.7/site-packages (from pytest-cov->dedalus) (5.5)

Installing collected packages: h5py

Successfully installed h5py-3.2.1

I am attempting to run a test case in an interactive session using 

mpiexec -n 4 python3 rayleigh_benard.py

where I have cloned the dedalus repository and am in the directory  /home1/06953/aaggarwa/dedalus/examples/ivp/2d_rayleigh_benard. But I receive the following error

Traceback (most recent call last):

  File "/home1/06953/aaggarwa/.local/lib/python3.7/site-packages/dedalus/libraries/fftw/__init__.py", line 4, in <module>

    from . import fftw_wrappers

ImportError: /opt/apps/intel18/impi18_0/fftw3/3.3.8/lib/libfftw3_mpi.so.3: undefined symbol: MPI_Send

Here are the modules I currently have loaded 1) intel/18.0.2   2) git/2.24.1   3) autotools/1.1   4) cmake/3.16.1   5) xalt/2.10.2   6) TACC   7) python3/3.7.0   8) fftw3/3.3.8   9) mvapich2/2.3.4

It looks like this is still an fftw3/mpi error. Perhaps Geoff is right and I need to install a new module with the two linked? I contacted the help desk a few days ago and have not heard back. 

-Ashna

[Jeffrey S. Oishi]

Hi Ashna,

Sorry, I should have caught this. This is a known problem with the intel libraries. FFTW in the intel math libraries is only partially implemented, and dedalus relies on the parts that are not implemented. The solution is to link against a real fftw library provided by your 8) fftw3/3.3.8 module.

j

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/21903a70-348c-4258-870d-135306d5d10dn%40googlegroups.com.

[Ashna Aggarwal]

Hello Jeff,

No worries. Okay I will try to link against the fftw3 library provided by fftw3/3.3.8. If I do 

module help fftw3/3.3.8

I find the following info about linking 

To use the FFTW3 library, compile your source code with:

        -I$TACC_FFTW3_INC

and add the following options to the link step for serial codes:

        -Wl,-rpath,$TACC_FFTW3_LIB  -L$TACC_FFTW3_LIB -lfftw3

for MPI codes:

        -Wl,-rpath,$TACC_FFTW3_LIB -L$TACC_FFTW3_LIB -lfftw3_mpi -lfftw3

I am guessing I need to use the latter in the link step. 

However, how do I link a python script against the fftw3 library? Should I use the mpicc wrapper? For a C code I think it would be something like mpicc rayleigh_benard.c -Wl,-rpath,$TACC_FFTW3_LIB  -L$TACC_FFTW3_LIB -lfftw3 but I am not familiar linking a python script against a library. 

Thanks again,

-Ashna

[Keaton Burns]

Hi Ashna,

The “linking” step here just entails setting FFTW_INCLUDE_PATH and FFTW_LIBRARY_PATH environment variables when you install dedalus.  The setup.py file in dedalus will internally pass these on when the internal C extensions are actually compiled and linked.

So you should just have to load and point those paths to a non-intel build of FFTW.  From the module help you’ve listed, I think you’ll just have to do

module load fftw3/3.3.8

export FFTW_INCLUDE_PATH=$TACC_FFTW3_INC

export FFTW_LIBRARY_PATH=$TACC_FFTW3_LIB

Best,

-Keaton

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/34e32cb3-e7b0-4ce5-aacd-a67b97a937acn%40googlegroups.com.

[Ben Brown]

Ashna,

      Here's the script I use for intel compiler installs on Pleiades; the block at the top has my compile instructions for FFTW with mpi.  That's being compiled using a different MPI stack than intel mpi (I think it's an SGI-based one; it's the NASA Pleiades standard).  Give that top block a try and see if it works?  Happy to help do some trouble shooting.

The script as a whole builds a full conda environment on top of a base conda install.

Here's my module env:

bpbrown@pfe27:~/conda_install> module list
Currently Loaded Modulefiles:
  1) mpi-hpe/mpt.2.23        2) comp-intel/2020.4.304   3) pkgsrc/2018Q3
bpbrown@pfe27:~/conda_install>

--Ben

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/CADZXxBhSqdm7Ur5MYCC8FkuR11dXDo9uhzk-AcLZqNdSoKX6ew%40mail.gmail.com.

[Victor Eijkhout]

Jeff, Keaton,

I'm the consultant helping Ashna with this problem. I can confirm that

%%%%%%%%%

 1015  module load fftw3/3.3.8

 1016  export FFTW_INCLUDE_PATH=$TACC_FFTW3_INC

 1017  export FFTW_LIBRARY_PATH=$TACC_FFTW3_LIB

 1018  pip3 uninstall dedalus

 1019  module load python3

 1020  pip3 install dedalus --user

 1021  python3

staff.stampede2(1027)$ python3

Python 3.7.0 (default, Feb  6 2019, 21:24:19)

[GCC Intel(R) C++ gcc 6.3 mode] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import dedalus.public

Traceback (most recent call last):

  File "/home1/06953/aaggarwa/.local/lib/python3.7/site-packages/dedalus/libraries/fftw/__init__.py", line 4, in <module>

    from . import fftw_wrappers

ImportError: /opt/apps/intel18/impi18_0/fftw3/3.3.8/lib/libfftw3_mpi.so.3: undefined symbol: MPI_Send

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/home1/06953/aaggarwa/.local/lib/python3.7/site-packages/dedalus/public.py", line 9, in <module>

    from .core import future as _future

  File "/home1/06953/aaggarwa/.local/lib/python3.7/site-packages/dedalus/core/future.py", line 8, in <module>

    from .field import Operand, Data, Scalar, Array, Field

  File "/home1/06953/aaggarwa/.local/lib/python3.7/site-packages/dedalus/core/field.py", line 15, in <module>

    from ..libraries.fftw import fftw_wrappers as fftw

  File "/home1/06953/aaggarwa/.local/lib/python3.7/site-packages/dedalus/libraries/fftw/__init__.py", line 7, in <module>

    raise ImportError("Make sure to build using `python3 setup.py build_ext --inplace`")

ImportError: Make sure to build using `python3 setup.py build_ext --inplace`

%%%%%%%%%%%

(How does one insert a code block on this forum?

(And my apologies for quoting hundreds of lines of earlier posts. I can't figure out how to get rid of it.)

Anyway. Let me know if you have further ideas.

Victor.

[Adrian Fraser]

Hi all,

I'm working on installing Dedalus on XSEDE's new computer Expanse, and am finding a similar error to what Victor posted here. Any chance you all were able to resolve the issue on Stampede2? I wonder if your solution might also help me out.

I loaded the following modules:

Currently Loaded Modules: 

1) shared                  8) py-six/1.14.0 

2) cpu/0.15.4              9) py-mpi4py/3.0.3 

3) DefaultModules         10) py-h5py/2.10.0-openblas 

4) gcc/10.2.0             11) py-numpy/1.19.2-openblas 

5) slurm/expanse/20.02.3  12) py-scipy/1.5.2-openblas 

6) openmpi/4.0.4          13) fftw/3.3.8 

7) python/3.8.5           14) py-pip/20.2

I then set FFTW_PATH and MPI_PATH to $FFTWHOME and $OPENMPIHOME, which are the env vars set by the fftw and openmpi modules. I then installed from source using "python3 -m pip install --user -e ." which finishes without error. When I run the test suite or try to import dedalus, however, I get almost the same error as above, except that the very top part refers to a slightly different "undefined symbol":

________________ ERROR collecting dedalus/tests/test_output.py ________________

ImportError while importing test module '/home/afraser/dedalus/dedalus/dedalus/tests/test_output.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

dedalus/libraries/fftw/__init__.py:4: in <module>

    from . import fftw_wrappers

E   ImportError: /cm/shared/apps/spack/cpu/opt/spack/linux-centos8-zen2/gcc-10.2.0/fftw-3.3.8-fxnokelozt4zqvy4wnkqgrke5hja5mr2/lib/libfftw3_mpi.so.3: undefined symbol: ompi_mpi_op_sum

Any tips on how to proceed are appreciated.

Thanks,

Adrian

[Ashna Aggarwal]

Hi Adrian,

With Keaton/Jeff/Ben's help, we were able to get Dedalus working on Stampede2. Below is how we set it up and installed. Let me know if that helps at all. 

-Ashna

# Turn off threading

export OMP_NUM_THREADS=1

# Modules

module load intel/19.1.1

module load impi/19.0.9

module load fftw3/3.3.8

module load phdf5/1.10.4

module load python3/3.7.0

# Paths for Dedalus

export FFTW_PATH=$TACC_FFTW3_DIR

export MPI_PATH=$TACC_IMPI_DIR/intel64

export HDF5_DIR=$TACC_HDF5_DIR

export FFTW_STATIC=1

# Compiler commands for distutils

export CC=mpicc

export LDSHARED="mpicc -shared"

With those environment settings, you can install install Dedalus with the following commands:

# Upgrade pip

pip install --upgrade --user pip

# Build h5py

python3 -m pip install --user --no-cache --no-binary=h5py h5py

# Build dedalus

python3 -m pip install --user --no-cache --no-build-isolation dedalus

[Keaton Burns]

Thanks for passing this along, Ashna.  Just to add some commentary for future reference — the crucial step here seemed to be setting the compiler aliases CC and LDSHARED, since those will add the correct linking/rpath information for finding MPI at runtime. I’m not sure why having the MPI path in LD_LIBRARY_PATH didn’t work, but so it seems.

The other thing to note is that the FFTW_STATIC option should only be required when using intel compilers or MKL.

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/c248993d-3020-4bb3-aa54-03ff184c9108n%40googlegroups.com.

[Adrian Fraser]

Ashna and Keaton,

The CC and LDSHARED env vars did the trick, I now have Dedalus running on Expanse! Woohoo! Thanks for the help. I plan to do some scaling tests in the near future, and am happy to share the results with anyone who's interested.

In case any future Dedalus users come across this post while trying to install Dedalus on Expanse: I loaded the modules and set FFTW_PATH and MPI_PATH as described in my previous reply, and set the same CC and LDSHARED env vars described in Ashna's reply, then installed setuptools using "python3 -m pip install --user setuptools", then went to the directory where I downloaded the dedalus source from github and added "import site" and "site.ENABLE_USER_SITE = "--user" in sys.argv[1:]" to setup.py (right after all the other import statements at the top), then did "python3 -m pip install --user -e ." and everything installed smoothly. I should add that this would not have been possible without some great help from Mahidhar over at Expanse.

Adrian

[Adrian Fraser]

Hi Ashna, et al.,

Did you ever test various dedalus.cfg settings on Stampede2 to see which options gave you the best performance? A student and I might be running on Stampede2 soon and are wondering if you've found any helpful tips you'd be willing to share.

Thanks!

Adrian

[Ashna Aggarwal]

Hi Adrian,

I have not carried out any scaling tests thus far (but I can if that would be helpful). 

I only modified the dedalus.cfg file when I encountered some memory issues and changed MATRIX_FACTORIZER to SuperLUNaturalSpsolve. There was enough slowdown that I changed it back to SuperLUNaturalFactorized and simply increased the number of nodes. 

However, let me know if there's anything that I can test that might be helpful for you and your student!

-Ashna

[Adrian Fraser]

Hi Ashna,

No need for you to do scaling tests for our sake -- we certainly can and should do our own scaling tests, especially since we may have quite different systems. I only wondered if there were any major, noteworthy performance observations you came across that we should prepare for. The matrix factorizer bit is very helpful to know, thank you!

Adrian