Install dedalus on google colab

林軒誠

unread,

Aug 28, 2023, 2:28:17 AM8/28/23

to Dedalus Users

Hi all dedalus users,
I am new to dedalus and trying to install dedalus on google colab. For the past few weeks, I have been using the method below to install dedalus without any issues. However, in the last two days, an error suddenly appeared.

So far, I still haven't found any clues or leads regarding the issue. Any help would be appreciated.

Installation method

/-----------------------------------------------------------------------------------------------------------------------------------------/

# Step 1: Install FFTW
!apt-get install libfftw3-dev
!apt-get install libfftw3-mpi-dev

# Step 2: Set paths for Dedalus installation
import os
import matplotlib.pyplot as plt
os.environ['MPI_INCLUDE_PATH'] = "/usr/lib/x86_64-linux-gnu/openmpi/include"
os.environ['MPI_LIBRARY_PATH'] = "/usr/lib/x86_64-linux-gnu"
os.environ['FFTW_INCLUDE_PATH'] = "/usr/include"
os.environ['FFTW_LIBRARY_PATH'] = "/usr/lib/x86_64-linux-gnu"

# Step 3: Install Dedalus using pip
!pip3 install --no-cache http://github.com/dedalusproject/dedalus/zipball/master/

/-----------------------------------------------------------------------------------------------------------------------------------------/

Error message

/-----------------------------------------------------------------------------------------------------------------------------------------/

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libfftw3-dev is already the newest version (3.3.8-2ubuntu8).
0 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libfftw3-mpi-dev is already the newest version (3.3.8-2ubuntu8).
0 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.
Collecting dedalus
Using cached dedalus-2.2207.3.tar.gz (111 kB)
Installing build dependencies ... done
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

/-----------------------------------------------------------------------------------------------------------------------------------------/

Thanks,

Peter

W Ahmed

unread,

Aug 28, 2023, 6:33:30 AM8/28/23

to dedalu...@googlegroups.com

How did you install it? As I know, no version of dedalus on colab.

--
You received this message because you are subscribed to the Google Groups "Dedalus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/3c6d2888-fa3a-41b0-b9cb-899d7dfd71e2n%40googlegroups.com.

林軒誠

unread,

Aug 28, 2023, 7:37:07 AM8/28/23

to Dedalus Users

I followed the instructions from the link below to install Dedalus, and it indeed worked before.

Dedalus in Colab

hwa...@gmail.com 在 2023年8月28日星期一下午6:33:30 [UTC+8] 的信中寫道：

mncr...@gmail.com

unread,

Aug 31, 2023, 7:56:17 AM8/31/23

to Dedalus Users

Hi,

I'm getting a similar error when installing to an anaconda environment using:

# Install Dedalus v2 from conda-forge to build stack conda install -c conda-forge dedalus c-compiler "h5py=*=mpi*" # Upgrade to Dedalus v3 conda uninstall --force dedalus CC=mpicc pip3 install --no-cache http://github.com/dedalusproject/dedalus/zipball/master/
This has worked previously but now 'transposes.pyx' seems to be having some trouble importing modules in Cython. Maybe this is the same problem as you're having?

Matt

The error is:

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting http://github.com/dedalusproject/dedalus/zipball/master/
Downloading http://github.com/dedalusproject/dedalus/zipball/master/
- 24.0 MB 18.3 MB/s 0:00:01

Installing build dependencies ... done

Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1

╰─> [269 lines of output]

Error compiling Cython file:
------------------------------------------------------------
...
from mpi4py.MPI cimport Comm as py_comm_t
from mpi4py.libmpi cimport MPI_Comm as mpi_comm_t
from libc.stddef cimport ptrdiff_t as p_t

from ..libraries.fftw import fftw_wrappers as fftw
from ..libraries.fftw cimport fftw_c_api as cfftw
^
------------------------------------------------------------

dedalus/core/transposes.pyx:17:0: 'dedalus/libraries/fftw.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...
cdef readonly int[::1] RL_reduced_shape
cdef readonly double[::1] CL_buffer
cdef readonly double[::1] RL_buffer
cdef readonly cnp.ndarray CL_view
cdef readonly cnp.ndarray RL_view
cdef cfftw.fftw_plan CL_to_RL_plan
^
------------------------------------------------------------

dedalus/core/transposes.pyx:57:9: 'cfftw' is not declared

Error compiling Cython file:
------------------------------------------------------------
...
cdef readonly double[::1] CL_buffer
cdef readonly double[::1] RL_buffer
cdef readonly cnp.ndarray CL_view
cdef readonly cnp.ndarray RL_view
cdef cfftw.fftw_plan CL_to_RL_plan
cdef cfftw.fftw_plan RL_to_CL_plan
^
------------------------------------------------------------

dedalus/core/transposes.pyx:58:9: 'cfftw' is not declared

Error compiling Cython file:
------------------------------------------------------------
...
for f in flags:
intflags = intflags | fftw.fftw_flags[f]

# Build buffers
cdef p_t alloc_doubles, local0, start0, local1, start1
alloc_doubles = cfftw.fftw_mpi_local_size_many_transposed(
^
------------------------------------------------------------

dedalus/core/transposes.pyx:150:24: undeclared name not builtin: cfftw

Error compiling Cython file:
------------------------------------------------------------
...

# Build buffers
cdef p_t alloc_doubles, local0, start0, local1, start1
alloc_doubles = cfftw.fftw_mpi_local_size_many_transposed(
2,
shape,
^
------------------------------------------------------------

dedalus/core/transposes.pyx:152:16: Cannot convert 'ptrdiff_t *' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
2,
shape,
howmany,
block0,
block1,
comm,
^
------------------------------------------------------------

dedalus/core/transposes.pyx:156:16: Cannot convert 'MPI_Comm' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
shape,
howmany,
block0,
block1,
comm,
&local0,
^
------------------------------------------------------------

dedalus/core/transposes.pyx:157:16: Cannot convert 'ptrdiff_t *' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
howmany,
block0,
block1,
comm,
&local0,
&start0,
^
------------------------------------------------------------

dedalus/core/transposes.pyx:158:16: Cannot convert 'ptrdiff_t *' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
block0,
block1,
comm,
&local0,
&start0,
&local1,
^
------------------------------------------------------------

dedalus/core/transposes.pyx:159:16: Cannot convert 'ptrdiff_t *' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
block1,
comm,
&local0,
&start0,
&local1,
&start1)
^
------------------------------------------------------------

dedalus/core/transposes.pyx:160:16: Cannot convert 'ptrdiff_t *' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
n1,
n0,
howmany,
block1,
block0,
&self.CL_buffer[0],
^
------------------------------------------------------------

dedalus/core/transposes.pyx:177:16: Cannot convert 'double *' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
n0,
howmany,
block1,
block0,
&self.CL_buffer[0],
&self.RL_buffer[0],
^
------------------------------------------------------------

dedalus/core/transposes.pyx:178:16: Cannot convert 'double *' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
howmany,
block1,
block0,
&self.CL_buffer[0],
&self.RL_buffer[0],
comm,
^
------------------------------------------------------------

dedalus/core/transposes.pyx:179:16: Cannot convert 'MPI_Comm' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
n0,
n1,
howmany,
block0,
block1,
&self.RL_buffer[0],
^
------------------------------------------------------------

dedalus/core/transposes.pyx:187:16: Cannot convert 'double *' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
n1,
howmany,
block0,
block1,
&self.RL_buffer[0],
&self.CL_buffer[0],
^
------------------------------------------------------------

dedalus/core/transposes.pyx:188:16: Cannot convert 'double *' to Python object

Error compiling Cython file:
------------------------------------------------------------
...
howmany,
block0,
block1,
&self.RL_buffer[0],
&self.CL_buffer[0],
comm,
^
------------------------------------------------------------

dedalus/core/transposes.pyx:189:16: Cannot convert 'MPI_Comm' to Python object
warning: dedalus/core/transposes.pyx:310:38: Index should be typed for more efficient access
Looking for mpi include path
Cannot find env var MPI_INCLUDE_PATH
Looking for mpi prefix
Found matching library in /local/mcrowe/Anaconda/envs/d3_Pytorch/lib
Looking for fftw include path
Cannot find env var FFTW_INCLUDE_PATH
Looking for fftw prefix
Found matching library in /local/mcrowe/Anaconda/envs/d3_Pytorch/lib
Looking for fftw library path
Cannot find env var FFTW_LIBRARY_PATH
Looking for fftw prefix
Found matching library in /local/mcrowe/Anaconda/envs/d3_Pytorch/lib
Looking for mpi library path
Cannot find env var MPI_LIBRARY_PATH
Looking for mpi prefix
Found matching library in /local/mcrowe/Anaconda/envs/d3_Pytorch/lib
Compiling dedalus/libraries/fftw/fftw_wrappers.pyx because it changed.
Compiling dedalus/core/transposes.pyx because it changed.
Compiling dedalus/libraries/spin_recombination.pyx because it changed.
[1/3] Cythonizing dedalus/core/transposes.pyx
Traceback (most recent call last):
File "/local/mcrowe/Anaconda/envs/d3_Pytorch/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/local/mcrowe/Anaconda/envs/d3_Pytorch/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local/mcrowe/Anaconda/envs/d3_Pytorch/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-ot88z2um/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-ot88z2um/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-ot88z2um/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 507, in run_setup
super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-ot88z2um/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
exec(code, locals())
File "<string>", line 235, in <module>
File "/tmp/pip-build-env-ot88z2um/overlay/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
cythonize_one(*args)
File "/tmp/pip-build-env-ot88z2um/overlay/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: dedalus/core/transposes.pyx
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

林軒誠

unread,

Aug 31, 2023, 8:15:37 AM8/31/23

to Dedalus Users

Hello Matt.
Yesterday, I attempted to install Dedalus on my own computer (using the exact same instructions as you). There were no issues during the uninstallation of Dedalus v2. However, when executing
'CC=mpicc pip3 install --no-cache http://github.com/dedalusproject/dedalus/zipball/master/'
I encountered errors about Cython. The errors presented during the installation process seemed to differ slightly from what you described but still included messages like 'Cannot convert 'double *' to Python object' and 'self._get_build_requires()'.
I think we had same troubles, but for now I can't find a solution.

Thank your response!

mncr...@gmail.com 在 2023年8月31日星期四晚上7:56:17 [UTC+8] 的信中寫道：

mncr...@gmail.com

unread,

Aug 31, 2023, 10:02:35 AM8/31/23

to Dedalus Users

I've just tried various python versions using my Anaconda stack and a new Miniconda stack. It doesn't seem to be a problem with my installation.

Matt

林軒誠

unread,

Aug 31, 2023, 11:10:50 AM8/31/23

to Dedalus Users

Now I'm using miniconda, on my lab server
OS: macOS (arm64-apple-darwin21.5.0) CPU: 20 × Apple M1 Ultra Python 3.10.10
I tried again and below was my error message, just exactly same as your previous installation.

I'm sorry that I have already spent too much time dealing with the installation problem, and my professor has suggested use Julia to build a 2D spherical shallow water equation.
Probably going to give up on the installation. Still, thank you very much for your reply.

/-----------------------------------------------------------------------------------------------------------------------------------------/
(Pcore_dedalus3) b08209033 @natpc1 /Volumes/Expansion3/User_backup2/b08209033 % CC=mpicc pip3 install --no-cache http://github.com/dedalusproject/dedalus/zipball/master/

Collecting http://github.com/dedalusproject/dedalus/zipball/master/
Downloading http://github.com/dedalusproject/dedalus/zipball/master/

| 24.0 MB 5.3 MB/s 0:00:04

Found matching library in /Users/b08209033/miniconda3/envs/Pcore_dedalus3/lib

Looking for fftw include path
Cannot find env var FFTW_INCLUDE_PATH
Looking for fftw prefix

Found matching library in /Users/b08209033/miniconda3/envs/Pcore_dedalus3/lib

Looking for fftw library path
Cannot find env var FFTW_LIBRARY_PATH
Looking for fftw prefix

Found matching library in /Users/b08209033/miniconda3/envs/Pcore_dedalus3/lib

Looking for mpi library path
Cannot find env var MPI_LIBRARY_PATH
Looking for mpi prefix

Found matching library in /Users/b08209033/miniconda3/envs/Pcore_dedalus3/lib

Compiling dedalus/libraries/fftw/fftw_wrappers.pyx because it changed.
Compiling dedalus/core/transposes.pyx because it changed.
Compiling dedalus/libraries/spin_recombination.pyx because it changed.
[1/3] Cythonizing dedalus/core/transposes.pyx
Traceback (most recent call last):

File "/Users/b08209033/miniconda3/envs/Pcore_dedalus3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/Users/b08209033/miniconda3/envs/Pcore_dedalus3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main

json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/Users/b08209033/miniconda3/envs/Pcore_dedalus3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/jg/7hc7yjgn1jxf7t4jbvm7g9640000gw/T/pip-build-env-rfiwz8z8/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel

return self._get_build_requires(config_settings, requirements=['wheel'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/private/var/folders/jg/7hc7yjgn1jxf7t4jbvm7g9640000gw/T/pip-build-env-rfiwz8z8/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
self.run_setup()
File "/private/var/folders/jg/7hc7yjgn1jxf7t4jbvm7g9640000gw/T/pip-build-env-rfiwz8z8/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 507, in run_setup
super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
File "/private/var/folders/jg/7hc7yjgn1jxf7t4jbvm7g9640000gw/T/pip-build-env-rfiwz8z8/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup

exec(code, locals())
File "<string>", line 235, in <module>

File "/private/var/folders/jg/7hc7yjgn1jxf7t4jbvm7g9640000gw/T/pip-build-env-rfiwz8z8/overlay/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
cythonize_one(*args)
File "/private/var/folders/jg/7hc7yjgn1jxf7t4jbvm7g9640000gw/T/pip-build-env-rfiwz8z8/overlay/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one

raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: dedalus/core/transposes.pyx
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

/-----------------------------------------------------------------------------------------------------------------------------------------/

mncr...@gmail.com 在 2023年8月31日星期四晚上10:02:35 [UTC+8] 的信中寫道：

林軒誠

unread,

Sep 2, 2023, 1:02:36 AM9/2/23

to Dedalus Users

Somebody on GitHub has already provided a solution, and it worked for me.

Here's the link for reference : Installation currently requires Cython<3

林軒誠在 2023年8月31日星期四晚上11:10:50 [UTC+8] 的信中寫道：

Reply all

Reply to author

Forward