Fwd: trouble installing Dedalus on hpc brc

Daniel Lecoanet

unread,

Apr 9, 2015, 2:28:44 AM4/9/15

to dedal...@googlegroups.com, Eve Lee

Hi all,

We're trying to get Dedalus back running on the berkeley savio cluster.

Any idea what's going on with this hgapi install error in the attached email? Ben -- this seems reminiscent of a problem that was coming up on Janus.

Anyway, the matplotlib issue goes away when you use version 1.3.1. I tried removing hgapi from the requirements.txt since it doesn't seem like it's actually required for the installation. However, when I run the install script, I get a bunch of cython errors:

(python_build_2)[lecoanet@ln003 dedalus]$ python3 setup.py build_ext --inplace

Looking for fftw prefix

Found env var FFTW_PATH = /global/software/sl-6.x86_64/modules/intel/2013_sp1.4.211/fftw/3.3.4-intel

Looking for mpi prefix

Found env var MPI_PATH = /global/software/sl-6.x86_64/modules/intel/2013_sp1.2.144/openmpi/1.6.5-intel

Looking for fftw prefix

Found env var FFTW_PATH = /global/software/sl-6.x86_64/modules/intel/2013_sp1.4.211/fftw/3.3.4-intel

missing cimport in module 'mpi4py.MPI': dedalus/libraries/fftw/fftw_wrappers.pyx

missing cimport in module 'mpi4py.mpi_c': dedalus/libraries/fftw/fftw_wrappers.pyx

missing cimport in module 'mpi4py.mpi_c': ./dedalus/libraries/fftw/fftw_c_api.pxd

missing cimport in module 'mpi4py.MPI': dedalus/core/transposes.pyx

missing cimport in module 'mpi4py.mpi_c': dedalus/core/transposes.pyx

missing cimport in module '..libraries.fftw': dedalus/core/transposes.pyx

Compiling dedalus/libraries/fftw/fftw_wrappers.pyx because it changed.

Compiling dedalus/core/transposes.pyx because it changed.

Cythonizing dedalus/core/transposes.pyx

Error compiling Cython file:

------------------------------------------------------------

...

import logging

logger = logging.getLogger(__name__.split('.')[-1])

from mpi4py import MPI

from mpi4py.MPI cimport Comm as py_comm_t

^

------------------------------------------------------------

dedalus/core/transposes.pyx:12:0: 'mpi4py/MPI.pxd' not found

Error compiling Cython file:

------------------------------------------------------------

...

import logging

logger = logging.getLogger(__name__.split('.')[-1])

from mpi4py import MPI

from mpi4py.MPI cimport Comm as py_comm_t

^

------------------------------------------------------------

dedalus/core/transposes.pyx:12:0: 'Comm.pxd' not found

Error compiling Cython file:

------------------------------------------------------------

...

import logging

logger = logging.getLogger(__name__.split('.')[-1])

from mpi4py import MPI

from mpi4py.MPI cimport Comm as py_comm_t

^

------------------------------------------------------------

dedalus/core/transposes.pyx:12:24: Name 'Comm' not declared in module 'mpi4py.MPI'

Error compiling Cython file:

------------------------------------------------------------

...

import logging

logger = logging.getLogger(__name__.split('.')[-1])

from mpi4py import MPI

from mpi4py.MPI cimport Comm as py_comm_t

from mpi4py.mpi_c cimport MPI_Comm as mpi_comm_t

^

------------------------------------------------------------

dedalus/core/transposes.pyx:13:0: 'mpi4py/mpi_c.pxd' not found

Error compiling Cython file:

------------------------------------------------------------

...

import logging

logger = logging.getLogger(__name__.split('.')[-1])

from mpi4py import MPI

from mpi4py.MPI cimport Comm as py_comm_t

from mpi4py.mpi_c cimport MPI_Comm as mpi_comm_t

^

------------------------------------------------------------

dedalus/core/transposes.pyx:13:0: 'MPI_Comm.pxd' not found

Error compiling Cython file:

------------------------------------------------------------

...

import logging

logger = logging.getLogger(__name__.split('.')[-1])

from mpi4py import MPI

from mpi4py.MPI cimport Comm as py_comm_t

from mpi4py.mpi_c cimport MPI_Comm as mpi_comm_t

^

------------------------------------------------------------

dedalus/core/transposes.pyx:13:26: Name 'MPI_Comm' not declared in module 'mpi4py.mpi_c'

Error compiling Cython file:

------------------------------------------------------------

...

FFTW Cython declarations. Comments refer to corresponding sections of the

FFTW3 documentation.

"""

from mpi4py.mpi_c cimport MPI_Comm as mpi_comm_t

^

------------------------------------------------------------

dedalus/libraries/fftw/fftw_c_api.pxd:7:26: Name 'MPI_Comm' not declared in module 'mpi4py.mpi_c'

Error compiling Cython file:

------------------------------------------------------------

...

ptrdiff_t fftw_mpi_local_size_many_transposed(int rank,

ptrdiff_t *shape,

ptrdiff_t itemsize,

ptrdiff_t block0,

ptrdiff_t block1,

mpi_comm_t comm,

^

------------------------------------------------------------

dedalus/libraries/fftw/fftw_c_api.pxd:118:50: 'mpi_comm_t' is not a type identifier

Error compiling Cython file:

------------------------------------------------------------

...

ptrdiff_t itemsize,

ptrdiff_t block0,

ptrdiff_t block1,

double *in_,

double *out,

mpi_comm_t comm,

^

------------------------------------------------------------

dedalus/libraries/fftw/fftw_c_api.pxd:132:43: 'mpi_comm_t' is not a type identifier

Error compiling Cython file:

------------------------------------------------------------

...

pycomm : mpi4py communicator

Communicator

"""

cdef readonly py_comm_t pycomm

^

------------------------------------------------------------

dedalus/core/transposes.pyx:41:18: 'py_comm_t' is not a type identifier

Error compiling Cython file:

------------------------------------------------------------

...

pycomm : mpi4py communicator

Communicator

"""

cdef readonly py_comm_t pycomm

^

------------------------------------------------------------

dedalus/core/transposes.pyx:41:28: C attribute of type '<error>' cannot be accessed from Python

Error compiling Cython file:

------------------------------------------------------------

...

# def _fftw_setup(self, scales):

# logger.debug("Building FFTW transpose plan for (scales, axis, in_place) = (%s, %s, %s)" %(scales, axis, IN_PLACE))

def build_plans(self, p_t n0, p_t n1, p_t howmany, p_t block0, p_t block1,

py_comm_t pycomm, in_place, flags=['FFTW_MEASURE']):

^

------------------------------------------------------------

dedalus/core/transposes.pyx:108:20: 'py_comm_t' is not a type identifier

Error compiling Cython file:

------------------------------------------------------------

...

pycomm : mpi4py communicator

Communicator

"""

cdef readonly py_comm_t pycomm

^

------------------------------------------------------------

dedalus/core/transposes.pyx:235:18: 'py_comm_t' is not a type identifier

Error compiling Cython file:

------------------------------------------------------------

...

pycomm : mpi4py communicator

Communicator

"""

cdef readonly py_comm_t pycomm

^

------------------------------------------------------------

dedalus/core/transposes.pyx:235:28: C attribute of type '<error>' cannot be accessed from Python

Error compiling Cython file:

------------------------------------------------------------

...

"""

# Shape array

cdef p_t *shape = [n0, n1]

# C MPI communicator

cdef mpi_comm_t comm = pycomm.ob_mpi

^

------------------------------------------------------------

dedalus/core/transposes.pyx:133:13: 'mpi_comm_t' is not a type identifier

warning: dedalus/core/transposes.pyx:301:38: Index should be typed for more efficient access

Traceback (most recent call last):

File "setup.py", line 148, in <module>

ext_modules = cythonize(extensions))

File "/global/home/users/lecoanet/python_build_2/lib/python3.2/site-packages/Cython/Build/Dependencies.py", line 865, in cythonize

cythonize_one(*args[1:])

File "/global/home/users/lecoanet/python_build_2/lib/python3.2/site-packages/Cython/Build/Dependencies.py", line 985, in cythonize_one

raise CompileError(None, pyx_file)

Cython.Compiler.Errors.CompileError: dedalus/core/transposes.pyx

Any idea what this all means? It's similar to some errors we were getting before due to old versions of cython. However, I think I'm running with the latest & greatest cython:

(python_build_2)[lecoanet@ln003 dedalus]$ cython --version

Cython version 0.22

Keaton -- do you have access to Savio?

Daniel

---------- Forwarded message ----------
From: Eve Lee <eve...@berkeley.edu>
Date: Wed, Apr 8, 2015 at 11:20 AM
Subject: trouble installing Dedalus on hpc brc
To: Daniel Lecoanet <dlec...@berkeley.edu>

Hi Daniel,

I'm having some trouble installing Dedalus on hpc brc following the install notes online. I probably should ask via user list but since this is an issue of installation, would you mind if I ask you directly?

So first, the matplotlib installation gets stuck after printing out required dependencies (the install script correctly detects all required dependencies). I read somewhere that matplotlib no longer supports python 3.2.x so maybe this has to do with the issue?

Then if I just ignore that and continue to install dedalus using "pip-3.2 install -r requirements.txt", I get this:

Downloading/unpacking hgapi (from -r requirements.txt (line 4))

Downloading hgapi-1.7.2.tar.gz

Running setup.py egg_info for package hgapi

Traceback (most recent call last):

File "<string>", line 14, in <module>

File "/global/home/users/eflee/dedalus/python_build/lib/python3.2/encodings/ascii.py", line 26, in decode

return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 209: ordinal not in range(128)

Complete output from command python setup.py egg_info:

Traceback (most recent call last):

File "<string>", line 14, in <module>

File "/global/home/users/eflee/dedalus/python_build/lib/python3.2/encodings/ascii.py", line 26, in decode

return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 209: ordinal not in range(128)

Have you ever come across this? Any advice?

Thanks in advance!

Best,

Eve

Keaton Burns

unread,

Apr 9, 2015, 10:06:32 AM4/9/15

to dedal...@googlegroups.com, Eve Lee

Hi Daniel & Eve,

The stack of errors seems to indicate that mpi4py didn’t get correctly installed / linked with MPI. Also, hgapi is an optional requirement for the setup file, but should be a pretty light dependency.

I’m wondering if these issues are related to using python 3.2. Is there a newer version of python 3 available on the cluster? If not, it might be worth trying to build a 3.3 or 3.4 stack manually, pretty much following the install notes for other machines.

-Keaton

--
You received this message because you are subscribed to the Google Groups "Dedalus Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-dev...@googlegroups.com.
To post to this group, send email to dedal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/CAJoYf%3Dhy81ZEGLO-3s3MXop6ZCv3-J5_BiFb5MSwdhXc7pLdgQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Ben Brown

unread,

Apr 9, 2015, 10:35:42 AM4/9/15

to dedal...@googlegroups.com, Eve Lee

All,

The Pleiades Intel stack documentation is pretty up to date, though it's been a bit since I checked on them. Please let me know if there any problems.

http://dedalus-project.readthedocs.org/en/latest/machines/nasa_pleiades/pleiades_intel.html

--Ben

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/1567CE07-5530-4751-B6B3-7ED5C467127F%40gmail.com.

Daniel Lecoanet

unread,

Apr 9, 2015, 1:51:16 PM4/9/15

to dedal...@googlegroups.com, Eve Lee

Well mpi4py works in the sense that I am able to install & (I think) run the old version of Dedalus from my branch (dedalus2-lecoanet). I have run on savio before, and don't think they changed their MPI.

Maybe it would be better to try to install our own stack by hand... I was having some problems on large core counts before...

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/1567CE07-5530-4751-B6B3-7ED5C467127F%40gmail.com.

Daniel Lecoanet

unread,

Apr 16, 2015, 5:11:23 PM4/16/15

to dedal...@googlegroups.com, Eve Lee

Hi all,

I've tried using the Pleiades intel stack on Savio. However, when trying to install python3, I'm having a "_ctypes" error, which I think we've come across before. When I try to make/make install I get errors:

configure: error: in `/global/home/users/lecoanet/build/Python-3.4.2/build/temp.linux-x86_64-3.4/libffi':

configure: error: C++ preprocessor "/lib/cpp" fails sanity check

See `config.log' for more details

Failed to configure _ctypes module

Python build finished successfully!

The necessary bits to build these optional modules were not found:

_lzma _sqlite3

To find the necessary bits, look in setup.py in detect_modules() for the module's name.

and

Traceback (most recent call last):

File "/global/home/users/lecoanet/build/Python-3.4.2/Lib/runpy.py", line 170, in _run_module_as_main

"__main__", mod_spec)

File "/global/home/users/lecoanet/build/Python-3.4.2/Lib/runpy.py", line 85, in _run_code

exec(code, run_globals)

File "/global/home/users/lecoanet/build/Python-3.4.2/Lib/ensurepip/__main__.py", line 4, in <module>

ensurepip._main()

File "/global/home/users/lecoanet/build/Python-3.4.2/Lib/ensurepip/__init__.py", line 209, in _main

default_pip=args.default_pip,

File "/global/home/users/lecoanet/build/Python-3.4.2/Lib/ensurepip/__init__.py", line 116, in bootstrap

_run_pip(args + [p[0] for p in _PROJECTS], additional_paths)

File "/global/home/users/lecoanet/build/Python-3.4.2/Lib/ensurepip/__init__.py", line 40, in _run_pip

import pip

File "/tmp/tmp673kcmsp/pip-1.5.6-py2.py3-none-any.whl/pip/__init__.py", line 9, in <module>

File "/tmp/tmp673kcmsp/pip-1.5.6-py2.py3-none-any.whl/pip/log.py", line 9, in <module>

File "/tmp/tmp673kcmsp/pip-1.5.6-py2.py3-none-any.whl/pip/_vendor/colorama/__init__.py", line 2, in <module>

File "/tmp/tmp673kcmsp/pip-1.5.6-py2.py3-none-any.whl/pip/_vendor/colorama/initialise.py", line 5, in <module>

File "/tmp/tmp673kcmsp/pip-1.5.6-py2.py3-none-any.whl/pip/_vendor/colorama/ansitowin32.py", line 6, in <module>

File "/tmp/tmp673kcmsp/pip-1.5.6-py2.py3-none-any.whl/pip/_vendor/colorama/winterm.py", line 2, in <module>

File "/tmp/tmp673kcmsp/pip-1.5.6-py2.py3-none-any.whl/pip/_vendor/colorama/win32.py", line 7, in <module>

File "/global/home/users/lecoanet/build/Python-3.4.2/Lib/ctypes/__init__.py", line 7, in <module>

from _ctypes import Union, Structure, Array

ImportError: No module named '_ctypes'

make: *** [install] Error 1

I remember that Ben made some intel patch, so I tried using it, but got the same errors. I'm using intel/2015.0.090 and openmpi/1.8.3-intel and python version 3.4.2. Here's what I'm doing for the installation:

./configure --prefix=$BUILD_HOME \
CC=mpicc CFLAGS="-mkl -O3 -axCORE-AVX2 -xSSE4.2 -fPIC -ipo" \
CXX=mpicxx CPPFLAGS="-mkl -O3 -axCORE-AVX2 -xSSE4.2 -fPIC -ipo" \
F90=mpif90 F90FLAGS="-mkl -O3 -axCORE-AVX2 -xSSE4.2 -fPIC -ipo" \
--enable-shared LDFLAGS="-lpthread" \
--with-cxx-main=mpicxx --with-system-ffi

make
make install

Any thoughts?

Daniel

Ben Brown

unread,

Apr 17, 2015, 10:38:56 AM4/17/15

to dedal...@googlegroups.com, Eve Lee

Daniel,

This looks like a higher level compiler suite error: the "c++ compiler fails sanity check" is a new and different error.

Could we try mpicxx on a known good c++ hello world code? I can send one later this afternoon. Maybe also try 1.8.4 for openmpi?

--Ben

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/CAJoYf%3DhmR0SFsXtwjCS_OBwJJa6dOA%2B_%3D2c19cKxd%2B%2BTn%2Bpr3A%40mail.gmail.com.

Daniel Lecoanet

unread,

Apr 17, 2015, 1:25:07 PM4/17/15

to dedal...@googlegroups.com

1.8.3 is the only one already installed, but I could install 1.8.4 myself.

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-dev/CAHqBLzyddH9heKKnfCBz5a%2B_j0SDpzSVzkT8xy%3DQ1MOWWYmRZw%40mail.gmail.com.

Daniel Lecoanet

unread,

Apr 25, 2015, 2:14:41 AM4/25/15

to dedal...@googlegroups.com

Hey Ben,

I installed openmpi-1.8.4 by making a couple of small modifications to the pleiades install instructions. However, I again get an error when I try to make python (again, following the pleiades install instructions):

checking how to run the C++ preprocessor... /lib/cpp

configure: error: in `/global/home/users/lecoanet/build/Python-3.4.2/build/temp.linux-x86_64-3.4/libffi':

configure: error: C++ preprocessor "/lib/cpp" fails sanity check

See `config.log' for more details

Failed to configure _ctypes module

Python build finished successfully!

The necessary bits to build these optional modules were not found:

_lzma _sqlite3

To find the necessary bits, look in setup.py in detect_modules() for the module's name.

running build_scripts

creating build/scripts-3.4

copying and adjusting /global/home/users/lecoanet/build/Python-3.4.2/Tools/scripts/pydoc3 -> build/scripts-3.4

copying and adjusting /global/home/users/lecoanet/build/Python-3.4.2/Tools/scripts/idle3 -> build/scripts-3.4

copying and adjusting /global/home/users/lecoanet/build/Python-3.4.2/Tools/scripts/2to3 -> build/scripts-3.4

copying and adjusting /global/home/users/lecoanet/build/Python-3.4.2/Tools/scripts/pyvenv -> build/scripts-3.4

changing mode of build/scripts-3.4/pydoc3 from 644 to 755

changing mode of build/scripts-3.4/idle3 from 644 to 755

changing mode of build/scripts-3.4/2to3 from 644 to 755

changing mode of build/scripts-3.4/pyvenv from 644 to 755

renaming build/scripts-3.4/pydoc3 to build/scripts-3.4/pydoc3.4

renaming build/scripts-3.4/idle3 to build/scripts-3.4/idle3.4

renaming build/scripts-3.4/2to3 to build/scripts-3.4/2to3-3.4

renaming build/scripts-3.4/pyvenv to build/scripts-3.4/pyvenv-3.4

/usr/bin/install -c -m 644 ./Tools/gdb/libpython.py python-gdb.py

mpicxx -c -Wno-unused-result -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -mkl -O3 -axCORE-AVX2 -xSSE4.2 -fPIC -ipo -Werror=declaration-after-statement -I. -IInclude -I./Include -mkl -O3 -axCORE-AVX2 -xSSE4.2 -fPIC -ipo -fPIC -DPy_BUILD_CORE -o Modules/_testembed.o ./Modules/_testembed.c

icc: command line warning #10006: ignoring unknown option '-Wno-unused-result'

mpicxx -lpthread -Xlinker -export-dynamic -o Modules/_testembed Modules/_testembed.o -L. -lpython3.4m -ldl -lm

# Substitution happens here, as the completely-expanded BINDIR

# is not available in configure

sed -e "s,@EXENAME@,/global/home/users/lecoanet/build/bin/python3.4m," < ./Misc/python-config.in >python-config.py

# Replace makefile compat. variable references with shell script compat. ones; ->

sed -e 's,\$($[A-Za-z0-9_]*$),\$\{\1\},g' < Misc/python-config.sh >python-config

# On Darwin, always use the python version of the script, the shell

# version doesn't use the compiler customizations that are provided

# in python (_osx_support.py).

if test `uname -s` = Darwin; then \

cp python-config.py python-config; \

fi

Any thoughts?

Daniel

Daniel Lecoanet

unread,

Apr 29, 2015, 5:11:06 PM4/29/15

to dedal...@googlegroups.com

Hi all,

Some progress... Since we were having problem with the c++ compiler, I decided to try using a gcc stack instead of an intel stack. I was able to install everything without any issues. In fact, Dedalus even runs! However, it crashes after only a few iterations.

For instance, if I run the 3D RB test problem from the examples on a [8,4] mesh on 32 processors (each SAVIO node has 20 processors, so this is across two nodes), I get:

[lecoanet@ln003 3d_rayleigh_benard]$ mpirun -np 32 python3 rayleigh_benard.py

[ln003.brc:70595] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70566] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70577] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70573] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70575] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70585] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70587] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70589] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70592] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70568] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70564] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70580] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70567] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70574] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70581] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70582] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70591] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70594] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70572] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70590] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70565] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70569] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70570] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70571] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70578] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70583] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70588] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70584] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70586] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70593] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70576] Error in mx_init (error No MX device entry in /dev.)

[ln003.brc:70579] Error in mx_init (error No MX device entry in /dev.)

--------------------------------------------------------------------------

WARNING: There is at least non-excluded one OpenFabrics device found,

but there are no active ports detected (or Open MPI was unable to use

them). This is most certainly not what you wanted. Check your

cables, subnet manager configuration, etc. The openib BTL will be

ignored for this job.

Local host: ln003

--------------------------------------------------------------------------

2015-04-29 14:07:44,950 pencil 0/32 INFO :: Building pencil matrix 1/128 (~1%) Elapsed: 0s, Remaining: 8s, Rate: 1.6e+01/s

2015-04-29 14:07:46,200 pencil 0/32 INFO :: Building pencil matrix 13/128 (~10%) Elapsed: 1s, Remaining: 12s, Rate: 9.9e+00/s

2015-04-29 14:07:47,279 pencil 0/32 INFO :: Building pencil matrix 26/128 (~20%) Elapsed: 2s, Remaining: 9s, Rate: 1.1e+01/s

[ln003.brc:70562] 31 more processes have sent help message help-mpi-btl-openib.txt / no active ports found

[ln003.brc:70562] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

2015-04-29 14:07:48,247 pencil 0/32 INFO :: Building pencil matrix 39/128 (~30%) Elapsed: 3s, Remaining: 8s, Rate: 1.2e+01/s

2015-04-29 14:07:49,093 pencil 0/32 INFO :: Building pencil matrix 52/128 (~41%) Elapsed: 4s, Remaining: 6s, Rate: 1.2e+01/s

2015-04-29 14:07:50,298 pencil 0/32 INFO :: Building pencil matrix 65/128 (~51%) Elapsed: 5s, Remaining: 5s, Rate: 1.2e+01/s

2015-04-29 14:07:51,174 pencil 0/32 INFO :: Building pencil matrix 78/128 (~61%) Elapsed: 6s, Remaining: 4s, Rate: 1.2e+01/s

2015-04-29 14:07:52,418 pencil 0/32 INFO :: Building pencil matrix 91/128 (~71%) Elapsed: 8s, Remaining: 3s, Rate: 1.2e+01/s

2015-04-29 14:07:53,213 pencil 0/32 INFO :: Building pencil matrix 104/128 (~81%) Elapsed: 8s, Remaining: 2s, Rate: 1.2e+01/s

2015-04-29 14:07:54,247 pencil 0/32 INFO :: Building pencil matrix 117/128 (~91%) Elapsed: 9s, Remaining: 1s, Rate: 1.3e+01/s

2015-04-29 14:07:54,956 pencil 0/32 INFO :: Building pencil matrix 124/128 (~97%) Elapsed: 10s, Remaining: 0s, Rate: 1.2e+01/s

2015-04-29 14:07:55,365 pencil 0/32 INFO :: Building pencil matrix 128/128 (~100%) Elapsed: 10s, Remaining: 0s, Rate: 1.2e+01/s

2015-04-29 14:07:55,373 __main__ 0/32 INFO :: Solver built

2015-04-29 14:07:57,190 __main__ 0/32 INFO :: Initialization time: 12.748865

2015-04-29 14:07:57,190 __main__ 0/32 INFO :: Starting loop

2015-04-29 14:07:58,547 __main__ 0/32 INFO :: Iteration: 1, Time: 1.000000e-04, dt: 1.000000e-04

2015-04-29 14:07:58,555 __main__ 0/32 INFO :: Max Re = 0.000000

2015-04-29 14:08:26,242 __main__ 0/32 INFO :: Iteration: 101, Time: 1.164755e+00, dt: 7.014343e-03

2015-04-29 14:08:26,250 __main__ 0/32 INFO :: Max Re = 16.144441

salloc: Relinquishing job allocation 305787

salloc: Job allocation 305787 has been revoked.

And the job is killed. Sometimes I get lucky and make it to 201 iterations before the job gets killed. Get the same issue on e.g., 4 processors on a single node.