Mpiexec gives error

[Ish Kaul]

Hi All,

I have a script hydro.py, which I am running with the command mpiexec -n 4 python hydro.py but it raises the following error.

raise ValueError("Objects are not all equal.")

ValueError: Objects are not all equal.

    raise ValueError("Objects are not all equal.")

    raise ValueError("Objects are not all equal.")

    raise ValueError("Objects are not all equal.")

ValueError: Objects are not all equal.

ValueError: Objects are not all equal.

ValueError: Objects are not all equal.

--------------------------------------------------------------------------

Primary job  terminated normally, but 1 process returned

a non-zero exit code. Per user-direction, the job has been aborted.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpiexec detected that one or more processes exited with non-zero status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[45628,1],0]

  Exit code:    1

 I'm not sure why thats happening. For reference I have copied here the hydro script as well.

# Import Statements                                                                                           

import numpy as np

import matplotlib.pyplot as plt

import h5py

from mpi4py import MPI

import time                                                                               

from dedalus import public as de

from dedalus.extras import flow_tools

import logging

logger = logging.getLogger(__name__)

                                         

Lx, Ly = (2., 2.)

nx, ny = (128, 128)

# Create bases and domain                                                                                   

x_basis = de.Fourier('x', nx, interval=(-Lx/2, Lx/2), dealias=3/2)

y_basis = de.Fourier('y', ny, interval=(-Ly/2, Ly/2), dealias=3/2) 

domain = de.Domain([x_basis, y_basis], grid_dtype=np.float64)

viscosity = 0

problem = de.IVP(domain, variables=['omega','psi', 'u', 'v'])

problem.parameters['nu'] = viscosity

problem.add_equation("dt(omega) -nu*dx(dx(omega)) - nu*dy(dy(omega))= dy(omega)*dx(psi) - dx(omega)*dy(psi) "\

)

problem.add_equation("-omega-dx(dx(psi)) - dy(dy(psi)) = 0", condition="(nx != 0) or (ny != 0)")

problem.add_equation("psi= 0", condition="(nx == 0) and (ny == 0)")

problem.add_equation("u + dy(psi)=0")

problem.add_equation("v - dx(psi) = 0")

# Build solver                                                                                                

ts = de.timesteppers.RK443

solver =  problem.build_solver(ts)

# State variables                                                                                             

x = domain.grid(0)

y = domain.grid(1)

omega = solver.state['omega']

psi = solver.state['psi']

u = solver.state['u']

v = solver.state['v']

# Intializing sigma and mu                                                                                   

sigma = 1

mu1 = 0.25

mu2=-0.25

# Initializing Gaussian monopoles on omega                                                                   

dst1 = (x-mu1)**2 + (y)**2

dst2 = (x-mu2)**2 + (y)**2

omega['g'] = np.exp(-(dst1/(sigma**2 /20))) + np.exp(-(dst2 / (sigma**2 /20)))

solver.stop_sim_time = 60.01

solver.stop_wall_time = np.inf

solver.stop_iteration = np.inf

dt = 0.01

x = domain.grid(0,scales=domain.dealias)

y = domain.grid(1,scales=domain.dealias)

start_time = time.time()

t = 0

while solver.ok:

solver.step(dt)

    t = t+dt

end_time = time.time()

Thanks,

Ish

[Daniel Lecoanet]

Hi Ish,

It seems like every processor is giving the same error (Objects are not all equal), so I would guess that it is not an mpi error. Do you get the same error if you run on a single core? Can you attach the hydro.py file? Which version of Dedalus are you using? (looks like v2)

Thanks,

Daniel

--
You received this message because you are subscribed to the Google Groups "Dedalus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/a9ba1d53-088b-4256-ae79-e5596b37c3d3n%40googlegroups.com.

[Ish Kaul]

Hi Daniel,

Yes I'm using v2 of dedalus, and it seems to run fine on one processor. I have attached the script.

You received this message because you are subscribed to a topic in the Google Groups "Dedalus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dedalus-users/3JGGt-gCICU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dedalus-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/32046F0C-8B60-4A70-92D5-2ADDEB190653%40northwestern.edu.

[Daniel Lecoanet]

Hi Ish,

Runs for me, see attached image. Since I ran only 4 cores, this is only showing 1/4 of the domain because each core is only plotting its own data. Similarly, when you calculate the energy and enstrophy, that is only for 1/4 of the domain. If you want to plot the full data, I would recommend you write data to disk using an output task.

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/CAGcFrHFJPQPuJ7VRbEurWMKM7hHnJR3QFZnwos%2BXDA31ujxVaw%40mail.gmail.com.
<hydro.py>

[Ish Kaul]

HI Daniel,

Thanks for the quick reply!

Since it does not work for me, does this mean there's a problem with my installation? I seem to recall I had trouble with the mpi4py library during the installation process.

Also, how exactly do you stitch the outputs together from different cores? Is there any example in the Dedalus github that does this?

Thank you,

Ish

[Daniel Lecoanet]

Hi Ish,

Yes, sounds like it might be an installation issue. Did you use the recommended conda installation?

Many of our example problems involve making plots from simulations run in parallel. For instance, you could look at this one:

https://dedalus-project.readthedocs.io/en/latest/pages/examples/ivp_2d_rayleigh_benard.html

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/5a83bd42-27a5-463e-ba54-bc57b28eb05dn%40googlegroups.com.

[Ish Kaul]

Hi Daniel,

After some changes, I tried mpirun and srun on a cluster with two processors and it seems the error is not with the mpi installation (since I used the preinstalled versions in the cluster). This was run with a slurm script. The computing help group mentioned that it is probably something with dedalus since the error points to something within the source code. Here is the full error output I get

2022-06-22 13:29:39,450 pencil 0/2 INFO :: Building pencil matrix 1/32 (~3%) Elapsed: 0s, Remaining: 1s, Rate: 3.7e+01/s

2022-06-22 13:29:39,525 pencil 0/2 INFO :: Building pencil matrix 4/32 (~12%) Elapsed: 0s, Remaining: 1s, Rate: 3.9e+01/s

2022-06-22 13:29:39,626 pencil 0/2 INFO :: Building pencil matrix 8/32 (~25%) Elapsed: 0s, Remaining: 1s, Rate: 3.9e+01/s

2022-06-22 13:29:39,725 pencil 0/2 INFO :: Building pencil matrix 12/32 (~38%) Elapsed: 0s, Remaining: 1s, Rate: 4.0e+01/s

2022-06-22 13:29:39,824 pencil 0/2 INFO :: Building pencil matrix 16/32 (~50%) Elapsed: 0s, Remaining: 0s, Rate: 4.0e+01/s

2022-06-22 13:29:39,923 pencil 0/2 INFO :: Building pencil matrix 20/32 (~62%) Elapsed: 0s, Remaining: 0s, Rate: 4.0e+01/s

2022-06-22 13:29:40,020 pencil 0/2 INFO :: Building pencil matrix 24/32 (~75%) Elapsed: 1s, Remaining: 0s, Rate: 4.0e+01/s

2022-06-22 13:29:40,119 pencil 0/2 INFO :: Building pencil matrix 28/32 (~88%) Elapsed: 1s, Remaining: 0s, Rate: 4.0e+01/s

2022-06-22 13:29:40,217 pencil 0/2 INFO :: Building pencil matrix 32/32 (~100%) Elapsed: 1s, Remaining: 0s, Rate: 4.0e+01/s

Traceback (most recent call last):

  File "hydro.py", line 85, in <module>

    solver.step(dt)

  File "/home/ikaul/.conda/envs/dedalus/lib/python3.8/site-packages/dedalus/core/solvers.py", line 518, in step

    path.increment(self.state.fields)

  File "/home/ikaul/.conda/envs/dedalus/lib/python3.8/site-packages/dedalus/core/distributor.py", line 441, in increment

    self.increment_group(*fields)

  File "/home/ikaul/.conda/envs/dedalus/lib/python3.8/site-packages/dedalus/core/distributor.py", line 490, in increment_group

    scales = unify(field.scales for field in fields)

  File "/home/ikaul/.conda/envs/dedalus/lib/python3.8/site-packages/dedalus/tools/general.py", line 74, in unify

    raise ValueError("Objects are not all equal.")

ValueError: Objects are not all equal.

Traceback (most recent call last):

  File "hydro.py", line 85, in <module>

    solver.step(dt)

  File "/home/ikaul/.conda/envs/dedalus/lib/python3.8/site-packages/dedalus/core/solvers.py", line 518, in step

    path.increment(self.state.fields)

  File "/home/ikaul/.conda/envs/dedalus/lib/python3.8/site-packages/dedalus/core/distributor.py", line 441, in increment

    self.increment_group(*fields)

  File "/home/ikaul/.conda/envs/dedalus/lib/python3.8/site-packages/dedalus/core/distributor.py", line 490, in increment_group

    scales = unify(field.scales for field in fields)

  File "/home/ikaul/.conda/envs/dedalus/lib/python3.8/site-packages/dedalus/tools/general.py", line 74, in unify

    raise ValueError("Objects are not all equal.")

ValueError: Objects are not all equal.

--------------------------------------------------------------------------

Primary job  terminated normally, but 1 process returned

a non-zero exit code. Per user-direction, the job has been aborted.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpirun detected that one or more processes exited with non-zero status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[25990,1],1]

  Exit code:    1

--------------------------------------------------------------------------

[Daniel Lecoanet]

Hi,

Sorry, I cannot replicate your error. The script runs fine for me in parallel. That suggests there is a difference in the software stack. I’m running on v2_master — what version are you using?

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/dd1d9d03-5acb-4c8d-92b3-1ef3b9b8a876n%40googlegroups.com.

[Ish Kaul]

Ah, I'm running v2.2006. I'll try to run this with v2_master also.

Ish

[Ish Kaul]

Okay I tried with v2_master on my pc: starting with no installation, I did curl . . . to install the conda install script, then ran the install script, activated the dedalus2 env, and did mpiexec for hydro.py. I still got the exact same error.

[Daniel Lecoanet]

Hi Ish,

Sorry I can’t help you because I cannot reproduce the error myself. Maybe someone else can try the script and reproduce your error.

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/CAGcFrHEpsAvoYC7SdbifGhqga%2B2vCu_cGnap2vuFK7zUmORnjQ%40mail.gmail.com.

[Keaton Burns]

Hi Ish,

This is a problem on v2.2006 that should be fixed on v2_master. The conda install script does not install this version, it installs the latest form pip. You’ll have to manually update to the tip of the git repo to get v2_master.

However, the problem can be avoided with a better equation formulation, by using substitutions for omega, u, and v instead of extra equations and variables. See the attached script for this form (which should run on v2.2006 and v2_master).  The only complication is that you have to manually convert your initial conditions for omega into initial conditions for psi by applying an inverse Laplacian.

Best,

-Keaton

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/7953EF2F-8361-46D3-B2D1-57C05C7181C2%40northwestern.edu.

[Ish Kaul]

Hi Keaton,

Thank you so much for the fix! The code works okay now.

Thank you,

Ish

To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/CADZXxBhOf4FvZbOrFU7sYq4eEag1uS6p%3DByG1_%2Bj%3DfbMcMsh-A%40mail.gmail.com.