Merging output when solving an IVP in parallel

[Joseph Hall]

Hello,

I am attempting to animate the output of an initial value problem. Up until this point, the code has worked fine since I have been running the whole thing in serial on my own machine, but now I am distributing it across multiple cores to speed things up. It's possible that this is a very primitive way to get things done, but I have the output saving grid space to a set of numpy arrays that I then plot and turn into the frames of my animation. However, now that dedalus is slicing up the domain to run it in parallel, my code no longer works because only one slice is being animated, at least from what I can tell; the rendered videos are coming out as garbage.

How can I concatenate all of my local outputs into one file that I can then use for plotting the whole grid space? Apologies if this is a simple question, I have almost no experience with parallel computation.

(lines 189 onward in my code are where the trouble happens, I believe)

[Jeffrey S. Oishi]

Hi Joseph,

Dedalus has a dedicated output system for handling exactly this problem. Delete your code for saving lists (that's great for small problems run in memory in a notebook, but not so good for large runs in parallel):

if ivp_solver.iteration % 20 == 0:
        Vr_list.append(np.copy(Vr['g']))
        Er_list.append(np.copy(Er['g']))
        Ea_list.append(np.copy(Ea['g']))
        Ez_list.append(np.copy(Ez['g']))
        t_list.append(ivp_solver.sim_time)

and follow the example here:

https://github.com/DedalusProject/dedalus/blob/master/examples/ivp/2d_boundary_forced_waves/boundary_forced_waves.py#L81

on line 84, you see that the entire state vector is output to the hdf5 datafiles created by the analysis task. Dedalus writes a single hdf5 file per processor, but if you make sure to append the code here:

https://github.com/DedalusProject/dedalus/blob/master/examples/ivp/2d_boundary_forced_waves/boundary_forced_waves.py#L100

it will automatically create merged data files for you. You can then read the data files into a separate python plotting script using h5py. Dedalus provides some plotting tools to aid this process, with examples here:

https://github.com/DedalusProject/dedalus/blob/master/examples/ivp/2d_boundary_forced_waves/plot_slices.py

Doing so will vastly simplify your data analysis and also drastically reduce memory consumption.

Jeff

--
You received this message because you are subscribed to the Google Groups "Dedalus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dedalus-users/3f166073-d114-41dc-bd2e-9b7a8e0500d4n%40googlegroups.com.

[Joseph Hall]

Jeff,

Thank you for your help. I have attached an updated version of my code to help anyone who comes by this problem in the future; I had to do some things slightly differently to get it to run without errors.