Merging Virtual Sets

[Oliver Brown]

Hi there,

I am trying to merge my set files into one large file; I want to combine my analysis outputs from my entire simulation into one analysis.h5 file (virtually merged or not). I know that dedalus automatically virtually merges processor files into sets, but I'm struggling with combining all of those sets. In v2 it was pretty simple, I just used:

srun python3 -m dedalus merge_sets analysis/analysis.h5 analysis/*.h5

which would combine all of the sets into one file. I am not understanding the docs page for the post module (https://dedalus-project.readthedocs.io/en/latest/autoapi/dedalus/tools/post/index.html), and the few things I have tried (merge_virtual_analysis, merge_virtual_file, merge_analysis) have not worked.

To make sure what I am asking is clear, here is an example folder structure:

analysis

     - analysis_s1

          - analysis_s1_p0.h5

          - analysis_s1_p1.h5

     - analysis_s2

          - analysis_s2_p0.h5

          - analysis_s2_p1.h5

     - analysis_s1.h5

     - analysis_s2.h5

And I wish to combine the virtually merged analysis_s1.h5 and analysis_s2.h5 files into one file, analysis.h5. Is there a simple way to do this that I am missing? Any help is greatly appreciated, thanks so much!

Oliver

[Yi-De Liou]

You can store the results in a single file by enabling the 'mpio' option:

snapshots = solver.evaluator.add_file_handler(path_out, sim_dt=sim_dt, max_writes=30, parallel='mpio')

Similarly, for existing virtual files, you can combine them into a single file by loading each dataset and re-evaluating it into a new output file.

# load

hdf5 = h5py.File(path, mode='r')
f.load_from_hdf5(hdf5, -1, 'f')
f.change_scales(dealias)

# save

from dedalus.core.evaluator import Evaluator
evaluator = Evaluator(dist, locals())
snapshots = evaluator.add_file_handler(path_out, parallel='mpio')
snapshots.add_task(f, scales=dealias)
evaluator.evaluate_handlers([snapshots], iteration=10)

Oliver Brown <ob58...@gmail.com> 於 2025年10月17日 週五 下午12:00寫道：

--
You received this message because you are subscribed to the Google Groups "Dedalus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dedalus-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dedalus-users/ccc19bb8-40bc-4f21-8052-e5bb5b0fe093n%40googlegroups.com.