Allocation problems for too many receivers

[Felix]

Dear Salvus group,

I have some problems when I try to run a simulation with many receivers (1000 or more).
For only 100 receivers  everything was fine but when I increase the number of receivers
densifying the same area the run stops directly after the problem's initialization.

My output is :

====================================================
Salvus version 0.0.1-801-g6d1e549
Floating point size: 32
Compiled for GLL orders: 4
Running on 1 rank(s).
====================================================

--source-time-delay not set. Will auto-calculate.
Initializing problem.
Timestep set by user as: 2e-05 s.
Automated timestep would be set to: 1.94912e-05 s.
Begin time loop.
Can't allocate memory for WavefieldContainer. Use --memory-for-IO-per-rank.


I tried to add --memory-for-IO-per-rank,but it seem like the option expects some number?

For 100 receiver salvus allocates 120M for the receivers.h5 file , so I would have expected salvus to allocate 1.2G for 1000 receivers.
On my ext4-disk is plenty of space (2.6T), so this should not be the problem.

Any ideas?

Thank you, Felix

[Christian Boehm]

Hi Felix,

Thanks for your email! We used to have a bug in handling the receiver output, but this should have been fixed in the current version of the master branch.

I also noticed that the error message printed a deprecated option name and I have just changed this as well.

After a `git pull` and rebuilding Salvus everything hopefully works properly again. The current version is 812 (gf1b06de)

Some background information:

In order to avoid frequent disk access (which can be very slow on HPC systems) Salvus internally buffers all output and only dumps intermediate data chunks when necessary. 

You can use the option `--io-memory-per-rank-in-MB` (this is now the correct name) to specify the size of the buffer per MPI rank. The default is 1000 MB.

If the problem persists, you might try to reduce the buffer size, e.g., using

`--io-memory-per-rank-in-MB 100`.

Moreover, we will have a new release next week with several improvements so I recommend to either wait or to update again next week.

Hope that helps! 

Cheers

christian

--
You received this message because you are subscribed to the Google Groups "Salvus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salvus+unsubscribe@googlegroups.com.
To post to this group, send email to sal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salvus/3265dd63-e904-44c2-a52d-c5877297f519%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.