Reducing runtime and increasing CPU efficiency by parallelizing InMAP code?

[Emma Rieves]

Hi fellow InMAPpers,

I'm writing because I'm (hoping to run) running CONUS-wide simulations using InMAP with 1 x 1 km input emissions and outputting results to a 1 x 1 km grid. The problem is that, even using a supercomputer, these simulations are unable to complete because of the runtime limits for each node; they also have very low CPU efficiency.

I've been thinking that parallelizing the InMAP code could be a solution to make these high resolution scenarios more efficient, but am unfamiliar with the entire InMAP code base. Can anyone point me to scripts/functions/etc. in the InMAP code that could be good candidates for parallelization? Let me know if you have suggestions or further clarifying questions.

Emma

[Chris Tessum]

Hi Emma,

It does currently run the main computations in parallel: https://github.com/spatialmodel/inmap/blob/master/run.go#L85 . IIRC last time I checked it had pretty good scaling efficiency up to 8 CPU cores, and then diminishing returns after that, but we often run it on 40-core CPUs on our cluster. It currently isn't able to do MPI parallelization, but that would be a significant undertaking. If it currently is only working for you on one CPU core, you could try setting the GOMAXPROCS environment variable.

Chris

--
You received this message because you are subscribed to the Google Groups "inmap-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inmap-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/inmap-users/1c46a1af-7250-4e28-9253-86dc6d2b1a25n%40googlegroups.com.

[Emma Rieves]

Hi Chris,

Thanks for your prompt response. We're running things on a 47-core CPU, but it appears to have very low CPU efficiency. We tried adjusting the GOMAXPROCS environment variable to see if this improved things, but it did not make a difference. We adjusted it within the run.go file (see screenshot below); is this what you were envisioning in terms of setting it, or do you recommend something else?

Also, we are having memory issues. We're running things with 1 TB of memory, but keep having jobs fail due to out-of-memory errors. We are trying to create a 4 x 2 x 1 dynamic grid across CONUS, but this is taking up to 973.89 GB of memory before failing. Do you have any suggestions on how we can reduce the memory usage? Could there potentially be a memory leak?

Any insights you can share would be very helpful.

Thanks,

Emma

[Broderik Craig]

If you're working with an HPC, you can allocate multiple CPUs to your job. In my SLURM batch script I included 

#SBATCH --cpus-per-task=8

export OMP_NUM_THREADS=8

and got a much faster execution.

Broderik

On Tuesday, November 5, 2024 at 3:34:34 PM UTC-7 Chris Tessum wrote:

[Chris Tessum]

Hi Emma,

The grid creation algorithm is not super computationally efficient, so that may be the issue. Do you mean that the grid will be 4km for the largest grid cells? That would probably be more grid cells than we have successfully created with that algorithm. It does a lot of computational geometry to figure out how many people would be in each potential grid cell before splitting the grid cells, which generates a lot of objects that need to be garbage collected, which is why you can get large spikes in memory usage. Also, some parts of the grid creation algorithm are parallelized, but I think it spends a lot of time doing things in serial. I haven't really looked into why that is or if it could be sped up. You might be better off creating the grid first and then running it with a static grid, but even then it might be a lot for that algorithm to handle.

Chris

[Emma Rieves]

Hi all,

@Broderik Craig: thanks for the suggestion. I tried it but it didn't fix my problem.

@Chris Tessum: That explanation all makes sense. Initially, we were trying to do a 4x2x1 grid across CONUS (where the largest grid cell was 4 km), but the runs would fail due to (1) out of memory issues or (2) running out of runtime on the supercomputer (runs are limited to 167 hours). We're now trying to do a 20x4x2x1 grid (where the largest grid cell is 20 km) for just the western half of CONUS.

Now, the run is "working" but we're having an issue with the .gob file saving (error screenshot attached below). It seems like the run is now completing, but InMAP is unable to save the file. What does this error message mean (I'm not super familiar with go)? Do you have any suggestions on how to address it?

Thanks!

[Chris Tessum]

Hi Emma,

Unfortunately, there is a hard-coded 8gb file size limit for that file format. (If we had known this, we probably would have used a different format.) I think would you would have to do is to change the size limit here: https://github.com/golang/go/blob/b50ccef67a5cd4a2919131cfeb6f3a21d6742385/src/encoding/gob/decoder.go#L19 and then recompile the go compiler, and then use that version to compile InMAP. Alternatively, we'd be happy to accept code that gave InMAP an option to use a different file format :)

Best regards,

Chris