parallelize nested for loop

[Juan Xie]

Dear Nimble group,

Is it possible to parallelize the nested for loop with NIMBLE?

Specifically, I have six regions (# locations are different across regions). And for each region, I have multiple sets of observations. I wanted to fit Gaussian process model for each set of observations for each region. Initially, for a given region, I parallelized the nimble pipeline for the multiple sets of observations by following the example at https://r-nimble.org/nimbleExamples/parallelizing_NIMBLE.html , and I repeated the process for all the regions separately. I was wondering if there is a way to parallel the process across regions?

Thanks!

[Chris Paciorek]

If I'm understanding correctly you want to fit 6 regions x 10 sets = 60 separate MCMCs? 

If so, you'll need to set it up as 60 separate tasks and then use the approach given in the "parallelizing nimble" example. 

Something like this:

```

chain_output <- parLapply(cl = this_cluster, X = 1:60,
                          fun = run_MCMC_allcode,
                          data = myData, code = myCode,
                          useWAIC = useWAIC)

```

`run_MCMC_allcode` would look something like this:

```

run_MCMC_allcode <- function(i, data, code, useWAIC = TRUE) {
  library(nimble)

  # determine region and set from flattened index

  region <- ((i-1) %/% 10) + 1

  set <- ((i-1) %% 10) + 1

  region <- i %% 10

   # then your code needs to appropriate adjust to the region and set...

 ....

}

```

-Chris

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/a1b49388-4d79-4054-9aa1-a41fd71653a4n%40googlegroups.com.

[Juan Xie]

Hi Chris,

Thank you very much for the help, really appreciated it!

I have another question: is it ok to use a  for loop to go over the regions, and within each region, use a parallelized code to model the multiple sets of observations?

Thanks!

[Chris Paciorek]

Yes, that should be fine, provided you have enough cores. You generally wouldn't want to have more tasks running at any one time than cores available so you'll want to consider that if you are going to start 6 different clusters with `makeCluster`.

-Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/68baea1f-23f0-4bdb-ade3-3ca37febd586n%40googlegroups.com.

[Juan Xie]

Got it, thank you for the clarification, very helpful!

Best~