Speeding up my code

[Supply Chain Forecasting]

I've an interesting challenge to speed up my code in the simplest way possible.  I'm looping through hundreds/thousands of time series and performing rolling origin forecasting on each time series.  So this means multiple origins, multiple time series and multiple forecasting methods (eg exponential smoothing, ARIMA, stepwise regression, etc.)

Currently I'm making use of the cores available to me and multi-threading via the foreach package so I can run 8 simultaneous time series at a time (16 in parallel on the university cluster, potentially many more if I use multiple nodes).  My questions are related to speeding up the task:

1) would moving from out-of-the-box R to MRO be of benefit to me?  Is this because the MKL extensions make the calculations perform faster on a SINGLE core or because MKL enables parallel use of the cores?  I imagine some code profiling would be required to highlight exactly where my code is running slowest (e.g. solve, optim, lm)

2) I have access to a cluster where Intel Xeon Phis (MICs) are available.  Would the foreach package play nicely with the MIC or would it struggle with running 240 instances of R in parallel?  Does MRO play nicely with the MIC co-processor?  Do you have any case studies or documentation of using the MIC to run more simultaneous threads on MRO?

Any other suggestions you would make to advise me on optimisation of this code?

Many thanks in advance,

Matt

[adev...@microsoft.com]

Interesting question.

The Intel MKL will speed up any BLAS operations, e.g. matrix multiplication, cholesky decomposition, etc. So solve() might run faster, but optim() probably won't (depending on what you're trying to optimise). One would think that lm() also falls into this category, but the lm() code does not call out to any BLAS functions, so it will be the same speed.

The foreach package does not have a parallel backend for Xeon Phi, as far as I know.

However, the MKL does support Xeon Phi, as this blog post explains: http://blog.revolutionanalytics.com/2015/05/behold-the-power-of-parallel.html, but beware, the author effectively says "here be dragons".

I hope this helps.

Good luck with your project!

Andrie