Parallel Processing and Optimal Hardware?

[Jaison Renkenberger]

Best hardware configuration? 

I am working with my research group to build a server to do a variety of research operations. One of these includes running SWAT-CUP with the parallel processing module. Does anyone have a good hardware setup/configuration that they would like to share to give me an idea on what to request? Thanks in advance!

Upon the reading of the parallel processing paper and having and IT background..... Here is where I am at. 

What I do know:

SWAT-CUP can "see" hyper threaded (HT) or virtual cores. So, for example, a Xenon (intel) processor with 10 physical cores and 20 HT cores means I can theoretically run 20 instances of swat-cup. Provided that I didn't care to run anything else. Having a server with two of these processors means a possible 40 HT instances of swat-cup.

What I don't know:

I have thoughts and questions to those that might know more about this. 

1) From what I read, the number of parameters that are being changed directly impacts how much ram is being used. Is this correct?

2) What is the relationship between RAM usage and the number of HRU's (essentially the size of the project)? It isn't clear.

3) Is caching and synchronization done solely on HDD's? What takes place in RAM vs on HDDs?

4) What are the HDD specs used in the experiments (i.e., SAS, SATA, IDE, Partitions, RAID 0 and etc...)

While the paper was somewhat informative I found that the lack of information on HDD specs and various possible configurations disheartening. When considering computing performance with any application CPU, RAM and HDD configurations should all be considered at the very least. This paper hardly mentions the third major component (HDD configuration). 

When looking at the portion of the figure (below) from the parallelization paper, all I see is where the HDD bottleneck is but without information on the HDDs. Dr. Abbaspour? Maybe you have this information somewhere?

A parallelization framework for calibration of hydrological models

E. Rouholahnejad a,*, K.C. Abbaspour a, M. Vejdani b, R. Srinivasan c, R. Schulin d, A. Lehmann e

a Eawag, Swiss Federal Institute of Aquatic Science and Technology, Ueberlandstrasse 133, CH-8600 Duebendorf, Switzerland

b Neprash Technology, 1625 Sundew Pl, Coquitlam, B.C., V3E 2Y4, Canada

c Spatial Sciences Laboratory, Texas A&M University, Texas Agricultural, Experimental Station, College Station, TX, USA

d ETH Zürich Institute of Terrestrial Ecosystem, Universitätstr. 16, 8092 Zürich, Switzerland

e University of Geneva, Climatic Change and Climate Impacts, 7 Route de Drize, CH-1227 Carouge, Switzerland

Lastly (A warning!!!!) - DO NOT USE SSDs for SWAT-CUP Calibration,

At the end of the paper this is lightly mentioned as a possible suggestion for improving performance. While SSDs are significantly faster than a traditional HDD, and even shows some performance over SAS drives, SSDs have a limited read/write life. Because SWAT can easily contain tens of thousands of text files that are written too many times per iteration, the life of your SSD will be dramatically shortened. This problem is exacerbated by the fact that writing to a SSD first requires the information to specifically be erased. Some PCs also contain hybrid SSDs and traditional metallic disks. These should also be avoided. 

Thanks in advance for any help!

Respectfully,

Jaison Renkenberger