Future 95PPU in LSU scale

[Payam]

 Hi Dr. Nguyen and all,

I am trying to use HPCs to run my calibrated SWAT+ model for the future climate dataset and extract the 95 PPU ranges for all the grid cells that I have in my model (32,000 LSUs) and map them. Since I am running the model for the future, clearly I do not need to implement an objective function because there is literally no observed data to run the objective function against. All I need is to extract the LSU-scaled 95 PPUs for all the grid cells that I have through a 100-simulation iteration. Could you please give me some advice on how I can update the 'RSWAT_script_SUFI2' code to run on HPCs and achieve my goal?  

[Tam Nguyen]

could  you  share with me  the RSWATproject.rds,  I can  make an example  code and share the  code here

[Payam]

Hi Dr. Nguyen,

Thanks for getting back to me.

"RSWATproject.rds" could be found attached.

[Tam Nguyen]

Now you could install RSWAT package (https://github.com/tamnva/R-SWAT/tree/master) and then you will see the script for doing this in RSWAT vignettes (look like the attached file)

(Must install RSWAT to run this script)

[Payam]

Thank you so much, Dr. Nguyen, for sending the script. Just a small correction that I found needs to be applied in the 'readAllOutVar' function. In line 48 of the code ("" for (ifolder in 1:length(coreFolder)){""), 'coreFolder' should actually be 'coreFolders', where the last 's' was omitted. I made the adjustment on my end, and it worked perfectly.

I have another question regarding the extraction of the 95PPU for the number of subbasins. Currently, the code only extracts the 95PPU for one subbasin (Reach = c("1")). How can we modify it to extract the 95PPU for all the units specified with year, month, day, and unit number in the output file?

Your assistance is greatly appreciated.

[Tam Nguyen]

Thanks for spotting this mistake. I have fixed this issue and update the GitHub repo (reinstall RSWAT package to update this)

If you want to extract for 100 reaches (just example) you need to replace line 54 in this file https://github.com/tamnva/R-SWAT/blob/master/vignettes/run_SWATplus_user_parameters.R with

Reach = paste(as.character(c(1:100)), collapse  = ",")

For this question: "...for all the units specified with year, month, day, and unit number in the output file?" I am not clear what do you mean, could you give the example data file here

Tam

[Payam]

I tried updating 'Reach = paste(as.character(c(1:100)), collapse = ",")' and extracted data from 'lsunit_wb_mon.txt' for the years 2003 to 2004 (basically 'numTimesteps <- c(24)'). However, I encountered an error in the 'output <- readAllOutVar(workingFolder, numTimesteps)' section of the code. Here's the error:

Error in matrix(data[, 1], nrow = numTimesteps[var] + 1) :
invalid 'nrow' value (too large or NA).

Regarding the example data file you mentioned, I included a sample with two variables (let's say ET and PET) and two units over the two years. This represents the format I'm aiming to extract. Please ignore the actual numbers; they're just for illustrative purposes."

[Tam Nguyen]

if you did for 100 reaches, your numTimesteps should be a vector with a length of 100, each element of this vector indicates the number of output timesteps of each variable. For examples incase you output timesteps are the same for all 100 reach, use

numTimesteps <- rep(24,100)

[Payam]

I tried that too. But still doesn't like it (I am extracting from "lsunit_wb_mon.txt" file for the period of ("2002-01-01", "2003-12-31")) for 100 reaches:

numTimesteps <- rep(24,100)
output <- readAllOutVar(workingFolder, numTimesteps)

Error:  Error in output[[var]] : subscript out of bounds

[Tam Nguyen]

There is a bug in the readAllOutVar function, I have fixed this and updated the code, it should work now

[Payam]

Hi Dr. Nguyen,

Thank you so much for solving the issue. it's working now.

I have another question about running your code in R instead of RStudio. you have mentioned in the R-SWAT github that "if you run in R, package vignettes cannot be built". I am wondering if this version of 95ppu code could be run in R as well since I need to run the code in HPCs. 

if yes, how "package vignettes" can be created?

[Tam Nguyen]

package vignettes are a kind of documentation of the package. This DOES NOT affect the functionalities of the package. You can install using R and still use all functions of the package. I have update the README - please see detail there

https://github.com/tamnva/R-SWAT

[Payam]

Hi Dr. Nguyen,

Thank you for your explanation. I could successfully install the ""RSWAT" library on the Linux cluster. However, the only issue that exists now is executing the SWAT+ executable file. Your script creates multiple copies of the TextInOut folder with the SWAT+ executable file included in the 'workingFolder'. However,  the following error is showed up when it comes to running them even though I have switched to the Linux environment version of the executable file :

Error : "sh: line 1: /home/R-SWAT-master/EXE_File/rev60.5.4_64rel.exe: Permission denied"

Do you happen to see such an error before?

[Tam Nguyen]

actually I haven't tested with linux. In linux the executable SWAT+ does not have an extension .exe. I will check it next week. You can check this out this week, something relates to calling this file in RSWAT (lines 185-189) in this file https://github.com/tamnva/R-SWAT/blob/master/R/runSWATpar.R

[Tam Nguyen]

it should work now on linux. I have updated the code

[Payam]

Hi Dr. Nguyen,

I was able to use the previous version of the code using the Linux format of SWAT+ model (rev60.5.4_64rel_linux). Can you please elaborate on what is the main difference of the current version of the code with the previous one?

[Tam Nguyen]

There is a very small changes. Example in window I have the SWAT executable file is "swat.exe" and linux the SWAT executable file is "rev60.5.4_64rel_linux ". Calling these executable file different in window and linux

in Window:    system("swat.exe")

in Linux:         system("./rev60.5.4_64rel_linux ")

In the code line 189-193 were change from:

previous version:

system(trimws(exeFile[length(exeFile)]))

to current version:

    if(.Platform$OS.type == "unix") {
      system(paste0("./", trimws(exeFile[length(exeFile)])))
    } else {
      system(trimws(exeFile[length(exeFile)]))
    }

you can see the updated file here https://github.com/tamnva/R-SWAT/blob/master/R/runSWATpar.R (L189-193)

[Payam]

Hi Dr. Nguyen,

I hope you are doing well!

I was able to run SWAT+ in the Linux cluster using the previous version of the R-SWAT-master depository which I had installed. Yesterday, I updated the R-SWAT library and the R-SWAT-master folder and this time the script that I was using can not execute "rev60.5.4_64rel_linux" file and I am hitting the following error. I am wondering if you have changed each of the codes recently since I was able to run SWAT+ using the previous version of the library.

> runSWATpar(workingFolder,TxtInOutFolder,outputExtraction,ncores,SWATexeFile,
+           parameterValue,paraSelection,caliParam,copyUnchangeFiles,fileCioInfo,
+           dateRangeCali,firstRun)
sh: line 1: .//home/s_jalalija/R-SWAT-master/data/rev60.5.4_64rel_linux: No such file or directory
sh: line 1: .//home/s_jalalija/R-SWAT-master/data/rev60.5.4_64rel_linux: No such file or directory
sh: line 1: .//home/s_jalalija/R-SWAT-master/data/rev60.5.4_64rel_linux: No such file or directory
sh: line 1: .//home/s_jalalija/R-SWAT-master/data/rev60.5.4_64rel_linux: No such file or directory
sh: line 1: .//home/s_jalalija/R-SWAT-master/data/rev60.5.4_64rel_linux: No such file or directory
sh: line 1: .//home/s_jalalija/R-SWAT-master/data/rev60.5.4_64rel_linux: No such file or directory
sh: line 1: .//home/s_jalalija/R-SWAT-master/data/rev60.5.4_64rel_linux: No such file or directory
sh: line 1: .//home/s_jalalija/R-SWAT-master/data/rev60.5.4_64rel_linux: No such file or directory

Error in { : task 1 failed - "cannot open the connection"
Calls: runSWATpar -> %dopar% -> <Anonymous>
Execution halted

[Tam Nguyen]

Hi Payam, it should work now, before there was an error because the file path in windows and unix are different which I didn't know, Now I have fixed that and test on unix, it runs without issue

Note, remember to set writting right to your whole folder so SWAT+ can run and write output files there