ArcSWAT Text Output into RStudio

[Ryan Byrnes]

Anyone have any thoughts about how to import the ArcSWAT text file into R? The way Arc is generating the output text file is making importing it into R difficult.

Thanks

Ryan

[Jim Almendinger]

I assume you're talking about the output text files in the txtinout folder.  These are written by SWAT and not ArcGIS.  

This web site gives an example R script for reading the output.hru file:

http://santiago.begueria.es/2013/10/reading-swat-output-hru-files-in-r/

I learned a lot from this script.  As in all things R, it is detailed and esoteric but also efficient and powerful.  

Good luck,

-- Jim

---

--
You received this message because you are subscribed to the Google Groups "SWAT-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to swatuser+u...@googlegroups.com.
To post to this group, send email to swat...@googlegroups.com.
Visit this group at https://groups.google.com/group/swatuser.
For more options, visit https://groups.google.com/d/optout.

---

Spam
Not spam
Forget previous vote

--

---

James Almendinger, Ph.D. Director, St. Croix Watershed Research Station 16910 152nd St N, Marine on St Croix, MN  55047 e: jalmen...@smm.org o: 651-433-5953 x 11 https://www.smm.org/scwrs

[Ryan Byrnes]

Hi Jim,

Thanks for this reply too! I did find this today as well but am having some errors and warnings that I can't quite figure out. I'm not fluent in building/troubleshooting these kinds of functions and can likely figure it out but thought I'd post it back, maybe there's something simple I'm missing...

Code: 

# Read in output data as an HRU file #

file <- 'C:/Users/rbyrnes/Desktop/SWAT R Analysis/SWAT Output/output2.hru'

swat.hru.test <- swat_readOutputhru(file, col=w, ver=2012)

Output:

Error in Summary.factor(c(NA_integer_, NA_integer_, NA_integer_, NA_integer_,  : 

  ‘min’ not meaningful for factors

In addition: Warning messages:

1: In Ops.factor(res$MON, 12) : ‘<=’ not meaningful for factors

2: In Ops.factor(res$MON, 12) :

 Show Traceback

 

 Rerun with Debug

 Error in Summary.factor(c(NA_integer_, NA_integer_, NA_integer_, NA_integer_,  : 

  ‘min’ not meaningful for factors 

[Jim Almendinger]

If I remember, I used the script from the web page to learn the basics of reading the data into R, but I'm pretty sure I had to modify the details a bit, since it's entirely possible (likely) that the output.hru text file has changed with newer versions of SWAT since the R script was written.  It takes some detailed column-counting to get all the fields and field-widths just right.  Most columns are the same width, but there are a few that are different, and that will cause a fatal error if you don't catch it.  It can take (me at least) most of a day to work through a script like this to learn what each function does and to get the parameters correct for the exact format of your output.hru file.  

[Ryan Byrnes]

Jim,

Thanks again for the follow up. Yes, I'll likely need to do some more digging to get it to behave. In the meantime, a colleague suggested this simple solution:

Judging from the example data at: http://swat.tamu.edu/software/links/, the files are simple fixed-width/space separated formatted files with a header section. The two tricks are:

1) find the start of the table: you can do this with readLines and then grep to find the column names

2) read in the table with read.table

For the example output file attached, this is sufficient:

txt= readLines("output.hru")
i = grep("LULC",txt)[1]
df = read.table("output.hru",skip= i)

This works for now, need to rename the columns but that's easy and just make sure things are being imported properly. Once I get around to figuring out the function mentioned before, I'll post any adjustments I make to make it behave.

Ryan

[Willem Vervoort]

Hi Ryan and Jim,

I use the package data.table in R, which works well

Attached is a function for reading the sub and one for the rch file

Again, no guarantees, you will need to tinker and get familiar with R (see other help discussion groups on the web, I can recommend stack overflow)

Willem

To unsubscribe from this group and stop receiving emails from it, send an email to swatuser+unsubscribe@googlegroups.com.

[José Monteiro]

Dear Ryan,

Why don´t you read the output oyu want directly from the SWATOutput.mdb? You can find some hints on how to connect to an mdb file in R over here:

https://www.researchgate.net/publication/318110074_Overcoming_challenges_of_large-scale_SWAT_applications_with_R_Modelling_of_the_Amazon_basin.

Note that you need to use a 32 bit version of R. When you install R, both 64 and 32 version are installed, if you use RStudio you can switch between version easily.

Kind regards,
José

[Ryan Byrnes]

Hi Willem,

Thanks for this, I'll have a look at these scripts and see what I can glean. 

Ryan

[Ryan Byrnes]

Hi José,

Good idea, I'll ponder this as well, the main issue is that we are trying to get away from using anything large like a database file though as we are modeling the entire central valley of California irrigated lands and using SSURGO, hence a very large number of HRU's and subsequent rows and a text file may just be easier to handle but I'll give this a run too and see how it can work for us.

Thanks again!

Ryan

[Ryan Byrnes]

Hi Jim,

A bit of follow up with the function you provided, I've been dabbling with it a bit and have it mostly working but the remaining issue, and it's kind of big, is that the YEAR column created for the monthly output isn't populating correctly. Something is happening between lines 62-70, where the loop up to line 70 seems to be really messing things up. I'm not really sure how to fix it nor am I confident in what the loop is doing, any ideas or thoughts? I'm attaching my adapted script here. One more point, we are using the monthly output data and for our purposes, we need to preserve this, just letting you know because a simple way to get around this issue would be to use yearly data from the outset.

Thanks,

Ryan

[Peter Bishop]

Hi Ryan, 

I'm not sure if this is helpful - you'll have to check if it works, but I edited the code you mention above (santiago.begueria.es/2013/10/reading-swat-output-hru-files-in-r/) for SWAT 2012 recently. 

Hopefully it can help in some way or another. 

- Peter

[Jim Almendinger]

This looks like nice code (I appreciate the many comments -- very helpful.  Even the one that says **DON'T UNDERSTAND**.  I've always got a few of those).  When I reviewed my version of the code I downloaded from Begueria, I see I had changed a lot specific to my project, and just used it for annual output, so it's not generally useful for others. 

-- Jim

---

[Peter Bishop]

Yes, it is very possible that this code might be too personalised to my project - I haven't looked at it in a while. I mainly use the one I made for the output.STD file (but it is coded for the SWAT-MODFLOW output). 

Yes - this was my first time working with R, so there are a few "I don't understand" haha. 

- Peter

[Daniel Fuka]

Aloha Ryan,

Along with the R SWATmodel and EcoHydRology packages there are some routines for reading output.[hru,sol,etc] files. specifically readSWAT.R . One has been updated in the source code tree.

install.packages("SWATmodel")  

source("https://r-forge.r-project.org/scm/viewvc.php/*checkout*/pkg/SWATmodel/R/readSWAT.R?root=ecohydrology")

# the source is to update the readSWAT function.

[Daniel Fuka]

I just realized that the idiot that wrote this function didn't build a worthwhile man page for it. 

# if you have an ArcSWAT project "01493112v04"

library(SWATmodel)

source("https://r-forge.r-project.org/scm/viewvc.php/*checkout*/pkg/SWATmodel/R/readSWAT.R?root=ecohydrology")

setwd("~/junk/01493112v04")   # base directory of your project

hrudf=readSWAT("hru","Scenarios/Default/TxtInOut/")   # output.[hru,sub,

head(hrudf)

str(hrudf)

subdf=readSWAT("sub","Scenarios/Default/TxtInOut/")

rchdf=readSWAT("rch","Scenarios/Default/TxtInOut/")

str(rchdf)

# Note that there is a column "mdate" of type Date added to the other SWAT output columns for each

> class(hrudf$mdate)

[1] "Date"

[Palki Arora]

Hey Everyone,

I am running the below-mentioned code for accessing my .hru file with R programming. Earlier the code was working fine and I got .rch, .sub and .hru file from the same code but now I am getting the error: "Error in readSWAT("rch", "Scenarios/Default/TxtInOut/") : 

  unused argument ("Scenarios/Default/TxtInOut/")". My readSWAT function is also working fine. Could anyone please suggest me why all of a sudden this error came?

Thanks

Palki Arora

[Daniel Fuka]

Aloha Palki,
Can you make sure you have the latest version of readSWAT() and test?

source("https://r-forge.r-project.org/scm/viewvc.php/*checkout*/pkg/SWATmodel/R/readSWAT.R?root=ecohydrology")

readSWAT("rch", "Scenarios/Default/TxtInOut/")

Sorry I have not updated this yet... hoping to have a new version
release with some other fun stuff included.

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "SWAT-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/swatuser/I-_y24lQXnw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[ampar...@hotmail.com]

Hello!

I am trying to open  .sol and .mgt file in order to make some changes from R. Even using the source you commented, I cannot open them, just sub and rch file...
Is it just to poen output.  files? How can I open .mgt files in order to change them?


Thanks!

Amparo

[Willem Vervoort]

Dear Amparo,

these are really R questions not SWAT questions. You should look at all the resources associated with R.

Try using a search engine and: "read text file R". That should give you plenty of answers.

We should try an keep the SWAT google group to SWAT related questions

However, if there is lots of interest, there might be scope for a R and SWAT group, for example around the package Ecohydrology

Willem

--

You received this message because you are subscribed to the Google Groups "SWAT-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to swatuser+unsubscribe@googlegroups.com.

[Brian Hall]

I retrieved the .sol file by reading the access database table directly into R studio.  I use relative file addresses as shown by the "..\\TablesIn\\ etc.  

 My code looks like this:

install.packages("RODBC") #comment this line out after you install it the first time

library(RODBC)

channel <- odbcConnectAccess2007("..\\TablesIn\\Cocheco11FullHRUsAlteredUrbanFIMP.mdb") #@@@@ change the name to you own database

SOL.import<-sqlFetch(channel,"sol") #here you mention the table you want

odbcClose(channel)

# calc means for SOL_AWC by soil layer.

HydricClass.A <- subset (SOL.import, HYDGRP == "A")

HydricClass.B <- subset (SOL.import, HYDGRP == "B")

HydricClass.C <- subset (SOL.import, HYDGRP == "C")

HydricClass.D <- subset (SOL.import, HYDGRP == "D")

I used similar code to import the mgt1 Access Database table.

 Good luck,

Brian Hall

[Daniel Fuka]

Aloha Amparo,

That readSWAT() function is only for reading the output.[sub,hru,rch] files with daily output. Your note made me realize I should name it something that clarifies that. 

Per Brian's reply, it is easiest to get this info from the mdb files, either using RODBC (in Windows) or Hmisc (in Mac/Unix). In the EcoHydRology package, there are some

functions that read and write initialization files for calibration with DEoptim, and there is a release coming up that makes these functions a bit more clear. 

#See:

library(SWATmodel)

?calib_swat_ex

Note that the get_cfsr_latlon() service currently has an outage. 

If there is a desire to use this to run swat, I can help you make an example that will fit your use case. 

[ampar...@hotmail.com]

Thanks!

[Joanna D.]

Hi!

I'm wondering if there are any new developments in terms of packages or code available for opening the output files in R? I'm mostly interested in output.rch.

[Daniel Fuka]

While the CRAN SWATmodel R package has been archived, grabbing the readSWAT function from the package repository still works in R: 

source("https://r-forge.r-project.org/scm/viewvc.php/*checkout*/pkg/SWATmodel/R/readSWAT.R?root=ecohydrology")

readSWAT("rch", "Scenarios/Default/TxtInOut/")

[summera khan]

@DANIEL FUKA

the link gives the following result

An Exception Has Occurred

Unknown location: /*checkout*/pkg/SWATmodel/R/readSWAT.R

HTTP Response Status

404 Not Found