dataframe2asc and transformations
Justin...@csiro.au
Hi All
I just finished a battle with dataframe2asc and thought I should share the issue in case anyone is having similar problems.
My issue:
I converted form asc to dataframe and analysed 10 years of landsat images (30m resolution) for the pilbara that were in UTM’s.
I then transformed the derived data frame from UTM to GDA94 lat lon using the rgdal package.
Once transformed I could no longer convert the files to asc using dataframe2asc in SDMTools.
Various errors were thrown including “too many elements” and :”nrow too large or NA by coercion” all of which point to memory issues when you look at the on line forums. In the first instance I reduced the data frame to 10 rows and asc2dataframe worked. However, I couldn’t get the function to work with anything bigger than 100 rows.
The remedy:
The actual issue was the data structure.
I checked the structure of the original file and it was “x”, “y”, “value” (int, int, numeric)
After the file was transformed to lat lon using rgdal the structure was “lat”, “lon”, “value” (numeric, numeric, numeric)
To remedy the issue I used as.integer to convert the lat and lon from numeric to integer and then asc2dataframe worked perfectly.
Cheers
Justin
Justin Perry
CSIRO Ecosystem Sciences
CSIRO
Phone: +61 7 4753 8554 | Mobile: 0408457607 | Fax: +61 7 4753 8600
justin...@csiro.au
|
www.csiro.au
|
www.csiro.au/people/Justin.Perry.html
Address:
ATSIP Bld 145 James Cook Drive, James Cook University Douglas Campus Qld 4811, Australia
Postal Address: Private Mail Bag, Aitkenvale Qld, 4814, Australia
PLEASE NOTE
The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent
permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference.
Please consider the environment before printing this email.
Helga Wiederhecker
Just for curiosity. Did you get the same number of columns and rows? What about the cell size did it change? It always happens to me and I end up having to cut and paste the asc headings.
Take care,
Helga
--
An R group for questions, tips and tricks relevant to spatial ecology and climate change.
All R questions welcome.
---
You received this message because you are subscribed to the Google Groups "Tropical R" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tropical-r+...@googlegroups.com.
To post to this group, send an email to tropi...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/tropical-r/2866BA20DF55534CBFF46562F14AD90113D8307B%40ExMBX01-CDC.nexus.csiro.au?hl=en-GB.
For more options, visit https://groups.google.com/groups/opt_out.
Justin...@csiro.au
Ah just realised, by converting to integer I drop off the precision thus reducing the size of the matrix due to less x,y combinations which works but produces two rows and two columns when it should be around 7000 and 7000. Back to the drawing board.
JP
Justin Perry
CSIRO Ecosystem Sciences
CSIRO
Phone: +61 7 4753 8554 | Mobile: 0408457607 | Fax: +61 7 4753 8600
justin...@csiro.au
|
www.csiro.au
|
www.csiro.au/people/Justin.Perry.html
Address:
ATSIP Bld 145 James Cook Drive, James Cook University Douglas Campus Qld 4811, Australia
Postal Address: Private Mail Bag, Aitkenvale Qld, 4814, Australia
PLEASE NOTE
The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent
permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference.
Please consider the environment before printing this email.
--
Helga Wiederhecker
Helga
To view this discussion on the web, visit https://groups.google.com/d/msgid/tropical-r/2866BA20DF55534CBFF46562F14AD90113D84193%40ExMBX01-CDC.nexus.csiro.au?hl=en-GB.
Lauren Hodgson
Will have a look into this in the morn.
To view this discussion on the web, visit https://groups.google.com/d/msgid/tropical-r/51A55206.8020208%40gmail.com?hl=en-GB.
Stewart Macdonald
Stewart
Justin...@csiro.au
Sent from my iPhone
To view this discussion on the web, visit https://groups.google.com/d/msgid/tropical-r/CAE_Q_205txM0gaevwOf0r_1E9pML63Zu9BkEi%2Bi0EMg7jds-AQ%40mail.gmail.com?hl=en-GB.
Justin...@csiro.au
Sent from my iPhone
Lauren Hodgson
To view this discussion on the web, visit https://groups.google.com/d/msgid/tropical-r/4A33FEDE-34EB-4A45-947D-06A4C02AFD9F%40csiro.au?hl=en-GB.
Justin...@csiro.au
Hi Lauren
Here’s the code I used to transform the UTM to LAT LON
library (rgdal)
outdir = "/home/ctbccr/Pilbara/csvFiles/p114r074"
csvdir = "/home/ctbccr/Pilbara/csvFiles/"
setwd (paste(csvdir,"p114r074", sep=""))
pnts = read.csv ("lm_p114r074.csv") # read in the linear model, structure is "y","x","intercept","coefficient"
#7663510,487920,-14.8863636363662,0.00763636363636494 with data for >360k rows (one landsat scene, 30m resolution)
sp.pnts = SpatialPoints(cbind(pnts[2:1]),proj4string=CRS("+init=epsg:32750")) # UTM WGS84 ZONE 50s - define the datam /zone
out = spTransform(sp.pnts, CRS("+init=epsg:4326")) # WGS84 GEO - transform the data
out=as.data.frame(out) # convert to data frame
out2 = data.frame (out[,1], out[,2], (round(pnts[,4], digits = 4))) # create the output dataframe - round the coeffecient to reduce grid size
names (out2) = c("lon", "lat","p114r074coeff")
rm (pnts) # remove the orgiginal object to free up memory
write.csv(out2, (paste(outdir, "coeffGDA94p114r074.csv", sep = "")),row.names=F) # write out the csv
Here’s the code I tried for converting the lat lon dataframe into an asc.
library (SDMTools)
outdirs = "/home/ctbccr/Pilbara/csvFiles/p114r074"
csvdir = "/home/ctbccr/Pilbara/csvFiles/"
setwd (paste(csvdir,"p114r074", sep=""))
pnts = read.csv ("coeffGDA94p111r074.csv")
x = pnts[,2]; y = pnts[,1]
pnts2 = data.frame (x, y, round (pnts[,3], digits = 3))
dataframe2asc(pnts2, filenames = "gda_coeffp114r074", outdir = outdirs, gz=F)
I ended up batching the dataframe2asc script on the UTM grids with no issues and then mosaiced the UTM grids and transformed the one mosaic raster in arc which worked fine just not as easy as running the script and having everything done at once.
cheers
Justin
Justin Perry
CSIRO Ecosystem Sciences
CSIRO
Phone: +61 7 4753 8554 | Mobile: 0408457607 | Fax: +61 7 4753 8600
justin...@csiro.au
|
www.csiro.au
|
www.csiro.au/people/Justin.Perry.html
Address:
ATSIP Bld 145 James Cook Drive, James Cook University Douglas Campus Qld 4811, Australia
Postal Address: Private Mail Bag, Aitkenvale Qld, 4814, Australia
PLEASE NOTE
The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent
permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference.
Please consider the environment before printing this email.
To view this discussion on the web, visit https://groups.google.com/d/msgid/tropical-r/CAE_Q_22xRvkmDHdQxYxf3L1%3DcwqsXBkGE8CF7GgiZmefvdW4Ag%40mail.gmail.com?hl=en-GB.
Justin...@csiro.au
Hi all
The final instalment of the asc2dataframe.
Stewart looked at the values and realised that when the sptransform function is used it incrementally increases the last digit when converted from integer to numeric.
Example, these aren’t real values just an example
This:
632554
632554
632554
632554
Is transformed to this:
22.00456
22.00456
22.00457
22.00457
As such instead of having 7000 rows and 7000 columns which is large but not beyond the capacity of R, I was asking R to create an ascii with 360000 rows and 360000 columns. Given that 40000 rows produced a matrix that is around 10 gig with my data, 10 times that amount of data was destined to fail.
So the answer lies in being able to round the last two digits to the same value without losing precision and maintaining the same cell size of 30m.
Cheers
And thanks Stewart for trouble shooting this.
JP
Justin Perry
CSIRO Ecosystem Sciences
CSIRO
Phone: +61 7 4753 8554 | Mobile: 0408457607 | Fax: +61 7 4753 8600
justin...@csiro.au
|
www.csiro.au
|
www.csiro.au/people/Justin.Perry.html
Address:
ATSIP Bld 145 James Cook Drive, James Cook University Douglas Campus Qld 4811, Australia
Postal Address: Private Mail Bag, Aitkenvale Qld, 4814, Australia
PLEASE NOTE
The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent
permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference.
Please consider the environment before printing this email.
From: tropi...@googlegroups.com [mailto:tropi...@googlegroups.com] On Behalf Of Lauren Hodgson
Sent: Thursday, 30 May 2013 1:20 PM
To: tropi...@googlegroups.com
Subject: Re: [Trop-R] RE: dataframe2asc and transformations
Can you post your code, Justin?
To view this discussion on the web, visit https://groups.google.com/d/msgid/tropical-r/CAE_Q_22xRvkmDHdQxYxf3L1%3DcwqsXBkGE8CF7GgiZmefvdW4Ag%40mail.gmail.com?hl=en-GB.