Survey report

[Josh K]

Good afternoon,

I'm new to camtrapR and had a question about Survey reports. In my lab we have a dataset of about 57000 images and when we processed events, we labeled every image in the event with the total amount of individuals, so for example if we have a 75 image event of White lipped peccaries, we would label all 75 images with the total amount of individuals, so image 1-75 would all show 35 individuals. I'm just curious if this would effect the survey report, and if it would solutions to fix it. Thank you!

Best,

Josh K.

[Juergen Niedballa]

Hi Josh,

I don't think these tags would affect surveyReport in any way. Did you encounter a problem, since you asked for solutions to fix it? Would you mind elaborating what exactly needs to be fixed?

surveyReport uses a record table that has been prepared by recordTable() already. That means, surveyReport doesn't call exiftool to extract any tags, and it doesn't do any temporal filtering or whatnot. It uses the provided record table as it is, only reading date/time, station and species ID from the record table. 

If you create an output zip file with makezip = TRUE your record table (with tags and all) will be saved in the zip file.

Best,

Jürgen

[Josh K]

Thank you for the reply! There wasn't a problem with the survey reports, I am new to the camtrapR package. After reading the camtrapR pdf., would i be able to use a raw data export csv to create a record table? I apologize if these are simple questions, I'm just trying to get a better understanding of the program and of potential issues we may run into. Thank you!

Best,

Josh K.

[Josh K]

Attached to this message is our existing excel file that I'm trying to use as the Record table. Also for creating the survey report, I'm getting this error.  CostaRT$DateTimeOriginal <- strptime(paste(as.Date(CostaRT$Date, format = "%m/%d/%y"), 

+                                     CostaRT$Time), format = "%Y-%m-%d %H:%M:%S", tz = "UTC")

> reportTest <- surveyReport (recordTable          = CostaRT,

+                             CTtable              = costaCT,

+                             cameraCol            = "Camera",

+                             speciesCol           = "Species",

+                             stationCol           = "Station",

+                             setupCol             = "Setup_date",

+                             retrievalCol         = "Retrieval_date",

+                             CTDateFormat         = "%d/%m/%Y", 

+                             recordDateTimeCol    = "Date.Time",

+                             recordDateTimeFormat = "%Y-%m-%d %H:%M:%S",

+                             CTHasProblems        = TRUE)

Error: Cannot read datetime format in c("6/3/2019 8:02", "3/28/2019 9:35", "6/11/2019 8:49", "1/17/2019 11:57",  . Output is all NA.Cannot read datetime format in "4/4/2019 11:00", "3/10/2019 8:56", "1/26/2019 8:30", "5/31/2019 19:42",  . Output is all NA.Cannot read datetime format in "8/3/2019 8:00", "8/19/2019 8:22", "4/24/2019 7:03", "5/18/2019 13:10",  . Output is all NA.Cannot read datetime format in "4/11/2019 19:21", "9/10/2019 19:43", "9/29/2019 11:00", "8/26/2019 14:36",  . Output is all NA.Cannot read datetime format in "3/13/2019 23:52", "6/15/2019 9:55", "3/10/2019 11:00", "1/27/2019 15:08",  . Output is all NA.Cannot read datetime format in "6/18/2019 11:00", "5/6/2019 10:43", "8/7/2019 10:22", "1/25/2019 11:36",  . Output is all NA.Cannot read datetime format in "7/9/2019 19:39", "4/23/2019 11:43", "7/14/2019 19:49", "10/25/2019 13:40",  . Output is all NA.Cannot read datetime format in "5/4/2019 0:35", "9/11/2019 10:09", "3/12/2019 8:26", "6/22/2019 14:10",  . Output i

As i said in the previous message this is my first time using this package so I honestly am unsure if I'm missing anything. Also for determining Independent events, I've seen message threads talking about the Delta Time, but I'm unsure how to properly do that as well. We are trying to Classify Independent Events as 30 min. Thank you again for making this program!

[Juergen Niedballa]

Hi Josh,

the scary long error message:

Error: Cannot read datetime format in c("6/3/2019 8:02", "3/28/2019 9:35", "6/11/2019 8:49", "1/17/2019 11:57",  . Output is all NA. ...

is due to a bug in the current version. I'm sorry. Please try the development version from github, you should get a proper error message then (code to install at the bottom of this post).

It should read: Error: Cannot read datetime format in recordTable [, recordDateTimeCol], meaning that the argument:

recordDateTimeFormat = "%Y-%m-%d %H:%M:%S",

is not specified correctly (it doesn't match the actual format of the column "Date.Time" in the record table). In your case the format is e.g. "3/28/2019 9:35", so the argument should be:

recordDateTimeFormat = "%m/%d/%Y %H:%M",

or, if the lubridate package is available:

recordDateTimeFormat = "mdy HM",

Please note, I haven't tested this code, but I believe it should work.

In your record table you have genus and species a separate columns. You'll need to combine them into one column with paste(). Otherwise 2 species from different genus but same species epithet (e.g. "GenusA punctata", "GenusB punctata") would be considered you use speciesCol = "Species".

Regarding the independent events, it is not implemented as a standalone function yet and there's only the workaround found on some threads on this group. In your case, something like this:

# create DateTimeOrginal column in proper format

CostaRT$DateTimeOriginal <- strptime(CostaRT$Date.Time, format = "%Y-%m-%d %H:%M:%S", tz = "America/Bogota")  # change time zone as needed (see OlsonNames()  )

# filter records
records_filter30_min <- assessTemporalIndependence(intable = CostaRT,

                                                  deltaTimeComparedTo = "lastIndependentRecord",
                                                  columnOfInterest = "Species",             # change this to name of your combined genus + species column

                                                  stationCol = "Station",
                                                  minDeltaTime = 30,

                                                  camerasIndependent = FALSE

)

argument camerasIndependent is confusing here. In this context, it means that there is no camera column in the input data (only a station column). Main reason is that this is an internal function that is forced to work outside its natural habitat here, because I haven't written a proper standalone function to filter records yet. 

Let me know how it goes.

Best,

Jürgen

# install camtrapR development version from github

library(devtools)
install_github("jniedballa/camtrapR")

[Josh K]

Thank you! Where do we put the paste()? the Survey Report worked, but the paste function I tried did not. I tried this using tidyverse   CostaRT <- CostaRT %>% mutate(Species=paste(Genus,Species,sep='_')).

[Josh K]

Also when trying to run the Independent Events, i get this error.

records_filter30_min <- assessTemporalIndependence(intable = CostaRT,

+                                                    deltaTimeComparedTo = "lastIndependentRecord",

+                                                    columnOfInterest = "Species",             

+                                                    stationCol = "Station",

+                                                    minDeltaTime = 30,

+                                                    camerasIndependent = FALSE )

Error in assessTemporalIndependence(intable = CostaRT, deltaTimeComparedTo = "lastIndependentRecord",  : 

  could not find function "assessTemporalIndependence"

> 

Thank you again for helping with all of this!

On Wednesday, May 13, 2020 at 12:41:53 PM UTC-5, Juergen Niedballa wrote:

[Juergen Niedballa]

Hi,

first create a column with full species name:

CostaRT$species_full <- paste(CostaRT$Genus, CostaRT$Species)

then

records_filter30_min <- camtrapR:::assessTemporalIndependence(intable = CostaRT,

                                                    deltaTimeComparedTo = "lastIndependentRecord",

                                                    columnOfInterest = "species_full",             

                                                    stationCol = "Station",

                                                    minDeltaTime = 30,

                                                    camerasIndependent = FALSE )

I forgot to be explicit about  camtrapR:::assessTemporalIndependence

The camtrapR::: part is necessary because assessTemporalIndependence is not exported in the package namespace, so we need to tell R that way that it is in the internal functions of camtrapR. 

[Josh K]

Awesome thank you! how long should the records filter run for? we do have over 56000 images and so far its been running for over 12 hours. I'm not sure if its looping or there is something else happening as it just shows that R is running the code and no errors have came up yet. Thank you again for the all help!

[Juergen Niedballa]

Hi, 

it definitely shouldn't run that long. I realised that the code I shared was not complete. You need a column DateTimeOriginal, maybe that's why it got stuck. Presence of column "DateTimeOriginal" is not being checked in assessTemporalIndependence because it is an internal functions and unsually the format of the input table is checked before.

Also, there's many duplicate rows (with identical station, species and date/time), which I remove before running assessTemporalIndependence.

So here's a full worked example (took a minute or two to run).

CostaRT$species_full <- paste(CostaRT$Genus, CostaRT$Species)

# create column DateTimeOriginal 

CostaRT$DateTimeOriginal <- strptime(CostaRT$Date.Time, format = "%m/%d/%Y %H:%M", tz = "UTC")    # you may wish to change the time zone, but it shouldn't matter for this computation

# find duplicate rows (add more items if needed, e.g. "Site.Name")

duplicate_rows <- which(duplicated(CostaRT[, c("Station", "species_full", "DateTimeOriginal")]))

# remove duplicate rows

CostaRT_filter <- CostaRT[-duplicate_rows,]

# filter records

records_filter30_min <- camtrapR:::assessTemporalIndependence(intable = CostaRT_filter,

                                                              deltaTimeComparedTo = "lastRecord",

                                                              columnOfInterest = "species_full", 

                                                              stationCol = "Station",

                                                              minDeltaTime = 30,

                                                              camerasIndependent = FALSE )

records_filter30_min_v2 <- camtrapR:::assessTemporalIndependence(intable = CostaRT_filter,

                                                              deltaTimeComparedTo = "lastIndependentRecord",

                                                              columnOfInterest = "species_full", 

                                                              stationCol = "Station",

                                                              minDeltaTime = 30,

                                                              camerasIndependent = FALSE )

This is the dimensions of the output (rows / columns):

> dim(CostaRT)

[1] 55656    20

> dim(CostaRT_filter)

[1] 9945   20

> dim(records_filter30_min)

[1] 6616   25

> dim(records_filter30_min_v2)

[1] 6682   25

Regarding the difference between records_filter30_min and records_filter30_min_v2, (due to argument "deltaTimeComparedTo"), see this thread.

Hope this helps.

Best regards,

Jürgen

[Josh K]

Awesome thank you very much! The codes ran fast and produced no errors. The only issue now that we have and I'm not sure why is the # of events for each species is off. For some species where the number of images is low and the events only span a few images it produced an accurate count for events, but for species with more images and longer events, an example in our case being White Lipped Peccaries, the number of events produced by the report was 957, but when we went to verify for that species the total number of events was 267. I hope this explains the issue, thanks again for all the help. It's very appreciated!

Best,

Josh K.

[Juergen Niedballa]

Hi, 

would you please share the code you ran to get those results (957 / 267 records)?

Here's the number of peccaries I get (with the code above), and the numbers are different.

> spec_test <- "Tayassu pecari"
> sum(CostaRT$species_full == spec_test)
[1] 18240
> sum(CostaRT_filter$species_full == spec_test)
[1] 940
> sum(records_filter30_min$species_full == spec_test)
[1] 252
> sum(records_filter30_min_v2$species_full == spec_test)
[1] 257

Thank you,

Jürgen

[Josh K]

Yes we manually counted the events to get the 267, but here is the code that was producing the events to be high on most species.

CostaRT$species_full <- paste(CostaRT$Genus, CostaRT$Species)

CostaRT$DateTimeOriginal <- strptime(CostaRT$Date.Time, format = "%m/%d/%Y %H:%M", tz = "UTC") 

duplicate_rows <- which(duplicated(CostaRT[, c("Station", "species_full", "DateTimeOriginal")]))

CostaRT_filter <- CostaRT[-duplicate_rows,]

records_filter30_min_v2 <- camtrapR:::assessTemporalIndependence(intable = CostaRT_filter,

                                                                 deltaTimeComparedTo = "lastIndependentRecord",

                                                                 columnOfInterest = "species_full", 

                                                                 stationCol = "Station",

                                                                 minDeltaTime = 30,

                                                                 camerasIndependent = FALSE )

 

reportTest <- surveyReport (recordTable          = CostaRT_filter,

                            CTtable              = costaCT,

                            cameraCol            = "Camera",

                            speciesCol           = "Species",

                            stationCol           = "Station",

                            setupCol             = "Setup_date",

                            retrievalCol         = "Retrieval_date",

                            CTDateFormat         = "%m/%d/%Y",

                            recordDateTimeCol    = "Date.Time",

                            recordDateTimeFormat = "%m/%d/%Y %H:%M",

                            CTHasProblems        = FALSE)

                            

[Juergen Niedballa]

Hi,

I ran your code and records_filter30_min_v2 has 257 records for "Tayassu pecari". In which table did you manually count 267 events? In the original data CostaRT, or was it in your images?

I could not run the surveyReport part of your code because I don't have the object costaCT. Feel free to email it or share (remember, data shared on the group are public).

In running surveyReport, is there a reason you use the record table with duplicates removed, but without the 30min filter applied:

recordTable          = CostaRT_filter

and not the table with duplicates removed and the 30min filter applied:

recordTable          = records_filter30_min_v2
?