Error in reading date format

[Huyen Nguyen]

Dear all, 

I got the problem with date time format in both CTtable and rc.table. I have set the date time format of these file in lubridate and ran the survey.Report but I got the error notification: 

Error in as.POSIXlt.character(x, tz, ...) : 

  character string is not in a standard unambiguous format

> str(rc.table)

'data.frame': 1080 obs. of  5 variables:

 $ station         : chr  "UMT001" "UMT001" "UMT001" "UMT001" ...

 $ species         : chr  "Common-palm-civet" "Common-palm-civet" "Greater-coucal" "Greater-coucal" ...

 $ FileName        : chr  "IMG_0001.MP4" "IMG_0044.MP4" "IMG_0005.MP4" "IMG_0017.MP4" ...

 $ DateTimeOriginal: POSIXct, format: "2020-02-22 22:00:00" "2020-03-07 20:52:10" "2020-02-26 15:27:18" "2020-03-01 16:55:22" ...

 $ Directory       : chr  "F:\\SVW research\\U Minh\\U Minh Thuong NP\\Camera-trapping\\Targeted\\Fishing Cat_U Minh Thuong_2019 2020\\Raw"| __truncated__ "F:\\SVW research\\U Minh\\U Minh Thuong NP\\Camera-trapping\\Targeted\\Fishing Cat_U Minh Thuong_2019 2020\\Raw"| __truncated__ "F:\\SVW research\\U Minh\\U Minh Thuong NP\\Camera-trapping\\Targeted\\Fishing Cat_U Minh Thuong_2019 2020\\Raw"| __truncated__ "F:\\SVW research\\U Minh\\U Minh Thuong NP\\Camera-trapping\\Targeted\\Fishing Cat_U Minh Thuong_2019 2020\\Raw"| __truncated__ ...

> str(CTtable)

'data.frame': 35 obs. of  7 variables:

 $ station       : chr  "UMT001" "UMT002" "UMT003" "UMT004" ...

 $ utm_x         : num  511132 509292 512110 508691 512854 ...

 $ utm_y         : num  1061373 1064941 1060784 1063603 1058532 ...

 $ setup_date    : POSIXct, format: "2020-02-20" "2020-02-19" "2020-02-20" "2020-02-19" ...

 $ retrieval_date: POSIXct, format: "2020-05-15" "2020-05-15" "2020-05-15" "2020-05-15" ...

 $ Problem1_from : POSIXct, format: "2020-03-20" NA "2020-02-21" "2020-02-20" ...

 $ Problem1_to   : POSIXct, format: "2020-05-15" NA "2020-05-15" "2020-05-15" ...

> survey = surveyReport(recordTable = rc.table, CTtable = CTtable, speciesCol = "species", stationCol = "station", setupCol = "setup_date", retrievalCol = "retrieval_date", CTDateFormat = "%Y-%m-%d", recordDateTimeCol = "DateTimeOriginal", recordDateTimeFormat = "%Y-%m-%d %H:%M:%S", CTHasProblems = TRUE, sinkpath = getwd())

Error in as.POSIXlt.character(x, tz, ...) : 

  character string is not in a standard unambiguous format

I got this problem after I had updated the CamtrapR package. Any suggestions to solve this problem?

Thank you.

Kind regards,

Huyen

[Juergen Niedballa]

Hi Huyen,

can you try setting the date/time column in the record table to character instead of POSIXct please?

as.character(rc.table$DateTimeOriginal)

In CTtable, set the date columns to date with as.Date() please. 

That should hopefully solve the issue. 

Best regards,

Jürgen

[Huyen Nguyen]

Dear Juergen, 

Thank you for your support.

I have tried your solution but unfortunately, it did not work in my case.

> str(rc.table)

'data.frame': 1080 obs. of  5 variables:

 $ station         : chr  "UMT001" "UMT001" "UMT001" "UMT001" ...

 $ species         : chr  "Common-palm-civet" "Common-palm-civet" "Greater-coucal" "Greater-coucal" ...

 $ FileName        : chr  "IMG_0001.MP4" "IMG_0044.MP4" "IMG_0005.MP4" "IMG_0017.MP4" ...

 $ DateTimeOriginal: chr  "2020-02-22 22:00:00" "2020-03-07 20:52:10" "2020-02-26 15:27:18" "2020-03-01 16:55:22" ...

 $ Directory       : chr  "F:\\SVW research\\U Minh\\U Minh Thuong NP\\Camera-trapping\\Targeted\\Fishing Cat_U Minh Thuong_2019 2020\\Raw"| __truncated__ "F:\\SVW research\\U Minh\\U Minh Thuong NP\\Camera-trapping\\Targeted\\Fishing Cat_U Minh Thuong_2019 2020\\Raw"| __truncated__ "F:\\SVW research\\U Minh\\U Minh Thuong NP\\Camera-trapping\\Targeted\\Fishing Cat_U Minh Thuong_2019 2020\\Raw"| __truncated__ "F:\\SVW research\\U Minh\\U Minh Thuong NP\\Camera-trapping\\Targeted\\Fishing Cat_U Minh Thuong_2019 2020\\Raw"| __truncated__ ...

> str(CTtable)

'data.frame': 35 obs. of  7 variables:

 $ station       : chr  "UMT001" "UMT002" "UMT003" "UMT004" ...

 $ utm_x         : num  511132 509292 512110 508691 512854 ...

 $ utm_y         : num  1061373 1064941 1060784 1063603 1058532 ...

 $ setup_date    : Date, format: "2020-02-20" "2020-02-19" "2020-02-20" "2020-02-19" ...

 $ retrieval_date: Date, format: "2020-05-15" "2020-05-15" "2020-05-15" "2020-05-15" ...

 $ Problem1_from : Date, format: "2020-03-20" NA "2020-02-21" "2020-02-20" ...

 $ Problem1_to   : Date, format: "2020-05-15" NA "2020-05-15" "2020-05-15" ...

> report = surveyReport(recordTable = rc.table, CTtable, speciesCol = "species", setupCol = "setup_date", retrievalCol = "retrieval_date", stationCol = "station", CTDateFormat = "%Y-%m-%d", recordDateTimeCol = "DateTimeOriginal", recordDateTimeFormat = "%Y-%m-%d %H:%M:%S", CTHasProblems = TRUE, sinkpath = getwd())

Error in charToDate(x) : 

  character string is not in a standard unambiguous format

I tried to work with these files in the old version of CamtrapR in other computer and they worked well with the date columns set in Lubridate format. 

Kind regards, 

Huyen

[Juergen Niedballa]

Hi Huyen, 

that's strange, it worked for me after creating a sample camera trap table and record table from the str() output you shared. It worked for both lubridate format and the base R date/time formats. See code and output below. But it was only for 3 or 4 rows of your data, so maybe the problem is further below in your input tables.

Which version of camtrapR are you using? And do you get the same error when you use lubridate format? Also, can you check that all dates in the camera trap table are in the same format (ymd). If none of this helps please feel free to share / email me a sample data set to identify the issue. 

Thank you.

Kind regards,

Jürgen

rc.table <- data.frame(station =      c("UMT001", "UMT001", "UMT001"),

                       species  =       c("Common-palm-civet", "Common-palm-civet", "Greater-coucal"),

                       FileName    =    c("IMG_0001.MP4", "IMG_0044.MP4", "IMG_0005.MP4"),

                       DateTimeOriginal = c("2020-02-22 22:00:00", "2020-03-07 20:52:10", "2020-02-26 15:27:18"),

                       Directory =      "F:\\SVW research\\U Minh\\U Minh Thuong NP\\Camera-trapping\\Targeted\\Fishing Cat_U Minh Thuong_2019 2020\\Raw")

CTtable <- data.frame(

station       = c("UMT001", "UMT002", "UMT003", "UMT004"),

utm_x         = c(511132, 509292, 512110, 508691),

utm_y         = c(1061373, 1064941, 1060784, 1063603),

setup_date    = as.Date(c("2020-02-20", "2020-02-19", "2020-02-20", "2020-02-19")),    # for testing, class "Date" works

retrieval_date= c("2020-05-15", "2020-05-15", "2020-05-15", "2020-05-15"),             # character also works

Problem1_from = c("2020-03-20", NA, "2020-02-21", "2020-02-20"),

Problem1_to   = c("2020-05-15", NA, "2020-05-15", "2020-05-15"))

report = surveyReport(recordTable = rc.table, CTtable, speciesCol = "species", setupCol = "setup_date", retrievalCol = "retrieval_date", 

                      stationCol = "station", CTDateFormat = "ymd", recordDateTimeCol = "DateTimeOriginal", recordDateTimeFormat = "ymd HMS", 

                      CTHasProblems = TRUE)

$survey_dates

  station setup_date first_image_date last_image_date retrieval_date n_nights_total n_nights_active n_cameras

1  UMT001 2020-02-20       2020-02-22      2020-03-07     2020-05-15             85              29         1

2  UMT002 2020-02-19             <NA>            <NA>     2020-05-15             86              86         1

3  UMT003 2020-02-20             <NA>            <NA>     2020-05-15             85               1         1

4  UMT004 2020-02-19             <NA>            <NA>     2020-05-15             86               1         1

$species_by_station

  station n_species

1  UMT001         2

$events_by_species

            species n_events n_stations

1 Common-palm-civet        2          1

2    Greater-coucal        1          1

$events_by_station

  station           species n_events

1  UMT001 Common-palm-civet        2

2  UMT001    Greater-coucal        1

$events_by_station2

  station           species n_events

1  UMT001 Common-palm-civet        2

2  UMT001    Greater-coucal        1

[Huyen Nguyen]

Thank Jurgen for helping me solve this problem.

I am using camtrapR version 2.0.3 and Jurgen figured out that for this version, format of all date columns in CTtable should be set as character to facilitate surveyReport and it works.  

[Juergen Niedballa]

Thank you Huyen,

I will add the conversion of the problem columns to character in the function in the next release.  So in the future it should be converted automatically (there is code to do that already but for some reason it fails). Thank you for bringing the issue to my attention. 

Best regards,

Jürgen

[Christian Osorio]

Dear Jurgen:

I used recordTable without issues, and now I am trying to create detection histories using detectionHistory(recordTable, ...) and I have this error: "Error in as.POSIXlt.character(x, tz, ...) :  character string is not in a standard unambiguous format".

I checked the previous messages about this issue in the group and tried to solve it, but it it still there.

I used lubridate to convert the column DateTimeOriginal to a POSIXIt object and then converted back to text to have it standardized, but I still get that error. 

Please see the output below:

Thank you very much!

Bests,

Christian Osorio

www.carnivorosaustrales.org

library(lubridate)

rec.db.species30min_all_DateTimeFixed$DateTimeOriginalOK <- parse_date_time(rec.db.species30min_all_DateTimeFixed$DateTimeOriginal, orders = "dmy HM")

> str(rec.db.species30min_all_DateTimeFixed$DateTimeOriginalOK)

 POSIXct[1:5268], format: "2020-02-12 21:20:00" "2020-02-13 03:48:00" "2020-02-13 06:24:00" "2020-02-13 07:13:00" "2020-02-13 12:22:00" "2020-02-13 14:16:00" ...

rec.db.species30min_all_DateTimeFixed$DateTimeOriginalOKtxt <- as.character(rec.db.species30min_all_DateTimeFixed$DateTimeOriginalOK)

> str(rec.db.species30min_all_DateTimeFixed$DateTimeOriginalOKtxt)

 chr [1:5268] "2020-02-12 21:20:00" "2020-02-13 03:48:00" "2020-02-13 06:24:00" "2020-02-13 07:13:00" "2020-02-13 12:22:00" ...

> PumaDetHist <- detectionHistory(recordTable = rec.db.species30min_all_DateTimeFixed,

+                  species = c("Puma_concolor"),

+                  camOp = camOp,

+                  output = c("binary", "count"),

+                  stationCol = "Station",

+                  speciesCol = "Species",

+                  recordDateTimeCol = "DateTimeOriginalOKtxt",

+                  recordDateTimeFormat = "%Y-%m-%d %H:%M:%S",

+                  occasionLength = 1,

+                  #minActiveDaysPerOccasion = NULL,

+                  maxNumberDays = 736,

+                  day1 = "survey",

+                  #buffer = NULL,

+                  includeEffort = FALSE,

+                  #scaleEffort = TRUE,

+                  occasionStartTime = 0,

+                  datesAsOccasionNames = FALSE,

+                  timeZone = "Chile/Continental",

+                  writecsv = FALSE,

+                  outDir = "G:/My Drive/PhD Fisheries and Wildlife Sciences - VT/Dissertation/Sorting photos with CamTrapR",

+                  #unmarkedMultFrameInput = FALSE

+ )

Error in as.POSIXlt.character(x, tz, ...) : 

  character string is not in a standard unambiguous format

[Juergen Niedballa]

Hi Christian,

which version of camtrapR are you using? It seems like a bug, and a few other people also pointed out issues with date/time columns. Some of the issues are fixed in the development version on github, so you may want to give that a try:

library(devtools)
install_github("jniedballa/camtrapR", ref = "dev")

If you do, please see the news section, since a few things changed (particularly regarding cameraOperation, but it also affects detectionHistory).

Then I'd also suggest a few changes in the code for detectionHistory (please note, I didn't remove the + from your console):

> PumaDetHist <- detectionHistory(recordTable = rec.db.species30min_all_DateTimeFixed,

+                  species = c("Puma_concolor"),    # c() is not necessary

+                  camOp = camOp,

+                  output = c("binary", "count"),   # choose one, preferable "binary"

+                  stationCol = "Station",          # is default

+                  speciesCol = "Species",          # also default, can be removed

+                  recordDateTimeCol = "DateTimeOriginalOKtxt",

+                  recordDateTimeFormat = "%Y-%m-%d %H:%M:%S",     # default as well

+                  occasionLength = 1,

+                  #minActiveDaysPerOccasion = NULL,

+                  maxNumberDays = 736,            # usually not necessary. Did you operate cameras for 2 years straight?

+                  day1 = "survey",

+                  #buffer = NULL,

+                  includeEffort = FALSE,

+                  #scaleEffort = TRUE,

+                  occasionStartTime = 0,           # in the development version, this is deprecated and was moved to cameraOperation(), including it will throw a warning

+                  datesAsOccasionNames = FALSE,

+                  timeZone = "Chile/Continental",

+                  writecsv = FALSE,

+                  outDir = "G:/My Drive/PhD Fisheries and Wildlife Sciences - VT/Dissertation/Sorting photos with CamTrapR",    # not necessary if writecsv = FALSE

+                  #unmarkedMultFrameInput = FALSE

+ )

Best,

Jürgen

[Christian Osorio]

Dear Jurgen:

Thank you so much for your reply.

I installed the development version as you suggested, and I still get the same error:

> PumaDetHist <- detectionHistory(recordTable = rec.db.species30min_all_DateTimeFixed,

+                  species = "Puma_concolor",

+                  camOp = camOp,

+                  output = c("binary"),

+                  #stationCol = "Station",

+                  #speciesCol = "Species",

+                  recordDateTimeCol = "DateTimeOriginalOKtxt",

+                  #recordDateTimeFormat = "%Y-%m-%d %H:%M:%S",

+                  occasionLength = 1,

+                  #minActiveDaysPerOccasion = NULL,

+                  #maxNumberDays = 736,

+                  day1 = "survey",

+                  #buffer = NULL,

+                  includeEffort = FALSE,

+                  #scaleEffort = TRUE,

+                  #occasionStartTime = 0,

+                  datesAsOccasionNames = FALSE,

+                  timeZone = "Chile/Continental",

+                  writecsv = FALSE,

+                  #outDir = "G:/My Drive/PhD Fisheries and Wildlife Sciences - VT/Dissertation/Sorting photos with CamTrapR",

+                  #unmarkedMultFrameInput = FALSE

+ )

Error in as.POSIXlt.character(x, tz, ...) : 

  character string is not in a standard unambiguous format

Well, I guess I will have to face the little spreadsheet nightmare! :D

camptrapR has already made my life much easier! Thanks a lot!!!

Bests,

[Juergen Niedballa]

Hello Christian,

it must be some bug then, and I just don't fully understand the issue yet. If you don't mind, feel free to email or otherwise share your R workspace and I'll see what the problem is and how to fix it. Just send me a private email with the data or a link to keep your data confidential, and I'll try to fix whatever is going wrong.

Best,

Jürgen

--
You received this message because you are subscribed to a topic in the Google Groups "camtrapR" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/camtrapr/jHhaHR80CXY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to camtrapr+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/camtrapr/82617421-a7d7-4acf-b421-fc63393c9a8an%40googlegroups.com.