camtrapR 2.0.0 on CRAN, including video support

[camtrapR]

Dear camtrapR users,

I am happy to announce the release of camtrapR 2.0.0 on CRAN, now featuring video support! 

As usual, install it via:

install.packages("camtrapR")

In this version, recordTable() and recordTableIndividual() can process video files via a new argument "video". The workflow is identical to processing JPG images and supports identification via folder names and digiKam tags.

Due to video metadata being somewhat unreliable, there is no guarantee that it will work with any camera trap model. So first you need to confirm your cameras provide correct date and time in the video metadata.

You can use exifTagNames() to check if your video metadata contain a date/time tag (e.g. something like "QuickTime:CreateDate", not File Create / Access / Modify date). Make sure the value of the tag is identical to the time the video was taken. If a suitable date/time tag exists, Exiftool can extract it (analogous to extracting DateTimeOriginal from JPGs).

Furthermore, you can tag videos in digiKam, e.g. for species and individual identification or for assigning other information. camtrapR will then access the digiKam database to extract and tabulate those tags. But please be aware that (unlike JPGs) the video tags are not written into video metadata, so they are only present when the videos are in your digiKam database. They will be lost when videos are moved around or copied to other locations. Package "RSQLite" is required for this.

For more details on processing video files, please see the vignettes and function help files:

https://cran.r-project.org/web/packages/camtrapR/vignettes/camtrapr1.html#video_support_(since_version_200)

https://cran.r-project.org/web/packages/camtrapR/vignettes/camtrapr2.html#metadata_tagging   (find "And how about metadata extraction from videos?")

Note that currently video support is limited to two functions only. Moving forward, I am planning to enable video support in other functions of the workflow as well (imageRename, appendSpeciesNames, etc), but this will take some time to implement. 

Besides video support, there is a number of other improvements in the new version, the most important ones being:

• reliable creation of zip files in surveyReport() via package "zip"
• improved output of exifTagNames()   [which now seems to make problems on Linux, please bear with me]
• a set of sample images identified via metadata tags
• improved error messages in many functions
• prettier vignettes.

For a full list of changes and bugfixes, please see the package NEWS section

Video support is still in its early stages and somewhat experimental, so please let me know what you think, how it works for you, and how I can improve it for you. 

If you test the video feature, would you please let me know:

• what camera models did you use,
• did it work,
• what metadata tag did you use?

I can then provide information about what cameras are supported in the package documentation for future reference. Thank you very much!

One last thing, if you find camtrapR helpful and would like to support the development and support here in the Google Group, please consider making a donation to help me keep improving the package for you. See the welcome message on the Google Group main page for details, under "Donations". Thank you very much for your consideration.

All the best to you. Stay safe everyone!

Jürgen

[Avantika Thapa]

I was struggling with the video processing. Very glad to know about the update. 

--
You received this message because you are subscribed to the Google Groups "camtrapR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camtrapr+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/camtrapr/da94ddeb-6686-4199-a428-597c2eebfd30%40googlegroups.com.

--

Thanks & Regards,

Avantika Thapa

Junior Research Fellow

NMHS

Zoological Survey of India, Kolkata

[Alexandre Courtiol]

Nice!

--

You received this message because you are subscribed to the Google Groups "camtrapR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camtrapr+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/camtrapr/da94ddeb-6686-4199-a428-597c2eebfd30%40googlegroups.com.

--

Alexandre Courtiol

http://sites.google.com/site/alexandrecourtiol/home

"Science is the belief in the ignorance of experts", R. Feynman

[Arybeth Elizabeth Medrano Cosme]

Encontré lo siguiente para extraer los metadatos de los vídeos, pero me aparece un Error y no se como corregirlo ¿Podrían ayudarme con esto?

----rec_table5 <- recordTable(inDir = wd_images_ID_species, 

                                               IDfrom = "metadata", 

                                               metadataSpeciesTag = "Species", 

                                               video = list(file_formats = c("jpg", "mp4", "avi", "mov"),                  

                                               dateTimeTag = "QuickTime:CreateDate", 

                                               db_directory = "C:/Users/YourName/Pictures", 

                                               db_filename = "digikam4.db") 

Error: station directory videos/E1 was not found in digiKam albums. Skipping

El sábado, 18 de abril de 2020, 8:04:49 (UTC-5), camtrapR escribió:

Estimados usuarios de camtrapR:

¡Me complace anunciar el lanzamiento de camtrapR 2.0.0 en CRAN, que ahora cuenta con soporte de video! 

Como de costumbre, instálalo a través de:

install.packages ("camtrapR")

En esta versión, recordTable () y recordTableIndividual () pueden procesar archivos de video a través de un nuevo argumento "video". El flujo de trabajo es idéntico al procesamiento de imágenes JPG y admite la identificación a través de nombres de carpetas y etiquetas digiKam.

Debido a que los metadatos de video son poco confiables, no hay garantía de que funcione con ningún modelo de cámara trampa. Entonces, primero debe confirmar que sus cámaras proporcionan la fecha y hora correctas en los metadatos del video.

Puede usar exifTagNames () para verificar si los metadatos de su video contienen una etiqueta de fecha / hora (por ejemplo, algo como "QuickTime: CreateDate", no la fecha de creación / acceso / modificación de archivo). Asegúrese de que el valor de la etiqueta sea idéntico a la hora en que se tomó el video. Si existe una etiqueta de fecha / hora adecuada, Exiftool puede extraerla (de forma análoga a la extracción de DateTimeOriginal de JPG).

Además, puede etiquetar videos en digiKam, por ejemplo, para especies e identificación individual o para asignar otra información. camtrapR accederá a la base de datos digiKam para extraer y tabular esas etiquetas. Pero tenga en cuenta que (a diferencia de los JPG) las etiquetas de video no están escritas en metadatos de video, por lo que solo están presentes cuando los videos están en su base de datos digiKam. Se perderán cuando los videos se muevan o se copien a otras ubicaciones. Se requiere el paquete "RSQLite" para esto.

Para obtener más detalles sobre el procesamiento de archivos de video, consulte las viñetas y los archivos de ayuda de funciones:

https://cran.r-project.org/ web / packages / camtrapR / vignettes / camtrapr1.html # video_support_ (since_version_ 200)

https://cran.r-project.org/ web / packages / camtrapR / vignettes / camtrapr2.html # metadata_tagging    (encuentre "¿Y qué hay de la extracción de metadatos de los videos?")

Tenga en cuenta que actualmente el soporte de video está limitado a dos funciones solamente. En el futuro, planeo habilitar el soporte de video en otras funciones del flujo de trabajo también (imageRename, appendSpeciesNames, etc.), pero tomará algún tiempo implementarlo. 

Además del soporte de video, hay una serie de otras mejoras en la nueva versión, las más importantes son:

• creación confiable de archivos zip en surveyReport () a través del paquete "zip"
• salida mejorada de exifTagNames () [que ahora parece tener problemas en Linux, por favor tengan paciencia conmigo]
• un conjunto de imágenes de muestra identificadas mediante etiquetas de metadatos
• mensajes de error mejorados en muchas funciones
• viñetas más bonitas.

Para obtener una lista completa de cambios y correcciones de errores, consulte la sección NOTICIAS del paquete 

El soporte de video todavía está en sus primeras etapas y es algo experimental, así que hágame saber lo que piensa, cómo funciona para usted y cómo puedo mejorarlo para usted. 

Si prueba la función de video, por favor hágamelo saber:

• qué modelos de cámara usaste,
• funcionó,
• ¿Qué etiqueta de metadatos usaste?

Luego puedo proporcionar información sobre qué cámaras son compatibles con la documentación del paquete para referencia futura. ¡Muchas gracias!

Una última cosa, si considera que camtrapR es útil y desea apoyar el desarrollo y la asistencia aquí en el Grupo de Google, considere hacer una donación para ayudarme a seguir mejorando el paquete para usted. Consulte el mensaje de bienvenida en la página principal de Google Group para obtener más información, en "Donaciones". Muchas gracias por su consideración.

Todo lo mejor para usted. ¡Manténgase a salvo todos!

Jürgen

[Juergen Niedballa]

Dear Arybeth,

a few questions first:

- what is the file format of your videos?

- did you tag the videos in digiKam?

- is the code you shared the exact code you ran that caused the error?

And a few suggestions:

- first, run exifTagNames() on one of your videos and find a suitable date/time tag (that shows the correct date and time when the video was recorded). Pleaes note, there is no guarantee such a metadata tag exists in video metadata, it depends on the camera manufacturer. Feel free to share the output of exifTagNames() here if you are unsure.

- if there is a usable date/time tag in the video metadata, use it in dateTimeTag = "QuickTime:CreateDate" ("QuickTime:CreateDate" is an example, you may need to adapt it).

- include file formats you wish to extract data from in this argument:   file_formats = c("jpg", "mp4", "avi", "mov")

- db_directory = "C:/Users/YourName/Pictures" needs to be changed to the directory where the digiKam database file is located (by default, on Windows, in your user's "pictures" directory) - You can check in digiKam > settings > configure digiKam > database)

- db_filename = "digikam4.db" is the filename of the digiKam database. I belive the default should work, but please check in db_directory to make sure it is correct.

Please also see the section on videos in vignette 1 and the help file of ?recordTable()

Hope this helps.

I apologise for answering in English, I did so for it to be understandable to the general public. But I also checked the Google translate output in Spanish and it was correct, as far as I can tell.

Best regards,

Jürgen

[Estefania Salazar]

Dear Jürgen

I want to share with you my experience using camtrapR whit videos.

I was struggling with video metadata extraction since 2017 when I kwen camtrapR ver. 1

Before camtrap ver 2 I have created a simple function to extract video metadata with the next detail.

The file modification Date doesn't change if you don´t copy and paste the files, only drag the file to the new location.

So, I first check if the modification date is the same that appears in the original video.

If both dates are the same in camtrapR ver 2. I used the FileModifyDate to obtain the DateTimeOriginal.

That worked for my and I got a very nice recordTable

I used moultrie and bushnell cameras 

This was the code

recordTable(inDir="D:/Documents/Estefania/Fototrampeo/3_camtrapR/estaciones",

            IDfrom="directory",

            cameraID="directory",

            camerasIndependent=TRUE,

            minDeltaTime = 0,

            timeZone="America/Bogota",

            writecsv = FALSE,

            removeDuplicateRecords = FALSE,

            returnFileNamesMissingTags = FALSE,

            video = list(file_formats = c("jpg", "mp4", "avi", "mov"),                  

            dateTimeTag = "FileModifyDate")

Camtrap is the best package that I found to organize camera trap information. Thank you.

Sorry for my English.

El sábado, 18 de abril de 2020, 8:04:49 (UTC-5), camtrapR escribió:

[Juergen Niedballa]

Hi Estefania,

thank you for the information. I'm glad to know that date extraction also works for you with FileModifyDate. 

You said it only works if you copy/paste the videos and don't further modify them. That is very important, since any slight modification to the video files would overwrite this tag with the current date/time and make it unusable for creating record tables. So one needs to be very very careful when using this method. And of course test it before a survey and confirm it works. Also, when using digiKam to tag these videos, one would need to first confirm that loading it into digiKam doesn't change the FileModifyDate. Do you have any experience with this?

If it is available, I would always recommend to work with a date/time tag from the metadata as described above in this thread and in the vignettes. But it's good to know it also works with FileModifyDate. That makes the function usable also for camera that don't record date/time of videos in a dedicated metadata tag. 

Thank you!

Jürgen

[maria_majka]

Dear Jurgen,

First of all, thank you for creating the package, I have used it for a few studies now.

I am currently working with a set of videos and pictures, which I have tagged in digiKam. When I processed pictures on their own everything worked perfectly but for the whole set I keep getting this error coming up:

Error in `$<-.data.frame`(`*tmp*`, "cleartext_parent", value = c("Dorosle",  : 

  replacement has 10986 rows, data has 11004

And to be honest I do not understand what is wrong as it worked just fine for pictures. I have checked again for spaces in tag names and simplified my tag tree but nothing.

This is the exact code I used:

dir_spp_table <- recordTable(inDir = "C:/Users/majka/Documents/TPN/Images", IDfrom = "metadata", metadataSpeciesTag = "Gatunek", cameraID = "directory", camerasIndependent = TRUE, minDeltaTime = 0, timeZone = "Europe/Warsaw", stationCol = "Station", writecsv = TRUE, removeDuplicateRecords = FALSE, video = list(file_formats = c("jpg", "mp4", "avi", "mov"), dateTimeTag = "FileModifyDate", db_directory = "C:/Users/majka/Documents/TPN/", db_filename = "digiKam4.db"), outDir = "C:/Users/majka/Documents/TPN")

If you or anyone have any ideas or tips for where to look, I will be very grateful.

Many thanks,

Maria

[Juergen Niedballa]

Hi Maria,

this looks like a problem in reading / deciphering the tags from the digiKam database correctly. It's hard to say what exactly the issue is without seeing the actual data. What I can tell you already is that it is unexpected and I haven't seen that problem before.

Can you try running the function for the video only please (by removing "jpg" from the file_formats = c("jpg", "mp4", "avi", "mov") argument)? It will probably still give the same error with different numbers.

Does digiKam show the correct tags for the video files?

If possible, would please share the digikam4.db file? Feel free to share privately to my email. 

Thank you,

Jürgen

[Joel Ruprecht]

Hi Maria and Juergen,

I had the same (or similar) error when trying to run recordTable() on videos. I traced the issue to the digiKamVideoHierarchicalSubject() internal helper function. For me, the error was resolved by replacing 

ImageTags$cleartext_parent <- Tags$name[Tags$pid[match(ImageTags$tagid, Tags$id)]] ### line 1726 here https://github.com/cran/camtrapR/blob/master/R/variousOtherHelperFunctions.R

with 

parentNA<-which(is.na(Tags$name.parent))
Tags$name.parent[parentNA]<-Tags$name[parentNA]
ImageTags$cleartext_parent <- Tags$name.parent[match(ImageTags$tagid, Tags$id)]

Then I ran the various internal helper functions of recordTable() manually to generate the record table. Maybe that workaround will help you too, or at least help you diagnose where the error is occurring.

Joel

 

--

You received this message because you are subscribed to the Google Groups "camtrapR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camtrapr+u...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/camtrapr/1cf7f2f3-9519-4687-8dba-dcc0abf1077an%40googlegroups.com.

[Juergen Niedballa]

Hi Maria  & Joel,

thank you for the input. A new version with a bugfix is on github (see here). It uses a similar code to what you suggested, Joel. The precise code you suggested seemed to work, but only when there was at least one video which had a tag that was not in a tag group (had no parent tag). 

So the new solution works for both tags in groups and not in groups. Tags without parents now get "NA" as parent name.

Tags$parent_name <- Tags$name[match(Tags$pid, Tags$id)]
ImageTags$cleartext_parent <- Tags$parent_name[match(ImageTags$tagid, Tags$id)]

You can install the new version from github as usual:

library(remotes)
install_github("jniedballa/camtrapr")

With the new version it should be possible to get the desired output without having to run the individual helper functions. Please let me know if there's any issues with it. 

Thank you and best regards,

Jürgen