Export for independent observation

[Tomislav Gomercic]

Dear all,

We are working on the LIFE Lynx project where we do research on lynxes. The project is now ongoing for a year and we have set up a 100 phototraps for now. Camelot is doing an amazing job helping us with the photo analysis.

I am sending you this inquiry as I do not know how to produce "Custom reports" in Camelot.

In export (report) I need data of independent observation for further use in GIS application and analysis.

Is it possible to export independent observations? I assume it is a combination of the report "CameraR Record Table" and "Raw Data Export".

So I would like to export data for all independent observation with columns:

DateTime of independent observation (eg. DateTime of first picture of independent observation)

Trap Station Name

Trap Station Latitude

Trap Station Longitude

Genus Species

Number of animals in independent observation

Sex

Life stage

MyExtraField (Vehicle, lynx name, photoid)

not necessary

Directory, File Name (of first picture of independent observation)

I am really looking forward to the next Camelot version, especially to the "support for date and timestamp searches in the library" option and I hope that the possibility to download selected images from Library will be soon available. 

With regards, Tomislav Gomerčić

[Chris Mann]

Hi Tomislav,

I'm glad to hear Camelot is working well for you!  Sorry for the slow response, though I needed some time to look into this properly.

In short, Camelot's reporting system can definitely be used to produce this report, but it's far from straight-forward to do so.

For context, Camelot defines dozens of different column types -- from simple ones like, species, trap station name, etc, through to more complicated columns, such as percent of sightings at night. These columns are foundational to Camelot's reporting, as they provide the data, and determine how that data should be aggregated. Because these columns are already set up, usually writing a report is (in essence) a matter of specifying which columns you want, and which columns to aggregate.

But there are two things that make this report trickier than usual:

1. Camelot has an "independent observations" column, though the most fine-grained aggregation it can do is at a per-species per-session level. Camelot can and does calculate more fine-grained data internally, but a new custom column would need to be created to make this available in reports.
   
2. "time of first independent observation" would also need to be introduced as a custom column, with much the same requirements as above.
   

I expect this would take someone who knows the reporting system well the better part of a day to write and test.  The other option would be to produce a full data export and use another tool to process that CSV.

Which is to say, I don't really have a good solution to offer here. But let me know if Camelot still seems your most-viable option, and we can try to work something out.

Lastly, it's great to know the features that'd help you the most.  I'm happy to say that date range searching will definitely be in the next release.  For downloading the images, I'm still unsure how it should behave. Would you expect the downloaded images to have the information entered into Camelot available as metadata?  How this feature would be used and the expectations of it aren't clear in my mind, so it'd be great to have your thoughts on it.

-Chris

[Tomislav Gomercic]

Hi Chris,

Thank you very much for respons.

I am going to talk with my friends who are programers, they might find a solution. I will let you know if we manage to do something useful.

Considering image download - it would be great if I could download filtered and/or selected images from the library, plus .cvs file with the data of all downloaded photos (eg. file name, date, time, GPS coordinate, species etc.).

Example: Our project is focused on lynx and the main goal is to identify individual animals based on their coat pattern. So it would be great if I could filter all lynx photos from one site during the certain period and download all of them at once. Now, Camelot can filter that but I have to download each photo separately, one by one and manually write data in excel for each image (file name, date, time, number of animals). 

Tomislav

[Hannah de Villiers]

Dear Chris and Heidi, 

Following up on this old thread as Tomislav was looking for something very similar to what I am from the reports. I have 30 camera traps recording animal movement around a railway in South Africa's Kruger National Park. Rail expansion in Africa is huge and often cutting through protected areas, hence the study. Camelot has been an incredible help for image processing so far.

I am looking for a report that is perhaps a combination of the Record Table and Full Data Export, which contains an indication of "independent observation ID" or some other way to group photos into one observation, but also includes the metadata from my extra sighting fields, as well as quantity of animals, sex, life stage. I "simply" want to be able to group rows/images that are linked to the same independent observation, and pull the metadata so that I can analyse the metadata of observations rather than individual images (in Excel or R). 

Is this at all possible? The only option I can see is to use the Filename referenced in the Record Table, from which I could look up the metadata from that specific image in the Full Data Export table. But it would be better to have an independent observation ID column of sorts in the Full Export table, as then I could also see how many images per observation and the first and last photo time. 

Look forward to hearing from you, thanks again for all the help and the brilliant tool. 

Hannah

[Chris Mann]

Hi Hannah,

Sorry for the slow response.  This sounds like a important initiative.  In short what you want is possible, though it requires a new report to be added to your Camelot.  This is entirely possible but does require some familiarity with coding.

Unfortunately very low on free time at the moment, though over the Christmas break I could probably find time to put together such a report and from there you can install that report as an add-on into your own copy of Camelot relatively easily.

If this would still be useful, could you give me a list of the specific columns you'd like, including the "Sighting Field Key" for any sighting fields you've configured that you'd like in the report?

Thanks,

-Chris

--
You received this message because you are subscribed to the Google Groups "Camelot Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camelot-proje...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/camelot-project/eb4a4c40-2030-40de-ae32-ed64c03edbe1n%40googlegroups.com.

[Hannah de Villiers]

Hi Chris, 

Thanks so much for the response. A new report would be fantastic. Note - I’ve managed to find a (somewhat painstaking!) workaround for analysis of early results, by using the Record table (independent observations) and linking in the metadata / sightings fields from the Full Data Export table using the filename in the Record Table. But a new report would be a lot easier, and is not a rush at all, given I have a temporary solution. I will have cameras up until March’22 and can only really begin analysis using R after that, so if you don’t get to it over the holiday period its not a problem. I don’t have much experience with coding so your help is greatly appreciated in putting together such a report… 

Specific columns to please be added to the Record Table (if you start with this as a base); the ones in italics are my configured sighting fields:

• Class
• Family
• Order
• Genus
• Species
• Species Common Name
• Sighting Quantity
• Species Mass Start
• Species Mass End
• sociality (Dropdown with text)
• lifestage (Dropdown with text)
• sex (Dropdown with text)
• direction-of-travel (Dropdown with text)
• use-of-underpass (Dropdown with text)
• activity-outside (Text input)
• individual-name (Text input)
• rest-inside (Checkbox)
• rest-time (Number)
• interesting-behaviour (Text input)

These fields can be pulled from the first photo from the observation or whatever image the filename refers to in the record table. I have made sure that all photos in an observation have identical metadata so whichever image is easiest to reference. 

Let me know if there is any further detail / information I can provide!

Many, many thanks, 

Hannah

[Chris Mann]

Hi Hannah,

Please find attached the report. I've added the columns described on top of the usual record table report.  In order to install the report:

1. Navigate to:  %APPDATA%\camelot   in Windows Explorer (or to the equivalent "Configuration" folder for other operating systems: https://camelot-project.readthedocs.io/en/latest/advanced_config.html#configuration)
2. Create a folder called "modules" in that directory, should it not already exist
3. Move the downloaded "extended_record_table.clj" file here
4. Go to Camelot -> Your Organisation -> Reports

You should now see a new Report called "Record Table (Extended)" in the menu, and see the expected columns in the report produced.

Let me know how it goes,

Chris

[Hannah de Villiers]

Hi Chris, 

This is absolutely incredible. I can’t tell you how much I appreciate your help in creating this for me. It will save huge amounts of time - thanks so so much. 

Hannah

<extended_record_table.clj>

[Alex Wolf]

Hi Chris, and all,

A big thanks for customizing this report.  I am working on estimating deer abundance based on the Jacobsen method*, and this report makes extracting the data needed (independent observation counts by life-stage and of individuals) very easy.

For others' benefit, I found that several fields (columns) of the export were displayed as '[Translation missing]', which I realized was where my Camelot project lacked fields that Hannah had defined for hers.  I was thinking of asking Chris to add my 'Individual' field, but realized that Hannah already had that field under a different name, 'individual-name'.  I edited my custom field name to match Hannah's above, and it started appearing in my reports as needed with the individual IDs I'd added under the original field names.

A big thanks to you, Chris, and Heidi for all you do for Camelot!

-Alex

* Jacobson, H. A., J. C. Kroll, R. W. Browning, B. H. Koerth, and M. H. Conway. 1997. Infrared-triggered cameras for censusing white-tailed deer. Wildlife Society Bulletin 25: 547– 556.  PDF available via Google Scholar.

[Alex Wolf]

Hi all,

This year I was having trouble getting the extended report to show up in Camelot.  Whether this is due to our using multiple datasets, or a change in Windows 11, or something else, I can't say, but it took me a while to troubleshoot the issue.  I eventually discovered that adding the 'modules' folder into C:\Users\<username>\AppData\Roaming\camelot solved the issue.  I'm not sure if it needs to be there as well as C:\Users\<username>\AppData\Local\camelot, or just in the Roaming folder instead of the Local folder, but I'm not messing with it to find out at this point.

Chris also helped me to diagnose another issue with multiple datasets and reports - I added three named datasets but left the Default database in place, with no data.  Reports wouldn't generate at all from Camelot, but Chris pointed out that they work from the browser - apparently this is a bug related to the empty default dataset.

Cheers,

Alex