Most efficient method for uploading hundreds of observations with thousands of photos

[Joe Ledvina]

Hi,

I've been working with a PhD. student, collecting plants to inventory a complete flora of a natural area. We have a spreadsheet with herbarium-appropriate details of the 1003 specimens from last year. I took photos with my DSLR, and have an average of over 5 field photos per specimen. Almost all specimens have been keyed, so we're ready to upload to iNaturalist. From what I can tell, the most automated workflow would be:

1. Use a script and exiftool to get metadata into the photos -- GPS coordinates in the appropriate fields, taxon name and everything else as IPTC "subject" keywords

2. Crank through the web uploader, uploading batches, manually clicking on images to group them and specify them as single observations

I'm pretty sure I can figure out the first step, but the second step seems overly tedious and prone to error, so I'd much rather have it fully automated. I've found similar requests, but no workflow that would avoid the manual assignment of photos to observations. Is this possible?

Alternatively, I can think of another workflow that could work well:

1. upload the observations as a csv

2. export the observations from iNaturalist as a csv (to get the iNaturalist observation numbers)

3. tag (or rename) the images with the iNaturalist observation number

4. upload all of the images

Is there any way to automatically group photos as observations when uploading?

Joe

p.s. I have seen pyinaturalist and I think that I could potentially use that (and would use it instead of steps 3 and 4), but I was hoping for a solution not requiring another 3rd-party tool besides an exif tag editor.

[jesse rorabaugh]

This pretty much seems the use case for pyinaturalist. While I have yet to try it, doing all the steps as a python script seems credible. How do you have stored which photos are of the same plant? If it is data is consistent it should be possible to get a script to pull the data from the spreadsheet and exif then submit the observations with that data.

[Joe Ledvina]

Thanks for the reply, Jesse.

The photos were renamed after each collecting trip, to match a column (collector number) in the spreadsheet plus a trailing count (e.g. specimen DLS001 has "DLS001_01.jpg". The only useful exif data would be time of day, which is not very important. Everything else is in the spreadsheet already, including GPS coordinates.

My plan (if pyinaturalist was the only way) was to

1. upload the csv using the iNaturalist website
2. export all of the users observations to csv to get the iNaturalist observation numbers
3. copy the full paths of the image files to a column in Excel
   
4. make a column with just the collector number from the file names using a formula in Excel
   
5. use that to pull in the observation numbers
6. add columns containing the other parts of the pyinaturalist commands for uploading a photo to an observation
7. run that as a script in python
   

I was hoping for something more user-friendly, friendly enough for command-line-averse users, since there does seem to be demand for streamlining very bulky uploads.

Even a flexible python script to create the pyinaturalist commands (1. enter path to folder with images, 2. enter full path to spreadsheet, 3. enter how many characters in the filenames match the first field in the spreadsheet) would be simple enough for many. Unfortunately, I don't know python, only R.

search keywords: bulk upload, batch upload, big upload, many photos, backlog

[jesse rorabaugh]

I would be inclined to reduce that to:

1. Write a python script

2. Run a python script

Step 2 seems easy enough, step 1 of course is not so easy but still probably easier than the steps you described. For example, your step 3 and 4 is the sort of thing which would probably be five lines of code in python. The upload observations and upload photo commands are already coded.  The real problem is that no one has uploaded code for similar mass uploads using pyinaturalist so you are stuck writing from scratch.

I at least have basic python skills and have been looking for a use case to get myself to learn pyinaturalist. If you aren't in too much a hurry I suspect I could write you a script which would upload everything at the push of a button. Might take me a couple weeks and I am promising something I have never done but from what I read in the description on pyinaturalist it wouldn't be the most difficult script I have written. If you are interested, send me a message. I will need the images for a few observations and the spreadsheet with the data.
https://www.inaturalist.org/people/52839

[tony rebelo]

I am also interested.  I could in theory do it, but dont have the time at present.  I have gotten myself stuck with the authentication protocols, but I  have just realized that I probably dont need any of that at all.  I could just ditch the APIs and mimic a screen session.

[Johannes Klein]

Hello Joe,

yes, this is absolutely possible with a custom python script (and pyinaturalist is indeed a good entry point).

To all interested:

I am in the process of writing a blog post on exactly how to do this at the moment. Following the proposition in https://bsapubs.onlinelibrary.wiley.com/doi/full/10.1002/aps3.1193 . It will take some more time to finish it but I expect it to be published around one month from now. I will keep you updated on this. I would also very much appreciate any feedback on the blog post once online.

My example will use real data from a biologist, so that it may be a good entry point for you to customize your scripts. There will always be additional steps required to customize the script to your needs though.

After the blog post I am available to hire for consulting or coding your batch upload scripts.

[Joe Ledvina]

Hey, Johannes,

Thanks for letting us know that you've been working on that! Do you have any updates on when you'll post something?