Discrepancies in upload numbers

[Claire Waichler]

Hello WI community,

Our team has been using WI for many months and I keep encountering discrepancies between (1) the number of photos on my harddrive that I select for upload, (2) the number of photos that successfully upload and appear on WI and (3) the number of photos that appear on WI after some time. 

For instance, I had 2387 jpegs on my harddrive for a single deployment. I uploaded them in one go last winter, and 2372 images ended up in WI. I just went back to check the numbers and now there are only 2132 images associated with this deployment in WI. 

Two points/questions for the community:

1. Is there an expected margin of error to which we can relate our work?  Say, for every 100,000 photos that we attempt to upload, is there a percentage of inaccuracy that we can allow for, and still have the photos represent the field sample?  At this scale, it seems unrealistic to have to account for every last photo, so we just want to have a reference point, if possible.

2. What types of human and technical errors in the uploading process can be identified and how can they be avoided? 

Details going into our methods of troubleshooting for those who might be able to provide further suggestions:

• To verify the number of images that successfully uploaded, I go to "Identify" and filter down to the one deployment I am checking. I also crosscheck that no images from that deployment are already tagged by filtering in Catalogued as well.
• For 50,000+ photos uploaded, 0.62% of files that we had on the harddrive did not "appear" on WI after the initial upload. Per deployment, it ranged from missing 5.24% of images to showing  1.31% more images than we had in our drive. 
• If an upload was "off" from the harddrive number, we searched for photos by exact numbers and found them there (in a few cases when we received an error message that some photos had not been uploaded due to what was probably an internet glitch, we then carefully parsed out where the upload stopped and replaced the disrupted images. Those errors are not included in the rate I mention above). 

Big furry handshakes for any help,

Cal and the team

[Wildlife Insights]

Hi Cal,

Sorry to hear about these issues. We hope to make the upload process reliable enough so you won’t need to worry about missing or duplicate images! This detailed feedback is really helpful to understand what might be going on and I'd be happy to hear from others too. I just have a few follow up questions for you:

• When you see that there are fewer images in WI than you selected, do you receive an Upload report or does WI say that the upload was successful?
• For the deployment where there are only 2132 images now, is this a new issue or did you notice this happening in the past? Has it happened with other deployments?
• Can you invite suppor...@wildlifeinsights.org to your project as an Editor and let us know the name of the deployment where you’re now seeing less images? 

There are some technical issues that can happen during the upload, including points when the internet connection is lost. In this case, you may see fewer images in WI than you selected from your hard drive. We added the Skip duplicates option in the upload process so that it’d be easy to reselect all the images in a deployment in this case, and only reupload the missing images. However, this solution only works if the image filenames are unique within a deployment.

We’re also working on improving the Upload Report and building an Upload Notifications page to help you (and us) better track each upload. We think that this change will address some of the discrepancies that you’re seeing and we'll follow up when the changes are available (here and on our Features page). Thanks so much for your patience!

Nicole

Wildlife Insights team

[Andrew Sharp]

I would just like to add that I’ve also noticed this discrepancy, but it only started in the last couple months. The issue appears to be distinct from WI excluding duplicate images. 

Thanks,

Andrew Sharp

--
You received this message because you are subscribed to the Google Groups "Wildlife Insights" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wildlifeinsigh...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wildlifeinsights/418c1aff-e452-4cb8-915d-833ede9b7924n%40googlegroups.com.

[Wildlife Insights]

Hi all,

I just wanted to update everyone on this issue. We're currently addressing 2 bugs that can lead to a discrepancy in the number of images selected for upload and the number of images you see in Wildlife Insights:

1. There's a bug where only Project Owners can see images tagged as "Human" in the Identify tab. Project Editors, Contributors, Taggers and Viewers cannot see images of humans in the Identify tab even though the images are present in the project. So if you upload images and you aren't a Project Owner, you'll see less images in the Identify tab than you uploaded. Our team is currently fixing this issue.

2. If there's a drop in internet connection, Wildlife Insights will retry an image upload. In some instances this results in duplicates uploaded and can lead to more images appearing in Wildlife Insights than uploaded. We've been working on a new upload process that we believe will be more reliable and we're also thinking through ways that can help you quickly identify and delete duplicates.

I'll post more updates here. Thanks everyone for your feedback.

Nicole

Wildlife Insights team

[Wildlife Insights]

Hi Claire and all,

I just wanted to let you all know we released a few new features to help minimize any issues with the upload to image-based projects. You can find an overview of changes here. 

Thanks again for your feedback!

Nicole

Wildlife Insights team

[Dylan Hubl]

Hi,

I have recently experienced this same issue of a discrepancy in the number of uploaded images and the number of images I have on my hard drive for a particular deployment. The issue has now occurred twice on the same deployment but with a different number of images uploaded each time. This deployment has 54,290 images distributed in six folders. The first time I uploaded the images it came up about 160 images short. Each of the six uploads said "upload complete" and I wasn't able to find anywhere where I could see the upload reports to try to figure out what images were missing. So, I deleted the deployment and re-uploaded all of the images. This time, I am 6 images short.

I am listed as an owner of the project so I don't believe photos with humans are being screened from me. 

Is there anyway to see an upload report, for a sequence-based project, even if the upload says complete?

Best,

Dylan