Improving the reliability of Research Grade

[Ben Phalan]

I have some small suggestions which could help to cut down on mis-identified observations reaching Research Grade: making it a little more difficult to simply "agree" with an ID.

Something that seems to happen quite frequently is that someone adds an incorrect ID, and then someone else agrees with it, presumably through carelessness, or not really taking the time to assess the ID properly themselves. For example: http://www.inaturalist.org/observations/4718459

In these cases, there is really only one ID, and by making it easy to "agree", it is counted twice, leading to the observation reaching Research Grade, being added to GBIF, and largely disappearing from further scrutiny.

I have a few thoughts on how to cut down on this problem:

1. Remove the "Agree" button from the thumbnail view on the "Identify" page. Even for easily-identified taxa, I really think it is necessary to view a larger version of the image by clicking on the observation, to check if there are multiple images, comments, that the location makes sense, and so on. Do any of the power users really identify from the thumbnail view?

2. Change the text of the "Agree" button to "ID is correct" or something similar. The idea would be to remind people, psychologically, that they are not simply nodding along to someone else's suggestion, but making an independent evaluation of the ID.

3. As a more extreme version of #1, and an alternative to #2, remove the "Agree" button everywhere, so that users would have to type the taxon name to add an ID.

The downside of these suggestions (especially #3) would be that it would become a little harder for people to agree with IDs, but to me, that seems more positive than negative.

I'd be interested to hear what others think.

Ben

[Charlie Hohn]

the downside of #3 is most of the power users would dramatically reduce their IDs. I'd probably do a lot less ID help. It adds a lot of time and a lot of us go through hundreds in one sitting (and hopefully aren't where most of the mis-IDs come from though everyone makes mistakes).

Please don't do #3.

An alternative proposal: when setting up the ID Please algorithm, have it show observations that have research grade (from only 2 IDs) sometimes, maybe 1 out of 10, randomly. Getting them a third ID will detect most of those sorts of errors. 

Another alternative proposal: create student accounts - based on people being assigned to use iNat not on age - and give those accounts only 0.5 of an agree or something.

[Ben Phalan]

Thanks Charlie, I agree #3 is probably a step too far. I included it more to start a discussion.

I like the idea of randomly bringing RG observations back for review, and also the student account idea, which I know you and others have raised before. What do you think about #1 and #2?

[megatherium]

I like charlie's idea about showing observations that only have 2 IDs, and also dislike #3.  

#1:  As a "power user" I do indeed use thumbnail view to ID.  I have a particular bee in my bonnet about North American squirrels, which always have a high volume of observations posted.  Several species are very common and very distinctive, so with the filter set to an appropriate geographic area and any photo quality greater than "horrible" I can get through a lot of observations very quickly.  

--
You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist+unsubscribe@googlegroups.com.
To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

[Charlie Hohn]

No worries Ben, it is a good discussion, and I shouldn't react too strongly. I wouldn't stop doing ID help in that case I'd just be slower at it. I realized that i was doing that while helping with the 'blind ID test' project and it didn't slow me down too badly. But i think it would annoy people and also there are transcription errors with re-writing the names as well.

#1 - i don't care too much, i agree that it can be hard to tell some species from thumbnails, but with some plants it can definitely be possible to get a good ID from something like that. I don't use the thumbnails for ID that often, i page through the observations in the ID page and enlarge them when necessary. That being said I feel like the bad 'socially agreeing' IDs probably don't come from that page very often, but rather from people looking at their friends' observations. or sometimes it's just a mistaken click, or sometimes the person is just plain wrong. 

#2 - don't have a feel for it either way. It wouldn't bother me!

C

To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.

[Scott Loarie]

Hi Ben,

Thanks for your thoughts on this. I'd also like to take steps to improve the reliability of 'Research Grade'. 

One of the things I'd love to do before making any changes though its to first rigorously review the quality of a sample of observations and then do an analysis to:

(1) try to understand the characteristics of an observation that drive the quality of observation IDs (e.g. things like the past reputation of the IDers) It could be, as Charlie proposed, that most obs with 'agree' abuse come from new users (e.g. users without much earned reputation on the site) or maybe it comes from 'accidents' from experienced users. Maybe this info could guide which of your solutions is the most suitable.

(2) try to quantify the accuracy of obs IDs on iNat (all obs and also just the Research Grade subset) so we can make statements like Research Grade observations on iNat are 95% accurate etc. and then, if we think the Research Grade threshold is too low (or two high) consider adjusting the Research Grade algorithm accordingly (e.g. maybe there's a way to incorporate earned reputation into research grade so that more input than 2 students alone is necessary to achieve it)

We've taken steps to organize this rigorous review as discussed here:

https://groups.google.com/d/msg/inaturalist/m5M8T4oXyn8/IE7ZGiGfAAAJ

and explained more here (iNaturalist Identification Quality Experiment)

http://www.inaturalist.org/pages/identification_quality_experiment

We're waiting on 2 things re: the above experiment:

1) a functionality change https://github.com/inaturalist/inaturalist/issues/1318

2) gaining Internal Review Board Exemption for human subject research

that I hope will be sorted this week.

When thats done, I'd love to move forward with the actual study and it would be awesome to bring data from the study into this conversation to help you guys make recommendations

Scott

--

--------------------------------------------------
Scott R. Loarie, Ph.D.
Co-director, iNaturalist.org
California Academy of Sciences
55 Music Concourse Dr
San Francisco, CA 94118
--------------------------------------------------

[Charlie Hohn]

I'm confused... if i do blind plant ID it is considered human research? Anyhow... yeah i've held off on those IDs for that functionality change... but have done some already.

To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.

To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "iNaturalist" group.

To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.

To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

[Ben Phalan]

Hi Scott,

I really like the idea of your review to try and nail down where most of the problems really arise, and look forward to seeing the results of that.

In the meantime, it looks like of my suggestions, #1 and certainly #3 would not be acceptable to the community, whereas #2 might still be worth considering.