Lets talk about mega reviewers

[johnnybirder]

Hi all, 

I want to talk about mega reviewers for a moment. I really like the "identify" page on iNat - it has made reviewing much easier than before. Perhaps too easy. The current system enables reviewers to quickly amass a great number of IDs in a very short amount of time. This process facilitates uncritical IDs. That is, reviewers can easily "agree" from the main Identify page without looking at the range maps or other detail of the observation. In many cases, the uncritical reviewers miss subsequent IDs, often a subspecies ID or a subsequent correct ID, leading to spurious agreements and disagreements.

I still like the system as is. For example, I often go "into" the first observation on the Identify page, from where I flip through the observations with my arrow keys while refining/correcting IDs. After that I go back to the mail Identify page to agree with the rest. So I dont want to see that changed. But for the sake of better reviewing, I wonder if there is a need for a cap on the amount of IDs a reviewer should be able to make in a day. 

Kind regards,

Johnny

[johnnybirder]

As an aside, @jakob and I call the practice of IDing from the main page (without looking at range maps ect) thumb-nailing - sometimes I'm guilty of it too e.g. https://www.inaturalist.org/observations/8694045. It is this practice that I would like to see limited.

J

[Charlie Hohn]

capping number of IDs would be annoying. And it wouldn't change how long I looked at each one, it would just mean i made less IDs. Instead we could look at trying to get a third ID on a subset of observations, continued work on Atlases to detect out of range observations and prioritize for further evaluation, or some sort of merit-based reputation system. And also everyone makes mistakes, and unless they are prolific, better just to add a correct ID and move on. Mistakes occur at pretty high frequency in 'professional' science as well.

I'm ok with thumb nailing being limited. Making the range map a bit bigger on the ID page would be nice too.

[James Bailey]

If anything, more IDs means reviewers are more likely to catch mistakes. Capping IDs per day will not stop this problem, but rather slow the process.

[tony rebelo]

Capping the number of IDs would interferer with inviting taxonomic experts onto the system to make IDs.  I can for instance get the southern African Hopliini expert to review IDs and he can do about 50-100 per hour (many can only be done to generic level).  To have to tell him that sorry, he has to break the number of observations to 50 per day for two weeks would seriously upset him, enough to not bother doing IDs at all.

Tony 

[Cullen Hanks]

I imagine iNat can look at the data behind this, but my guess is that the people who do the most IDs are also the best identifiers.

-Cullen

--
You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.
To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

[Calebcam]

I would like to see the "Agree" on the thumbnail removed, but I don't think there should be any limit or cap. 

Caleb

[Johnny Wilson]

Yes. I think removing the "agree" on the thumbnails might be a great way to address the problem (bad IDs) while also avoiding the drawbacks mentioned.

Cullen, I guess it depends how you define "best identifiers". From a quick look at the piecharts some of the biggest IDers dont actually "improve" IDs - mostly just adding supporting IDs. Supporting IDs certainly don;t qualify as a measure of "best identifiers" since that the part of problem I'm addressing - uncritical supporting IDs. I guess a good way to quantify "best identifiers" would be to check whose maverick IDs eventually become leading IDs. 

Kind regards,

Johnny

***********************************************
Johnny Wilson, Ph.D.

Chapel Hill, NC

Mobile: +1-919-272-3522
URL:     http://www.johnnybirder.com

Reprint pdfs available at ResearchGate: https://www.researchgate.net/profile/John_Wilson35/

 

--
You received this message because you are subscribed to a topic in the Google Groups "iNaturalist" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/inaturalist/vg2Qhp5JFtI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to inaturalist+unsubscribe@googlegroups.com.

[megatherium]

I use the Agree on thumbnail view a lot, and don't think it's an issue with a modicum of care.  For example, Eastern Chipmunks are very common, very commonly posted, and completely unique & distinctive in appearance in the continent.  With the filter set to North America to remove any rare Siberian Chipmunk observations, a week of chipmunks can be confirmed in a few seconds. Same for European and Alaskan squirrels and pretty much any New Zealand mammal.  

My personal wish would be for the name of the mapped location to appear in the thumbnail view, but that's probably me being lazy about clicking through on every prairie dog to check the range.

To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist+unsubscribe@googlegroups.com.

[Cullen Hanks]

Yes, totally agree, by best I mean accurate not amount.  How you measure or identify that is another question.  But as I said, my guess is just based on my anecdotal experience and not the data.   I get the impression that the iNat team is looking at the data, taking a data driven approach, and it will be exciting to see how review evolves.  

Thanks,

Cullen

[Ian Toal]

Personally, if an observation has one or more confirmations, I don't add anything to it unless I am pretty sure it is the wrong ID. So if I scroll through Moths of NA, anything that has 'research grade' attached to it I ignore. However, I value an accurate ID more than I do my ID numbers.

Ian

[Donald Hobern]

I certainly tend to add identifications in big batches, particularly for Australian Lepidoptera.  I bring up all observations without research grade identificaitons for a family, genus or species and go through quickly confirming all those that are obvious, then switching to add identifications to any others I can.  I do this at irregular intervals, but may spend a couple of hours doing so.  It would not be helpful to hit a limit at such times.  For what I do, the "agree" on the thumbnails is a major help.

Donald

[Russell P]

I'd like to see the thumbnails be larger and fewer per page.

[Colin Purrington]

If the analysis at iNaturalist HQ shows far greater % of mistakes when IDs made via thumbnailing, perhaps the condition of ID could be appended onto the "agree with ID" phrase. Might make subsequent viewers slightly more hesitant of hitting the Agree button. I.e., subsequent viewers would know that an ID agreement was made without looking at all the photographs of the observation, without seeing collection location, etc. 

 

[Chris Vynbos]

Another way to help prevent people making agreements that are not appropriate would be if the thumbnails of the identity page has icons showing the number of comments and IDs. This would flag a user that the agreement they are about to make is possibly a contentious one. 

[Frank Fogarty]

I would agree that I've seen this as an issue, with several 'mega-reviewers' as repeat offenders that have needed corrections on hundreds of their IDs. There are some mega-reviewers who seem to really know the taxa they are reviewing and others not so much. Most of these misidentifications seem to stem from the observer selecting the wrong taxon and these mega-reviewers just clicking 'agree' and the record going to research grade. A lot of these are on relatively straightforward identifications, with the mis-ID being for a taxa in the wrong family or out of range, so I don't think this is just an issue of some taxa being difficult to accurately identify from photos.

My thought on the best solution would be upping the requirement for Research Grade to 3 'agrees', or at the least requiring 2 that aren't the observer. That wouldn't fix the issue in every instance, but I think it might prevent many cases (seems like the odds of getting 2 mega-reviewers in a row confirming bad IDs is a lot lower).

My two cents,

Frank

[stanw...@gmail.com]

I strongly support having three "agrees" as a bare minimum for Research Grade.  Two is really one because one of the two is the submitter and it seems to me having two others agree with the submitter should be a floor.

stan

[Charlie Hohn]

well, that's only an issue when the observer doesn't know w hat it is to start. IE: the first person puts in a pine as 'plants'. The second person puts it in as 'red pine'. The first person agrees but doesn't really know red pine. That's really only like one ID but gets research grade.

But there's a second scenario: i see a tree and am confident it is red pine. Someone else agrees with me. That's two ids.

I'm not sure what the answer is to #1 because it would be hard to set up the algorithm to account for that. However, due to changes in the community ID the tag now changes to the species when soeone adds an improving ID causing less motivation to agree. For instance someone identified a bunch of flies for me... i don't know anything about flies, if they ID to species i can get that ID on the map without agreeing with it just by leaving it alone.  If that makes any sense.

[Reuven Martin]

One feature I'd like is the ability to search or sort observations by number of identifications in identify mode. I'd use it to find research grade observations with only two identifications, as there are much more likely to be misidentified.

[Brian G]

I second Reuven Martin's suggestion about being able to filter observations with only two agreeing identifications. I also agree with the idea of making the "Research Grade" qualification more...scientific...or at least more comprehensive than two anonymous people agreeing.

Brian Gooding (@briangooding)

[Charlie Hohn]

well, i think we'd lose R.G. on about half our observations if we did that which would be a bummer. I've started doing plant ID without the filter that excludes research grade observations to see if i notice a lot of problems. So far i notice occasional problems especially surrounding those same student projects that accentuate all these issues... but not enough that imho we should bump the RG criteria to 3 which doubles the number of IDs we need which are already in low supply. 

[Donald Hobern]

I agree with Charlie.  In my experience, the problems with the process are rare and often corrected over time by those who take time to review all records of a species.  I think that the current threshold for research grade is correct, but it may be valuable to highlight the strength of agreement more clearly based on number of agreements and degree of unanimity.  GBIF would certainly be interested in receiving a data element which assigned a measure to identification confidence.

Donald

[Charlie Hohn]

One option would be a setting so that the 'random' filter or whatever would occasionally throw a RG record in with the others, even when set on 'needs ID'. This could function to look for any systematic errors and also we could even collect accuracy stats or something. Just a thought.

[bouteloua]

(Could be a it's a plant bias but) I almost never practice thumbnailing (i.e. clicking Agree from the thumbnail view on Identify). It's quite fast already to use the right arrow and "a" keys and I'm too concerned I'd be missing important locality, observation description details, other IDs, or comments that could affect the ID. I also frequently find that the user may have uploaded additional irrelevant photos of different species, but because they are "hidden" behind the 1st photo, they don't get caught. The only easy way to find these mistakes is from the taxon page photo page, where all photos, rather than just the 1st one, are displayed.

I would love to be able to search/sort by number of IDs or comments.

I'm also eagerly awaiting the Computer Vision ID label, which should make folks more tentative about hitting agree.
If the thumbnail Agree button is retained, perhaps the observations should indicate: has only one ID, ID is suggested via Computer Vision (attached)?

cassi

[tony rebelo]

iSpot had a cool feature, but it could only be used offline.  Although it used the iSpot reputation system, I think it could work quite well without one.

In fact, it could be simplified simply to the "score" shown in the About section of a community ID, except as that is scaled from 0 to 1, it should be the "score" X "identification count".   Anything below 1 would be highly suspect, and anyone could choose their own cutoff point (e.g. 3 for a reasonably-sure ID) at which they accepted an ID, or several cutoffs depending on what the data are to be used for.

[anisop...@gmail.com]

I don't really like the idea of needing 3 'agrees' to get to research grade.  While this might be great for the more common species, but there are some rare plant species that few people know that have a hard enough time getting to research grade as it is.  Having to get a 3rd ID would be extremely difficult.  For some, it would make research grade practically unachievable.  If it could be applied to only the most commonly observed taxa, that would probably be fine.  

I really like the idea of a reputation system or something like it that gives extra weight to those who are experts on a certain taxon (not sure if there is a way for it to only apply to that taxon, but that would be nice as well) and lower weight to individuals who consistently make bad identifications.  I think there might be some potential for abuse, so the structuring would have to be carefully considered.

Nathan Taylor

On Tuesday, February 20, 2018 at 11:46:14 AM UTC-6, Frank Fogarty wrote:

[anisop...@gmail.com]

As to "thumbnailing", I'd be in favor of removing the agree icon below the thumbnail.  I almost never use it as I'm usually looking for plant hairs.  I only use it in those rare instances where I've set a geographic boundary, the species is very distinctive, and the first photo is exceptional.  Honestly, even then I don't use it much because looking at the observation is so fast.

[Charlie Hohn]

I also barely if ever use the thumbnails to do IDs, it seems an awful idea for most plants anyhow. Unless you've got a close up of a super distinctive feature such as a white pine cone or i dunno, a moose's face, it's not really a good idea.

[Sam Kieschnick]

I'd like to chime in from another standpoint of ID'ing, and that's public engagement.  I'm fortunate to get to work with a lot of the public, and I get to speak to a lot of the observers in person.  I must be frank, people LOVE it when their observations are ID'ed -- even past the 'research grade' requirements.  It creates a positive feedback loop -- their observation is seen and validated by an agreeing ID (or multiple agreeing ID's), and it inspires them to make more observations.  For me, my click of an 'agree' on something as obvious as an extremely common bird creates that positive feedback loop.

EX: https://www.inaturalist.org/observations/9523397

Anyone with a kindergarten grade field guide can verify that it is indeed an alligator...  But, when I get to talk to Zach, he is stoked that other people looked at his observation and validated it through ID's!  He's actually giddy about it!  This is a positive feedback look and encourages the citizen scientists to keep collecting data.   

Public engagement may not have the same tangible data value as a GBIF record, but to me, it's quite valuable in nourishing a community of naturalists. 

My two cents!

~Sam

[Tony Iwane]

I'm in favor of keeping Agree on Thumbnails to quickly add IDs to obvious photos (I generally use a geographical restriction, so that does help) and get them out of the way so I can focus on the more interesting stuff. 

Tony

[Charlie Hohn]

just out of curiosity what sort of taxa do you use it for? There are times I can do it but it is rare, and it's so fast to flip through the 'cards' that i don't see a reason not to

[Tony Iwane]

Obvious birds, reptiles, amphibians, and insects (like monarchs). Some easy plants, but I don't ID many plants 'cause my knowledge there is fairly slim.

Tony

--

You received this message because you are subscribed to a topic in the Google Groups "iNaturalist" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/inaturalist/vg2Qhp5JFtI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to inaturalist+unsubscribe@googlegroups.com.

[megatherium]

On wifi flipping through the cards isn't very fast, so I make heavy use of geographic filters + thumbnail view.  I primarily ID mammals and invasive taxa of all stripes.  Semi-decent photos of Eastern Chipmunks and Fox Squirrels are extremely obvious in thumbnail view, and something like a blue hydrangea in the Azores or coconut palms in Hawai'i is also readily apparent.  

On Fri, Feb 23, 2018 at 10:32 AM, Charlie Hohn <naturalis...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "iNaturalist" group.

To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist+unsubscribe@googlegroups.com.

[Charlie Hohn]

hmm... makes sense. It's interesting how different taxa and different styles affect how people use the site. My internet is super slow at home, but i mostly notice it when enlarging a photo beyond the cards

To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.

[cassi saari]

​Examples of what I was talking about:

https://www.inaturalist.org/observations/9458800
https://www.inaturalist.org/observations/5639689
https://www.inaturalist.org/observations/3197005
https://www.inaturalist.org/observations/5960865
https://www.inaturalist.org/observations/2967550
https://www.inaturalist.org/observations/4138 (maybe?)​

[tony rebelo]

It varies.  Even in southern African Proteaceae,  the genera Protea and Mimetes can easily be identified from "thumbnails", but Serruria, Leucadendron usually require several pictures (so the ID tool is better), and Spatalla and Sorocephalus require almost microscopic closeups of flowers for an easy ID (so if the closeup is pic 1, then the thumbnails will do).

[tony rebelo]

Would a simple way around this issue, not be to simply adopt Picture_1 as the "type" for the observation and disregard all subsequent photographs?

.  

All this requires is a change in "convention".

That way at least the observation is not wasted: at least 1 ID can be "rescued" from the mistake.  If the observer returns at a future date they can remove the other photographs, or duplicate-create new observations.  If they never return then there is no loss.   It also means that an ID can be posted (for Pic 1)  while waiting for the observer to make the corrections, instead of having to wait until they make the edits (a few minute, hours, years later, or never).

[cassi saari]

I'm not sure what you're suggesting here. Do you suggest that all photos on an observation besides the 1st one be removed from the taxon page?  That would be quite a big change as it would require programming efforts, would disincentivize taking additional photos at various angles, and so would probably warrant a new thread for discussion.

cassi

[tony rebelo]

No.  Just a change in convention.  Ignore pictures after picture one if they are not of the same organism.  No need for any programming.  Also programming would impinge on users rights and expectations.   Many users will happily fix the observations once they discover how it works:  it is just the odd user who vanishes and never returns, or refuses to make a change. 

This is of course helped, if identifiers will co-operate and only make identifications for the first picture only and not post notes on the IDs of the other pictures.

[tony rebelo]

Cassi: Sorry: several issues with your query, now that I reread it. 

Firstly this is not the taxon page we are talking about but an observation.  I thought we were discussing observations with multiple species on subsequent photographs.  Taxon pages are not what we are discussing, I thought

It is the first picture on an observation that always displays on the thumbnail.

I was definitely not advocating the removal of anything at all!  I was merely suggesting that when there are many species on an observation, only the first picture counts for making an ID (i.e. it becomes the "type" for the observation) - and that all subsequent pictures are just ignored.  Ignored: not removed or hidden or flagged - certainly nothing programmatical!

On Saturday, 24 February 2018 18:12:38 UTC+2, bouteloua wrote:

[Charlie Hohn]

Tony, the issue is that if you have a picture of a Springbok then one of a chicken in the same observation, then what you get is the photo of the chicken imported into the set of springbok photos that inform the species page, the algorithm, etc etc etc. All photos from an observation are imported not just the first one. If we stop doing this we lose thousands of perfectly good photos in different life stages and plant parts and etc, just as cassi said. So we don't want that photo entering that system as a springbok Ope other option would be to decide that the first photo as the 'real' photo and creating a check box either in the offending photos or in data quality assessment when there are more than one species, and it would hide the photos just like the copyright infringement image does (but without banning the user). Maybe send the user a notification too to ask them to change it.

[James Bailey]

The simplest solution would be to remove the "agree" button from the thumbnails. So then at minimum you have to click on the observation on the Identify page. There, you can still hit a button to "agree" on the keyboard, or with the mouse, but you at least have to look at the range, and larger photos.

[tony rebelo]

But that is what I meant.  A single page is determining policy for the whole site.  People are redesigning and reprogramming the site for this one page.

Accept a bit of slack on that page, and restore a more balanced approach across the entire site.

(in case you have not figured it out, I dont use those pages much)

[Jane Widness]

Seconding Charlie -- just found a jaguar photo in the mountain lion taxon photos, the offending observation is here: https://www.inaturalist.org/observations/4846592.

I kicked out the jaguar photo by disagreeing with the mountain lion ID, but that doesn't solve the bigger issue