obs jakob and wouter have ID'd
202 views
Skip to first unread message
Scott Loarie
Mar 9, 2017, 4:06:17 PM3/9/17
to inatu...@googlegroups.com
I'm adding this to a new thread to keep 'should disagreeing with the
community ID make an observation "casual"?' on topic,
but I thought the exchange between Wouter et al about disagreements
interesting. Just to provide some numbers to Jakob's claims, there are
3,598 obs that both Jakob and Wouter have ID'd and it looks like Jakob
and Wouter's IDs are 'maverick' to each other (this is a word we've
been using to describe IDs that are not descendants or ancestors of
one another) only 2 times:
http://www.inaturalist.org/observations/812934
http://www.inaturalist.org/observations/5101903
So thats a 99.94% agreement which is a disagreement rate of < 1 in 1,000
I'd be very interested in trying to quantify the average 'error rate'
in iNat obs IDs. The biggest challenge towards doing this is finding a
metric of 'truth' that we can use to filter obs with IDs we are very
confident in for analyses. As has been pointed out, even 'experts' on
the site seem to have error rates of at least 1%. This is probably do
to a combination of real disagreement, accidentally clicking on the
wrong thing, confusion about the subject of an observation (the bee or
the flower), or taxonomic issues (do you mean the genus sensu-lato or
sensu-stricto?)
I've been playing with an approach of finding 'truth' that looks at
sample of 20k observations that have been ID'd by at least 3 IDers
like jakob with lots of ID's and low empirical error rates (ie novel
IDs that the community rejected over novel IDs the community embraced
- Jakob's empirical 'error' rate using this approach is 3% which is
the same as mine). If we call truth the common ancestor of these ID's
than you can assume that these obs are mis-ID'd the probability of all
3 people wrong ~ (1-0.03*0.03*0.03)*100 << 0.01 = 1 in 1,000
Assuming this 'truth' set is a somewhat representative sample, we can
use it to try to predict what kinds of observations tend to be
mis-ID'd and see what factors contribute to more accurate IDs (e.g.
things like rank, taxonomic clade, location, reputation of the IDer).
If we were to implement some sort of more sophisticated test for what
obs we think are 'research grade' over what we're doing now, I'd
personally like it to be informed by analyses along these lines.
-Scott
--
--------------------------------------------------
Scott R. Loarie, Ph.D.
Co-director, iNaturalist.org
California Academy of Sciences
55 Music Concourse Dr
San Francisco, CA 94118
--------------------------------------------------
community ID make an observation "casual"?' on topic,
but I thought the exchange between Wouter et al about disagreements
interesting. Just to provide some numbers to Jakob's claims, there are
3,598 obs that both Jakob and Wouter have ID'd and it looks like Jakob
and Wouter's IDs are 'maverick' to each other (this is a word we've
been using to describe IDs that are not descendants or ancestors of
one another) only 2 times:
http://www.inaturalist.org/observations/812934
http://www.inaturalist.org/observations/5101903
So thats a 99.94% agreement which is a disagreement rate of < 1 in 1,000
I'd be very interested in trying to quantify the average 'error rate'
in iNat obs IDs. The biggest challenge towards doing this is finding a
metric of 'truth' that we can use to filter obs with IDs we are very
confident in for analyses. As has been pointed out, even 'experts' on
the site seem to have error rates of at least 1%. This is probably do
to a combination of real disagreement, accidentally clicking on the
wrong thing, confusion about the subject of an observation (the bee or
the flower), or taxonomic issues (do you mean the genus sensu-lato or
sensu-stricto?)
I've been playing with an approach of finding 'truth' that looks at
sample of 20k observations that have been ID'd by at least 3 IDers
like jakob with lots of ID's and low empirical error rates (ie novel
IDs that the community rejected over novel IDs the community embraced
- Jakob's empirical 'error' rate using this approach is 3% which is
the same as mine). If we call truth the common ancestor of these ID's
than you can assume that these obs are mis-ID'd the probability of all
3 people wrong ~ (1-0.03*0.03*0.03)*100 << 0.01 = 1 in 1,000
Assuming this 'truth' set is a somewhat representative sample, we can
use it to try to predict what kinds of observations tend to be
mis-ID'd and see what factors contribute to more accurate IDs (e.g.
things like rank, taxonomic clade, location, reputation of the IDer).
If we were to implement some sort of more sophisticated test for what
obs we think are 'research grade' over what we're doing now, I'd
personally like it to be informed by analyses along these lines.
-Scott
--
--------------------------------------------------
Scott R. Loarie, Ph.D.
Co-director, iNaturalist.org
California Academy of Sciences
55 Music Concourse Dr
San Francisco, CA 94118
--------------------------------------------------
AfriBats
Mar 9, 2017, 4:51:21 PM3/9/17
to iNaturalist
Quick follow-up, Scott: the disagreement b/w Wouter and me was definitely quite a bit higher (10+ observations), which however is no longer visible because I updated my IDs where I was clearly wrong. The same now applies to the 2 observations listed by you above because I've update my IDs accordingly.
As mentioned in the other thread, I think that iNat's equitable (non-expert) system works fine. There are certainly several groups (especially tropical inverts) where iNat would hugely profit from attracting more specialists to have the frequently ambitious IDs brought down to realistic levels, so I expect that these groups would have much higher error rates compared to birds, herps, large mammals and the like.
Error rate should get higher the more a user moves out of his/her comfort zone. I usually stick to groups and regions where I think I have at least some basic knowledge, but I see other users obviously venturing into groups where they seem to have little if any expertise at all. The other extreme are probably specialist who will strictly stick to their turf and neither look left nor right. How this plays out in terms of an average error rate among the entire iNat user community would be very interesting to know!
I think the only way to measure iNat's (and other platforms') error rates is a random, global sample of RG observations, and to have these IDed by a bunch of independent specialists. Would be a nice publication to work on, ideally comparing different systems. All in the spirit to get rid of alternative facts ;-)
Jakob
As mentioned in the other thread, I think that iNat's equitable (non-expert) system works fine. There are certainly several groups (especially tropical inverts) where iNat would hugely profit from attracting more specialists to have the frequently ambitious IDs brought down to realistic levels, so I expect that these groups would have much higher error rates compared to birds, herps, large mammals and the like.
Error rate should get higher the more a user moves out of his/her comfort zone. I usually stick to groups and regions where I think I have at least some basic knowledge, but I see other users obviously venturing into groups where they seem to have little if any expertise at all. The other extreme are probably specialist who will strictly stick to their turf and neither look left nor right. How this plays out in terms of an average error rate among the entire iNat user community would be very interesting to know!
I think the only way to measure iNat's (and other platforms') error rates is a random, global sample of RG observations, and to have these IDed by a bunch of independent specialists. Would be a nice publication to work on, ideally comparing different systems. All in the spirit to get rid of alternative facts ;-)
Jakob
James Bailey
Mar 10, 2017, 7:03:29 PM3/10/17
to iNaturalist
One thing to consider though, is "truth" on iNaturalist really that different to other museums, sites, and aggregate databases? The practice behind it is different of course, here you have a community of experts, non-experts, and other users who pool together an ID. That's a little different to how, say, a museum specimen is identified, and unlike Calflora which doesn't take ID corrections unless you are a registered professional. But even in physical cases, experts make mistakes, and IDs are altered, changed and discussed variously, just like on iNaturalist.
I guess this is somewhat besides the point of the discussion here -- but just worth keeping in mind.
Oh and yes, I'm one of the guilty parties who tries to challenge themselves and jump outside of their comfort zone frequently...but I try not to go beyond family or genus unless there seems to be no doubt on a species ID. A few cases where I have stepped too far, but I try to be more careful these days.
James
Matthew Muir
Mar 12, 2017, 12:33:51 PM3/12/17
to iNaturalist
This is REALLY interesting Scott. It's really great to be able to communicate about the error rate across the site. Transparency is good, and would also highlight taxa where expertise is most needed (higher than the average error rate across the site) vs taxa where the non-expert system is working satisfactorily.
I would love to learn my own accuracy rate, by taxon group, and have sometimes wished for a rating for other users. As Jakob points out, a more useful metric would take into account past IDs that have been updated or withdrawn, not just the current state of disagreement. Cheers, Matt
wouter teunissen
Mar 12, 2017, 4:15:51 PM3/12/17
to iNaturalist
I only scanned quickly this message, not yet time to read it propperly.
But I assume you want to have some kind of figure on error rate on iNaturalist?
I just started on iNaturalist. I did some more difficult species and some easy species. In the easy species it's most of the time strange ID's, I mean I can't imagine people really make wrong ID's... (although... ;-) ). Probably caused by some misunderstanding, more than one photos with more species, photos that were later added (after someone gives an ID, the observer can add extra photos which might be something completely different from the species already Identified by the reviewers...) or just errorclicks.
But when identification gets a little more difficult (Song Thrush and Mistle Thrush is a good example because both species occur in the same area, look somewhat similar and are both common) errorrate increases and is really based on lack of knowledge!
An other good example is Lesser White-fronted goose, where most observations were either wrong identified, captive birds (and not tagged as such) or where the same photos had been used more than once...
Op donderdag 9 maart 2017 22:06:17 UTC+1 schreef Scott Loarie:
But I assume you want to have some kind of figure on error rate on iNaturalist?
I just started on iNaturalist. I did some more difficult species and some easy species. In the easy species it's most of the time strange ID's, I mean I can't imagine people really make wrong ID's... (although... ;-) ). Probably caused by some misunderstanding, more than one photos with more species, photos that were later added (after someone gives an ID, the observer can add extra photos which might be something completely different from the species already Identified by the reviewers...) or just errorclicks.
But when identification gets a little more difficult (Song Thrush and Mistle Thrush is a good example because both species occur in the same area, look somewhat similar and are both common) errorrate increases and is really based on lack of knowledge!
An other good example is Lesser White-fronted goose, where most observations were either wrong identified, captive birds (and not tagged as such) or where the same photos had been used more than once...
Op donderdag 9 maart 2017 22:06:17 UTC+1 schreef Scott Loarie:
Scott Loarie
Mar 18, 2017, 12:33:17 AM3/18/17
to inatu...@googlegroups.com
Hi folks,
We were inspired by Wouter's thoughts on data quality and Jakob's suggestion of trying to get iNat observations ID'd by a bunch of independent specialists, and then use this sample to assess the accuracy of 'the crowd'.
To that effect, we're interested in launching an iNaturalist Identification Quality Experiment where we will try to recruit taxonomic experts to register for a study and provide IDs using a new 'blind' version of the Identify tool (where the expert won't be influenced by the opinions of the iNat community). The goal of this study will be to write a paper (alla http://www.nature.com/articles/srep33634) and also to develop better algorithms for assessing the quality of iNaturalist identifications.
2) Imagine a monarch butterfly flew by and I was able to observe that it was definitely a monarch but I was only able to get a blurry photo that can't be used to rule out a Viceroy. Because I had information outside of the photo to suggest it was a monarch, I might upload the observation with an ID of monarch. The current etiquette on iNat is not to provide coarser IDs unless you really do disagree with finer ID (e.g. 'I can't tell what species it is but I know its not a monarch'). However, an expert participating in this experiment might add a coarser ID at the family level, not because they disagree with my ID of monarch (which they can't see) but just because they can't make a species level ID based on the photo evidence they were presented with. This will likely only happen rarely, but if any of your observation gets coarsened beyond your liking you can 'reject' the community ID. This may change the quality grade of the observation but will still allow you to retain control of the taxon associated with the observation.
We were inspired by Wouter's thoughts on data quality and Jakob's suggestion of trying to get iNat observations ID'd by a bunch of independent specialists, and then use this sample to assess the accuracy of 'the crowd'.
To that effect, we're interested in launching an iNaturalist Identification Quality Experiment where we will try to recruit taxonomic experts to register for a study and provide IDs using a new 'blind' version of the Identify tool (where the expert won't be influenced by the opinions of the iNat community). The goal of this study will be to write a paper (alla http://www.nature.com/articles/srep33634) and also to develop better algorithms for assessing the quality of iNaturalist identifications.
While we could do this experiment on a separate snapshot of the iNat database, its too tempting not to (a) share these expert IDs with the iNat community and (b) hopefully recruit some of these experts into the community. So we're proposing to do this experiment on the actual live iNat platform.
We've added functionality to track the state of observation at the time of expert review to address Jakob and Matt's legit concerns about the community updating their IDs after being influenced by the experts opinion. This will allow us to compare the expert ID to the state of the observation before the expert chimed in (as if the study were done on a separate snapshot in time).
However, we did want to talk through two potential side-effects of this study on the community and ways to remedy them:
1) Since the expert won't be able to see which observations 'Need IDs' vs those that don't, this could generate a flurry of what we call 'confirming IDs' (e.g. a 10th person chimes in on your Research Grade monarch butterfly observation also agreeing its a monarch). If you are a power user subscribed to a lot of observations and start getting overwhelmed with updates about confirming IDs, go to your 'Account' and under 'Activity settings' uncheck 'Notify me about confirming identifications'
2) Imagine a monarch butterfly flew by and I was able to observe that it was definitely a monarch but I was only able to get a blurry photo that can't be used to rule out a Viceroy. Because I had information outside of the photo to suggest it was a monarch, I might upload the observation with an ID of monarch. The current etiquette on iNat is not to provide coarser IDs unless you really do disagree with finer ID (e.g. 'I can't tell what species it is but I know its not a monarch'). However, an expert participating in this experiment might add a coarser ID at the family level, not because they disagree with my ID of monarch (which they can't see) but just because they can't make a species level ID based on the photo evidence they were presented with. This will likely only happen rarely, but if any of your observation gets coarsened beyond your liking you can 'reject' the community ID. This may change the quality grade of the observation but will still allow you to retain control of the taxon associated with the observation.
Please let us know if you have any thoughts on the study design. We're still waiting to make sure we don't have to jump through any 'human research' hurdles with this research. Hopefully we can launch this study soon. Once we launch, we could definitely use your help with a campaign to recruit 'professional taxonomic experts' from museums, wildlife management agencies, universities, etc. Many iNat users already fit this description, but you'll need to register for the study in order for us to be able to track your data.
Looking forward to any feedback or if you'd like to help either by helping spread the word about this study or by participating if you happen to be a 'taxonomic expert'
Thanks,
Scott
--
You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist+unsubscribe@googlegroups.com.
To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.
Charlie Hohn
Mar 18, 2017, 9:26:27 AM3/18/17
to iNaturalist
Neat! I'm in, if you think i qualify as an expert. But I am pretty sure that i will subconsciously at least be super hyper careful with ID so I think there will be some bias. It's not that I am not careful at other times but you know how it is with tests. I'll try not to modify my behavior.
On Thursday, March 9, 2017 at 4:06:17 PM UTC-5, Scott Loarie wrote:
Sam Kieschnick
Mar 20, 2017, 10:35:09 AM3/20/17
to iNaturalist
Really nice study -- looking forward to seeing what sort of taxonomic experts will join. I assume that the taxonomic expert will only deal with the taxon of their expertise, yeah?
I predict one of the major findings of this experiment will be that certain taxa can't be identified with as much confidence as others with just a photo-voucher. Also, some observers are quite adept at getting the important angles/shots for identification -- others aren't quite as practiced. This is the value of the "comment" section on each observation -- just as valuable as the "identification" section.
~Sam
I predict one of the major findings of this experiment will be that certain taxa can't be identified with as much confidence as others with just a photo-voucher. Also, some observers are quite adept at getting the important angles/shots for identification -- others aren't quite as practiced. This is the value of the "comment" section on each observation -- just as valuable as the "identification" section.
~Sam
lelliott
Mar 29, 2017, 7:02:37 PM3/29/17
to iNaturalist
I would love an analysis of how many observations on which I've offered an incorrect ID (from stupidity or misclick [also stupidity]) relative to non-contested IDs. This gets tricky relative to how many IDs have already been offered. Am I agreeing with Greg Lasley or is it a novel ID? Is it an ID that contradicts a previous ID or new one. Etc. I've got no idea how to go about querying for this kind of stuff.
wouter teunissen
Apr 4, 2017, 5:44:47 AM4/4/17
to iNaturalist
And be aware that not every expert is suited for the job!
There's the story of botanist who were experts on certain plant taxa but who could not find these taxa in the wild, simply because they were only used to work with herbarium material. Only when they dried the collected material and squashed it flat, they recognised what they had seen in the field.
The same applies for photo reviewing! Not everybody can do this. This has to do with several things like only used to work with dichotomous keys, only work with alcohol material (which dilutes the vibrant colours on which some sibling species are easily recognisable in the field), etc etc.
You may use my reviews for this study. I recently did around 40,000 (but must admit I started with most of the more easy species).
Op donderdag 30 maart 2017 01:02:37 UTC+2 schreef lelliott:
There's the story of botanist who were experts on certain plant taxa but who could not find these taxa in the wild, simply because they were only used to work with herbarium material. Only when they dried the collected material and squashed it flat, they recognised what they had seen in the field.
The same applies for photo reviewing! Not everybody can do this. This has to do with several things like only used to work with dichotomous keys, only work with alcohol material (which dilutes the vibrant colours on which some sibling species are easily recognisable in the field), etc etc.
You may use my reviews for this study. I recently did around 40,000 (but must admit I started with most of the more easy species).
Op donderdag 30 maart 2017 01:02:37 UTC+2 schreef lelliott:
Scott Loarie
Apr 5, 2017, 8:05:57 PM4/5/17
to inatu...@googlegroups.com
Hi folks -
Good point Wouter about material experts are familiar with. Hopefully this study will self select experts comfertable working with photos.
I've posted a little update on the study here:
If anyone else wants to contribute please register (step A here):
We need to be able to link accounts to external expertise (which the form does) and participants to use the blind ID interface to control for biases
Thanks!
Scott
--
You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist+unsubscribe@googlegroups.com.
To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages