data quality assessment
Ken-ichi
So we just released a new section on the observation pages that
summarizes the "data quality" of the observation and introduces the
concept of data quality grades. Basically all observations are
"casual" grade by default, but if it meets a number of requirements,
we consider it "research" grade. To be research grade, an obs must
* have an ID agreed upon by the community
* have a date
* have coordinates
* have a photo
* be wild/naturalized
* have a reasonable-looking location
Some of those require explanation. For an ID to be "agreed upon by
the community," the majority of identifications must match the
observer's ID or be of a taxon that descends from the observer's
opinion. So if I add an observation of Rodentia and biosam identifies
it as Rattus rattus, that's an agreement.
Whether or not the obs is wild or in a reasonable location is up to
the community: everyone (including the observer) can vote yes or no on
these questions. If the majority thinks it's *not* wild, then it
won't reach research grade. These two conditions can be met with 0
votes, so they're really mostly there to let people demarcate stuff
from gardens or stuff with locations that are clearly errors or very
rough estimations.
Ok, that's the rather long summary. Now, some questions for you all:
1) in this system, a more conservative ID than the observer's will
count as a *disagreement*. So if biosam ID's my rodent as Mammalia,
that's a disagreement. But if I add an obs without an ID and biosam
ID's it as Mammalia and I later ID it as Rattus rattus, biosam's ID
will still count as a disagreement, even though he was probably just
trying to help me get to the right ID. He can still be outvoted by
others, or change his ID, but I feel like this is still uncomfortable.
What do you guys think? Most of the solutions I've thought of are
unsatisfactory for various reasons.
2) Do you guys want to be notified when others vote on whether their
obs are wild or if the location looks good? Right now there's no
notification, but I could make it send an email or generate an update
on the dashboard or something.
3) I considered adding the ability to opt out of this entirely, but
eventually decided it wasn't necessary. Was I wrong?
4) any bugs, typos, confusing language?
Ok, congrats if you made it this far.
-ken-ichi
barbarab
through recent observations and noticed a problem. As an example,
Wabbytwax posted a photo of a hornet on July 10 and wingedwolfpsion
identified it as a Hymenoptera. Later I identified it as in the family
Vespidae and in a comment said I thought it was a Giant Hornet, Vespa
crabro. Wabbytwax then decided to identify it as Vespa crabro. If you
go back and look at this observation, you'll see that the data quality
assessment states that tow people disagree. ????? I see someone
posting a photo and asking for help with ID. As the help came in,
folks cautiously circled in on the identification, placing it first in
the correct order, then in the correct family, and then suggest a
species that it might be and letting the original observer make the
final call.
Scott Loarie
One think I think might be nice would be for anyone who's commented on an obs to recieve emails when the conversation around the obs continues. As it is, I often comment on an obs, the owner gets an email and respond, but I am not alerted to her response with an email so the conversation stops there. For example, if the whole community involved in this Vespa crabro conversation barbarab amymhv wabbytwax and wingedwolfpsion were all alerted as the conversation progresses it might help everyone come to a consensus. But the trade off is will all the emails this would create turn people off?
--
You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To post to this group, send email to inatu...@googlegroups.com.
To unsubscribe from this group, send email to inaturalist...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/inaturalist?hl=en.
--
--------------------------------------------------
Scott R. Loarie
Post Doctoral Fellow
Department of Global Ecology
Carnegie Institution for Science
260 Panama Street
Stanford, CA 94305 USA
www.stanford.edu/~loarie
Email: loa...@stanford.edu
Phone: 415-278-1220
--------------------------------------------------
mjmuir
quality is unquestionably a good thing. The conflicted part comes
from concerns over (a) barriers to iNat entry & sustained use, and (b)
the terminology of casual/research.
re: (a). I want more people to join and use iNat. I think that the
most important new users are from outside the science & conservation
community, because this builds the constituency who pay attention to
nature. If this new addition creates a barrier for entry or sustained
use by new users (for example, by being turned off by "another thing
to fill out," or being ghettoized as someone who only makes "casual"
observations), I think it's a problem.
re: (b). The terms "casual/research" don't feel accurate. People do
research on the presence of domesticated animals (e.g., feral cats in
wildlife reserves, impact of cattle grazing on grassland communities),
and cultivated plants (e.g., invasive species candidates). Under this
system, observations of these species, even if used in research, would
come out as "casual." It also depends somewhat on the critical mass
of iNat users in the area where the observation occurred. Ironically,
some of my observations that I think of as my most casual--i.e. my
California observations of flora & fauna, none of which I took a GPS
point for--are now classified as "research" grade, because Bob Dodge,
Eric et al. have been kind enough to add or confirm an ID. In
comparison, almost all the observations that were documented during my
PhD in Botswana in an area with very little ecological data available
(and almost all of them with GPS point) are classified as "casual"
grade, because there are few iNat users in Southern Africa. More
accurate terminology might be "community supported" vs "community
unsupported." (And remove the wild/naturalized bit and deal with
cultivars and domesticated animals some other way.)
Thanks, Matt
barbarab
own observations. I really think the observer/reporter should be the
first to assess the quality of their reporting and identification.
Folks looking at my observations can't tell which ones I made shooting
from the hip and which ones were made after careful deliberation, but
I know. At times I've taken a specimen, put it under the microscope
and carefully keyed it out to my satisfaction, and then asked one or
more people I trust to check it out to make sure I've gotten it right.
Maybe these careful observations are the only kind I should be
posting. But I guess you have to decide what the purpose of
inaturalist is. I agree with Matt Muir that this site is valuable for
novices trying to learn more about the common plants and animals in
their yard or neighborhood as well as for those collecting data.
It has been interesting to notice that I more often get inaturalist
users clicking agree on identifications I am not really confident I
got right.
Barbara
Winged Wolf
If your system records the DATE (or even just the order) that
suggestions and agreements are made, then you could have it select
agreement or disagreement by date. For example, since my ID of
Hymenoptera came before the the ID of Vespa, it counts as an
agreement, since it's merely narrowing it down. If it had come AFTER
the ID of Vespa, it would count as a disagreement, since that would
imply I don't think it's Vespa, but do think it's still in
Hymenoptera.
--
Winged Wolf
My Stores:
http://www.EclipseMetaphysical.com
EclipseMeta on Twitter
http://www.EclipseExotics.com
EclipseExotics on Twitter
Ken-ichi
Matt:
a) I agree "more things to fill out" can be a turnoff for newbies,
though none of this quality stuff *needs* to be filled out. I
actually chose the term "casual" to try and emphasize that it's not
bad, just normal. Frankly, it's quite hard to gauge the impact of
these things on new users without polling them or running a/b tests on
them (people generally don't reply to surveys and we still don't have
enough people on the site for significant a/b tests). If anyone
reading this just signed up and has some input, please chime in!
b) "community supported" and "community unsupported" are ok, but I
really want to avoid negative labels, for exactly the reason you
pointed out above: As far as the word "research" goes, most of the
researchers who've expressed an interest in iNat data have been
interested in it for distribution modeling and basic monitoring of
wild/naturalized organisms, so I consider things like cattle or pets
to be edge cases (cases that we can pretty easily deal with if people
want the data by selecting only obs of cattle or cats or what have
you).
The fact that you're "from-the-hip" CA observations are considered
research-grade while your obs conducted during actual research are
considered casual-grade is ironic, but I think it's ok given the
social notion of data quality we're trying to embrace: we have a lot
of CA naturalists who can verify observations from CA, but less people
who know Africa.
barbarb: we've considered adding a way to gauge the confidence of your
IDs, but I'm not sure it addresses the issue of suggestions ("looks
like Vespidae") vs. corrections ("it's not Vespa cabro, Vespidae is
the better ID"), because in both cases, the person is probably 100%
confident of their conservative ID.
wingedwolfpsion: yup, definitely a possibility, but there's this
problem: I add an unknown, biosam ID's it as Rodentia, and I add an ID
of Rattus rattus. In the system you proposed, biosam's ID is *not* a
disagreement. However, what if biosam thinks I'm wrong? He can
remove and re-add his ID to make it a disagreement, but that seems a
bit silly to me.
What do people think of a "correction" checkbox on the IDs to indicate
that a more conservative ID is intended as a disagreement rather than
a tip in the right direction? To much additional complexity?
-ken-ichi
Winged Wolf
> wingedwolfpsion: yup, definitely a possibility, but there's this
> problem: I add an unknown, biosam ID's it as Rodentia, and I add an ID
> of Rattus rattus. In the system you proposed, biosam's ID is *not* a
> disagreement. However, what if biosam thinks I'm wrong? He can
> remove and re-add his ID to make it a disagreement, but that seems a
> bit silly to me.
Not really--editing an ID is extremely fast and easy, after all. I
don't think people would find it too cumbersome.
> What do people think of a "correction" checkbox on the IDs to indicate
> that a more conservative ID is intended as a disagreement rather than
> a tip in the right direction? To much additional complexity?
>
> -ken-ichi
That would work, too.
Charlie Hohn
similar programs/apps was that people would map things in their
gardens, etc, and it would be inseparable in the data from wild or
naturalized occurrences. While there are lots of reasons to record
what does well in your garden or flowerbed, there needs to be a way to
separate that from things happening in nature (native plants, invasive
plants, range expansions, etc)... I made a suggestion to this effect a
day or two ago, and wondered if I had just missed this feature, but it
looks like it was just added... so I guess others had a similar idea
before I did!
-Charlie
slowwatermovement.blogspot.com
On Jul 15, 4:13 pm, Winged Wolf <wingedwolfps...@gmail.com> wrote:
george jackson
Well, that's it.
mjmuir
uploaded the observation, like what Barbara suggested. I would love a
sliding-bar feature of confidence associated with each ID. But even
with developing new features of confidence or community voting on the
ID, the taxonomic ID is less important in determining a truly research-
grade observation than diagnostic features visible in the photo, and
an accurate location. What's potentially misleading about the new
tool is the "majority rules" agreement on ID. Two yahoos voting on
the internet (of which I consider myself one!) does not a research-
grade observation make. It doesn't take into account sustained
monitoring like George's fox observations, or the deliberation that
Barbara spends on a specimen under a microscope, or the many
confidently IDed observations that slide by without comments or IDs
from other iNat users for all sorts of reasons.
The location aspect is the most befuddling to me. I don't know how
you deal with that. If I came to you as a researcher with a project
on "distribution modeling and basic monitoring of wild/naturalized
organisms," the quality of the location data would be the most
important thing to me (the IDs I would double-check myself). But as a
user, I know that I have uploaded observations in California that some
of iNat's expert naturalists have IDed, and voted that the "location
seems accurate," that I know do not have accurate location coordinates
for. I indicate my lack of precision in the description ("Location
approximate"), but that doesn't roll up. Does "location seems
accurate," mean accurate to the uploaded coordinates, accurate to <10
km radius, accurate to the county, accurate to the visible habitat
type? Because the scale of accuracy is open to interpretation (and
misjudgment), agreement on the location seems like a crude descriptor
of data quality. Again, better to give more control to the person who
uploaded the observation, and allow them to indicate confidence/
precision of location in a manner that can roll up.
I think whatever method gets chosen to distinguish quality, keep it
simple and elegant. Try to keep as much as you can behind the
curtain, or at least in a separate 'advanced' tab for observations.
Ken-ichi, can you explain a bit more about the "social notion of data
quality" that iNat is trying to embrace? Maybe that's where I'm not
getting it.
Thanks, Matt
> >> 'uncomfertable' but not totally clear what a better solution would be. I
> >> just agreed with Vespa crabro based on the Bugguide account
> >> (wingedwolfpsion and barbarabs). One more vote for Vespa crabro and the obs
> >> would go 'research grade'. As Ken-ichi said these coarser IDs (Hymenoptera
> >> and Vespidae) aren't necessisarily disagreements. In this case barbarab and
> >> wingedwolfpsion were helpfully refining the ID down to finer taxonomic
> >> levels. However, you could imagine that they were saying 'no, Vespa crabro
> >> is wrong, I'm not sure what species/genus is right but I think this ID
> >> should be more conservatively backed up to Vespidae or Hymenoptera for the
> >> time being'. Its hard to find a way to distinguish these intentions. What
> >> Ken-ichi is suggesting is that one way to make Wabbytwax's obs go research
> >> grade would be for other community members to give the Vespa crabro ID
> >> critical mass - it already has 2 votes and needs one more to be the majority
> >> - likewise, barbarab and/or wingedwofpsion could retroactively refine their
> >> IDs from Hymenoptera/Vespidae to Vespa crabro. Another way would be to alter
> >> the functionality in some clever way that maybe someone on the iNat page can
> >> help suggest.
>
> >> One think I think might be nice would be for anyone who's commented on an
> >> obs to recieve emails when the conversation around the obs continues. As it
> >> is, I often comment on an obs, the owner gets an email and respond, but I am
> >> not alerted to her response with an email so the conversation stops there.
> >> For example, if the whole community involved in this Vespa crabro
> >> conversation barbarab amymhv wabbytwax and wingedwolfpsion were all alerted
> >> as the conversation progresses it might help everyone come to a consensus.
> >> But the trade off is will all the emails this would create turn people off?
>
>
> read more »
Sam
tendency to agree on an ID without really knowing the true ID. On
occasion I've found myself assuming that because 2 or 3 people agree
on a species, and I presume them to be knowledgeable, that I'll throw
in my agreement, even if I'm not familiar with the organism. If other
people do this enough, it could lead to "flocking" around incorrect
ID's. I've even found myself suggesting species ID's when someone
posts a "something" with scientific name in the description, although
I may not know for sure (or at all) that that is what it is. If
someone thought that I was an expert, then agreed with me, it could
take a while until the correct ID was discovered by a real expert, but
then it would look like a disputed ID until everyone changed their
decision.
As Barbara said: "It has been interesting to notice that I more often
really confident I got right. "
you have reasonable, independent confidence in the ID, rather than
what others suggest. Maybe this is what most people do already.
-Sam
Nick Waters
Nick
FlaPack
identifications that we've made for others in light of later
refinements, it would be nice to have an easy way of knowing when
there is activity on an observation we've IDed. Email notifications
is a possibility but I'm sure I'm not the only one that gets numerous
notifications from multiple websites and sort of tunes them out when
the inbox is filling up. I've been thinking about this and was
thinking that for those who visit the site pretty regularly it would
be convenient if our dashboard was in a sense "subscribed" to anything
we've commented or IDed. Those "subscribed" items that have seen
subsequent action (further IDs or comments from others) in the last X
days would show up on the dashboard like the "recent comments"
currently do. They could even appear as collapsed threaded
discussions that could be expanded within the dashboard to quickly see
what IDs or comments have been left. Then you would easily know when
you might want to reconsider an ID based on further refinements by
others. Just an idea. I do really love all the upgrades to the sites
you guys have worked to implement over the years. Thanks.
Nick D Waters
Perhaps an expert panel on certain taxa which would vouch for others?
Nick
Ken-ichi
I guess was poorly-chosen. The intent was to demarcate observations
that we have more reason to believe are accurate than others. Clearly
Matt's African observations and George's foxes are potentially worthy
of a research data set, but this is the Internet: Matt and George
could be anyone. Without knowing them through continued use on the
site, there's no reason to trust their taxonomic expertise, or how
fastidiously they are about location. As Matt pointed out, "two
yahoos" making IDs do not necessarily make quality data, but I think
the more people that add IDs the more chance you'll get some
Houyhnhnmian input. And frankly, I think two earnest yahoos trying to
make an ID have a better chance of being right than one yahoo working
alone.
This is what I meant by a social conception of data quality: if sites
like Wikipedia have shown us anything, it's that large groups of
amateurs are often able to create more accurate and/or more complete
datasets than individuals or even small groups of experts.
Anyway, maybe we should just replace the word "research" with
"community" or something, so people don't feel like "casual"
observations are somehow not worthy of research.
Location accuracy is definitely serious and difficult issue. You
might have noticed that the iPhone app actually records positional
accuracy (which is often terrible, another issue). We can start
recording whether people entered coordinates manually or whether they
entered the name of a place and let Google look up the numbers. We
could also let people start manually editing the positional accuracy,
but we'd also have to start recording whether they did so. Even if we
did these things, though, I'm not sure any of it is really better than
the system we have now. They would basically allow observers to
voluntarily mark their locations as inaccurate (which they can already
do), but they would do nothing prevent people from exaggerating their
accuracy. We can't stop people from lying, so I think providing tools
for the community to call out obviously inaccurate stuff is the best
we can do. It might be comforting for consumers of the data to see an
accuracy number, but just because it's there doesn't really inspire
any more trust in the observer if you know they could have fudged it.
The flocking issue and the fact that unidentified stuff seems to get
more IDs than ID Please observations is super interesting, and
something that we can probably address with changes to the interface,
maybe doing a better job of visually indicating that an obs needs ID
help, as barbarb suggested.
flapack, we definitely want to start subscriptions for things you've
commented on / ID'd. That's a pretty conventional model from other
social networking sites, just haven't gotten there yet. Overhaul of
the update emails is long overdue, too.
Nick, you can add IDs of any taxon, so if you're only confident
calling something a bird, you can ID it as Aves. We might some day
start designating experts in certain places / taxa based on their ID
performance (some function of matching the observer's ID and time?),
but I'd say that's a ways off yet.
Whew! This is a really cool discussion. It's so great getting all
this feedback!
-ken-ichi
Chris Fullmer
Ken-ichi
with an ID you're unsure of. I realize there are multiple reasons you
might want an ID, but ID Please will be useless for people who
genuinely want help if other people flood it just to get their
observations to research grade. There's probably a better way to
manage these different meanings for an "identification," but until we
come up with something let's try and keep ID Please for what it was
intended.
-ken-ichi
> --
> You received this message because you are subscribed to the Google Groups
> "iNaturalist" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/inaturalist/-/R0P3SFzXqIIJ.
