Identification Tool Suggestions

[SummitMetroParks-NaturalResources]

First, I absolutely love the ability to use the new Identification Suggestion Tool in the iOS App!!!  I am eagerly looking forward to more universal iNaturalist application.

However, I really think there should be some changes in how it is used or handled.  I have been noticing the craziest species being posted to our projects and it just dawned on me that it is because the App is performing the ID wrong.  I have also noticed that I am now an expert in all sorts of taxa groups that I know nothing about.  The real reason that these examples are occurring is that direct human identification is not happening in many cases.  Perhaps we should think of a way to distinguish a largely computer generated ID from an personal or expert ID?  I also think that both this new ID tool as well as identifiers like myself should be able to assign confidence levels to our IDs.

So, in thinking about these issues, I would like to propose that every record begins with an automated suggestion or series of ranked suggestions from the new ID tool that will be understood by all to be iNaturalst's AI guess.  It can treated as a separate ID and be used to help the observer and every real person thereafter.  A setup such as this would help clarify what individuals are doing vs computer generated AI.

[Scott Loarie]

Thanks for the feedback - we definitely share your concerns and are
monitoring how new suggestions tool is monitoring behavior on the site
so we can tweak things as necessary. A few things:

1) We're tracking every ID that was made using the suggestions tool.
But this does get complicated, because using the ID tool doesn't
necessarily mean the user didn't also know the ID. I personally find
it useful as a kind of visual autocomplete that saves me time not
having to type in names for species I know.

2) We are currently presenting them as 'suggestions' rather than
automatically suggesting an ID. Some quick stats: Based on a test
sample of 50k photos representative of everything thats been posted to
iNat (so that includes some common things, but also some very rare
things) the suggestions tool places the correct species is in the top
10 results 78% of the time, in the top 2 results 66% of the time, and
in the top spot 57% of the time.

But we also tried to encourage people not to go straight to the top
suggestion by adding in the coarser ranked 'recommendation' at the
top. We are able to produce a common ancestor 77% of the time
(otherwise it says we're not confident enough to recommend something).
If it does produce a recommendation, its right 93% of the time. A key
goal is to bring all these stats up, but I'd like in particular the
recommendation accuracy to be higher since thats the choice we're
actually encouraging you to tap on

3) In parallel to this computer vision work, we're also conducting an
analysis of the quality of the identifications on iNat you can read
more and help here:
https://www.inaturalist.org/pages/identification_quality_experiment
We're a little behind on this (having gotten bogged down in computer
vision) but have more time for it now. The main goals of the study are
to see if we can assign a quantitative accuracy threshold to 'research
grade' - right now it means >2/3 people agree. We're working on a
model that incorporates more data about an observation such as each
IDer's past behavior (earned reputation) to come up with a
quantitative predicted estimate that an observation is accurately
ID'd. Once we have this we'll be able to adjust Research Grade so that
its some threshold (ie obs with >99% predicted accuracy).

I think this would solve issues associated with trusting the AI
because we'd be able to be explicit about some level of accuracy
associated with an observation ID (whether it comes from a completely
anonymous user using the suggestions tool, or some mix of that and the
opinions of community) so folks could judge for themselves what level
of accuracy they're comfortable with. Agreeing on a threshold for
Research Grade might be tricky though - should it include errors of 1
in 100, 1 in 1000 etc?

Scott

> --
> You received this message because you are subscribed to the Google Groups
> "iNaturalist" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to inaturalist...@googlegroups.com.
> To post to this group, send email to inatu...@googlegroups.com.
> Visit this group at https://groups.google.com/group/inaturalist.
> For more options, visit https://groups.google.com/d/optout.



--
--------------------------------------------------
Scott R. Loarie, Ph.D.
Co-director, iNaturalist.org
California Academy of Sciences
55 Music Concourse Dr
San Francisco, CA 94118
--------------------------------------------------

[SummitMetroParks-NaturalResources]

This sounds great.  Thank you for everything you are doing!

[Ken-ichi]

I'm curious, what the the URLs of some observations of "the craziest
species" that you think were identified using the new vision
suggestions in the iOS app? I've been noticing them in the areas I
monitor, but so far I haven't seen anything off the walls. Mostly just
conservative but accurate genus-level IDs.

On Thu, Jul 13, 2017 at 10:44 AM, SummitMetroParks-NaturalResources
<nrmsummit...@gmail.com> wrote:

[SummitMetroParks-NaturalResources]

http://www.inaturalist.org/observations/7014359 and http://www.inaturalist.org/observations/7014432

The second is not terrible, but it is not typical of human error either.

[SummitMetroParks-NaturalResources]

http://www.inaturalist.org/observations/7044404

[Charlie Hohn]

i am loving the ID algorithm. However I think one tweak would be to weight 'occurs nearby' much more heavily. I had a suggestion of Calandrinia menziezii for a Vermont plant the other day that got a laugh. No way that could ever be here. I propose for now we disallow it from proposing anything not observed within 100 miles or something, and eventually maybe build rough ecoregions on the California Floristic Provence type level, and restrict it to ecoregion. I realize in the short term it might limit how many species the app identifies, but it would help a lot with the weird way out of range species show up.

Another unrelated proposal: being able to withdraw some of my own observations for consideration, at least until we get a tagging or clipping type feature. For instance if I have a photo with six different plant species, or a photo with two bees on a flower, it is just going to confuse the algorithm. I'd like to either be able to tag the species i am looking at, like with facebook, or else just check a box that says 'not suitable for algorithm' or something. I think it would help!

But seriously, this thing is amazing. I was super skeptical because i've seen other things that attempt the same and don't work at all. This one works really really well for what it is and is only going to get better. Thanks for creating it and continuing work on it.

On Friday, July 14, 2017 at 7:27:18 AM UTC-4, SummitMetroParks-NaturalResources wrote:

http://www.inaturalist.org/observations/7044404