Hoping artificial intelligence IDs will get better.

[cray man]

I'm amazed at how well AI works on plants.  At the same time, I'm exasperated on how bad it is at IDing cicadas.  I just looked at an observation that was a super obvious Resh cicada, but AI's first choice was a Swamp cicada which is totally obviously wrong.  Resh was listed something like 7th choice.  Resh has 435 research grade observations.  Swamp cicada has 293.  I'm hoping AI will eventually get better.

[Charlie Hohn]

Add more cicadas to train it! Though I imagine just in general some taxa are easier for an algorithm than others. 

On Sun, Aug 12, 2018 at 8:00 AM cray man <danjoh...@gmail.com> wrote:

I'm amazed at how well AI works on plants.  At the same time, I'm exasperated on how bad it is at IDing cicadas.  I just looked at an observation that was a super obvious Resh cicada, but AI's first choice was a Swamp cicada which is totally obviously wrong.  Resh was listed something like 7th choice.  Resh has 435 research grade observations.  Swamp cicada has 293.  I'm hoping AI will eventually get better.

--
You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.
To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

--

Sent from Gmail Mobile

[David K]

It really depends on the taxa.  I had the opposite experience to yours today:  I uploaded 20 observations of mostly lepidoptera, a few plants and some herps, and it suggested correct answers in the first slot for 18 or 19 of those.  No doubt these are relatively common species and there is a large library to learn from.  And while I have also seen some wonky results (later today the AI offered something not found on my continent), overall, I am very impressed with how well it performs.  For many lepidoptera, it will produce correct answers from dorsal or ventral shots of adults, and many larval stages.

[jesse rorabaugh]

It isn't much good on arthropods in my experience. There is just too much diversity and too few photos to train on. Maybe another couple years and it will work well but I don't yet count on it,

It is getting almost creepy how good it is with plants in California though. It can't get everything but manages to get some very obscure species correct.

[Susan Hewitt]

The AI is currently really terrible on marine mollusks outside of California. It tries to ID everything in the world based on the most similar California species. Every cockle in the world is Trachycardium nuttalli, a California species. And so on. It also tries to put a few New Zealand species names  on some mollusks worldwide.

It needs to learn that there are marine faunal zones, and anything that is geographically in another faunal zone should be ID'ed to the family level for the time being, not to the species level.

I spend quite a bit of time correcting this kind of error.

[Susan Hewitt]

A big problem is that relative newcomers to iNat, when they are offered the AI suggestions, they assume that all the competence of the platform and all its users is behind these suggestions! Relatively new users have no idea that the suggestions are coming from an AI, and they also have no idea what the limitations are of an AI in training.

I wish there was some way of getting this across to people.

[Susan Hewitt]

Obviously the AI will get better over time, even on mollusks. But that process of improvement depends on there being more expert humans on here creating accurate mollusk identifications that can be used to train the AI. 

And currently on iNaturalist there just aren't enough really active malacologists, amateur or professional. In fact there are only about 400 malacologists in existence worldwide and goodness knows how many of them we can recruit to help out here. The question is, what can we do in the meantime, to soften the impact of so many wrong mollusk IDs being offered and embraced by new users ?

Susan

[bouteloua]

Previously on the iNat Google Group...

I would also like to see it be much more tentative in the groups where it is frequently incorrect. In crayman's example, if the AI had only offered tribe or family instead of species, his ID could then roll the ball forward to species rather than count as a disagreement and potentially require 2 more IDs to fix it. As Susan mentions, skilled identifiers are very limited for some taxa.

Here's another example: it seems to suggest Chrysoperla carnea for most green lacewings photos. This is a species that apparently cannot be visually distinguished. The issue of numerous misidentifications of this species on iNaturalist was brought to attention over a year and a half ago via BugGuide. I and others have coarsified hundreds (thousands?) of these observations since then. But still the AI often suggests the species directly, frequently not even genus, tribe, or family first instead. This is probably not an ideal way to spend one's time (one-by-one coarsifying) when the problem could be greatly reduced by just excluding this species from the AI. See also some moss examples. 

So:

• I'm not certain, but it seems if the AI suggestions don't at least share a common family, no higher level taxon is suggested and the top result is a species. This is a problem in these frequently-incorrect situations. If that's the case, can you instruct the AI to only suggest something like order, phylum, or even kingdom if the top few results don't even share a family? examples: https://www.inaturalist.org/observations/15333958, https://www.inaturalist.org/observations/15321724, https://www.inaturalist.org/observations/15312134 Maybe these should say "we're pretty sure this is in the class Insecta" instead of suggesting Chrysoperla carnea first
• Can certain species be removed from the AI completely? (inclusion could be reconsidered at a later date)
• In Susan's example, eliminate certain species based on range? (yes ranges are flexible, this would just be a select group of frequently-incorrect taxa that are unlikely to suddenly pop up halfway across the world)
• Can the AI be "branded" in some way to distinguish AI suggestions from human suggestions ("Computer Vision" or something else...maybe the return of "Identotron" lol)?
• To indicate to others be a little more tentative in agreeing with IDs and therefore bad AI IDs getting to Research Grade, please do implement the "@user selected a Computer Vision suggestion" feature.
• Can you update this page or create a new page with more recent information on how the AI is trained? https://www.inaturalist.org/pages/computer_vision_demo

I would love to hear what's in store for future iterations of the AI and its integration on iNat.

thank you!

cassi

[Riviera S]

How about if the AI suggestions first (in the initial suggestions) consider the continent or country-wide options. And then, there are two slots at the bottom which include the best worldwide matches. That would avoid the "out-of-range" matches always being at the top.

I second the proposal for a "tag" or other feature that separates "AI IDs" from manually entered ones.

[nolie_s...@rogers.com]

What does “coarsified” mean? I thought it was a typo or incorrect autocorrect, but the same word occurs twice in this paragraph, in different verb parts.  Just curious. These discussions are very interesting, by the way.

 

“Here's another example: it seems to suggest Chrysoperla carnea for most green lacewings photos. This is a species that apparently cannot be visually distinguished The issue of numerous misidentifications of this species on iNaturalist was brought to attention over a year and a half ago via BugGuide. I and others have coarsified hundreds (thousands?) of these observations since then. But still the AI often suggests the species directly, frequently not even genus, tribe, or family first instead. This is probably not an ideal way to spend one's time (one-by-one coarsifying) when the problem could be greatly reduced by just excluding this species from the AI. See also some moss examples.”

 

Nolie

 

 

 

 

From: bouteloua

Sent: Monday, August 13, 2018 3:42 PM

To: iNaturalist

Subject: [inaturalist] Re: Hoping artificial intelligence IDs will get better.

 

Previously on the iNat Google Group...

 

I would also like to see it be much more tentative in the groups where it is frequently incorrect. In crayman's example, if the AI had only offered tribe or family instead of species, his ID could then roll the ball forward to species rather than count as a disagreement and potentially require 2 more IDs to fix it. As Susan mentions, skilled identifiers are very limited for some taxa.

 

Here's another example: it seems to suggest Chrysoperla carnea for most green lacewings photos. This is a species that apparently cannot be visually distinguished The issue of numerous misidentifications of this species on iNaturalist was brought to attention over a year and a half ago via BugGuide. I and others have coarsified hundreds (thousands?) of these observations since then. But still the AI often suggests the species directly, frequently not even genus, tribe, or family first instead. This is probably not an ideal way to spend one's time (one-by-one coarsifying) when the problem could be greatly reduced by just excluding this species from the AI. See also some moss examples.

 

So:

• I'm not certain, but it seems if the AI suggestions don't at least share a common family, no higher level taxon is suggested and the top result is a species. This is a problem in these frequently-incorrect situations. If that's the case, can you instruct the AI to only suggest something like order, phylum, or even kingdom if the top few results don't even share a family? examples: https://www.inaturalist.org/observations/15333958, https://www.inaturalist.org/observations/15321724, https://www.inaturalist.org/observations/15312134 Maybe these should say "we're pretty sure this is in the class Insecta" instead of suggesting Chrysoperla carnea first
• Can certain species be removed from the AI completely? (inclusion could be reconsidered at a later date)
• In Susan's example, eliminate certain species based on range? (yes ranges are flexible, this would just be a select group of frequently-incorrect taxa that are unlikely to suddenly pop up halfway across the world)
• Can the AI be "branded" in some way to distinguish AI suggestions from human suggestions ("Computer Vision" or something else...maybe the return of "Identotron" lol)?
• To indicate to others be a little more tentative in agreeing with IDs and therefore bad AI IDs getting to Research Grade, please do implement the "@user selected a Computer Vision suggestion" feature.
• Can you update this page or create a new page with more recent information on how the AI is trained? https://www.inaturalist.org/pages/computer_vision_demo

I would love to hear what's in store for future iterations of the AI and its integration on iNat

 

thank you!

cassi

[bouteloua]

I meant it as:

ID1: @user selected computer vision ID of Chrysoperla carnea (species, a fine/narrow taxonomic level)

ID2: @bouteloua suggested an ID of Chrysopini (tribe, a coarse/broad taxonomic level), @bouteloua disagrees with the ID Chrysoperla carnea

sorry for my fake jargon :)

cassi

[nolie_s...@rogers.com]

Thanks, Cassi!

--

[Susan Hewitt or Ed Subitzky]

“Coarsified” means taking the ID down to a coarser level — things like reducing a species ID to a genus or family level ID.

Susan

You received this message because you are subscribed to a topic in the Google Groups "iNaturalist" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/inaturalist/PrjmKO9YvZ0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to inaturalist...@googlegroups.com.

[Colin Purrington]

Just wanted to second these sentiments/suggestions: 

Susan Hewitt wrote:

A big problem is that relative newcomers to iNat, when they are offered the AI suggestions, they assume that all the competence of the platform and all its users is behind these suggestions! Relatively new users have no idea that the suggestions are coming from an AI, and they also have no idea what the limitations are of an AI in training.

I wish there was some way of getting this across to people.

[Susan Hewitt or Ed Subitzky]

Yes, a @user selected a Computer Vision suggestion feature would be helpful, *but* it seems to me crucial that (somewhere and somehow) we need to admit that currently, for the less popular groups of organisms in many parts of the world, the species ID suggestions coming from from Computer Vision are often quite unreliable.

The average user has no idea at all of the limitations of Computer Vision and its learning curve, so labeling a species ID "Computer Vision created" does not help at all with this problem.

I suspect that for the average person, “Computer Vision” sounds more reliable than human vision,

[Colin Purrington]

I totally agree. Average person hearing "computer vision," "AI", or anything like that will quickly assume that offered ID is more reliable than what a human expert might provide. My thinking was that such wording might be of use to those reviewing the IDs accepted by novice users. But the important thing, as I think you're saying, would be for the AI to not provide such specific IDs when AI has a known history of not providing proper IDs (though of course that will change for some taxa over time). 

[Chris Cheatle]

One question that would need to be resolved is the Computer Vision not giving a proper ID versus the person picking the wrong option from the offered list of suggstions.

When I do corrections of obviously wrong selections, I often check to see if it was selected from the CV, and if so, run the suggestion myself to see what it spits out, and in many cases, the one the person chose is down the list and not the first option (I'm assuming of course that the order of suggestions I see would not for some reason be different than what was presented to the user).

[Whitney Mattila]

I wonder if changing the sentence in the app would help. Even if it seems obvious to us that it's just a helpful tool, if I put myself in the shoes of someone new to iNaturialist, I can see where the confusion might be. Currently it says:

"We're not confident enough to make a recommendation, but here are our top 10 suggestions."

I'd possibly change it to something like:

"The AI (or whatever is the correct term for it) is not confident enough to make a suggestion. Here are ten possibilities."

Switching out 'we' makes it clear that it's not a person or authority who is making the recommendations, but a computer, while making it clear that it doesn't have a solid answer. The second change makes it less likely that someone will assume that the first option is always the correct one.

I think we should get across that the suggestions are fallible. I don't know if simply changing some words will help any, but the posts here made me wonder about this.

[Chris Cheatle]

I do think putting a marker or flag on photos that allow them to be excluded from the training set is a great idea.

I'm not sure if marking certain species as not viable candidates for the CV is a workable solution. If say, lacewings are not possible to be visually separated, people are still going to photograph and seek to get an ID. If lacewings are not in the database of options, the CV will then default to mayflies, or whatever it thinks is the most visually similar, in its vocabulary and then you will see a spate of lacewings coming in identified as mayflies, in which case the errors or issues may be even harder to find.

[cassi saari]

It could still train on the whole dataset, but the final user interface could exclude them as clickable species-level suggestions. As I said earlier, CV should only allow IDs of higher taxa (tribe, family, order, class, etc.) in groups for which it is frequently incorrect, howsoever defined. In the lacewings example, it shouldn't move on to suggest specific mayfly or tree cricket species as even worse options, it should only suggest/allow IDing as "Chrysopidae" or just "insects."

cassi

--

[Faerthen Felix]

Maybe adding a certainty factor based on the number of records used to make the ID would be helpful? For instance, if the AI identifies a common plant in California, you could say "We are 80% confident this is ...", or "10,453 observations suggest that this is ..."

[Charlie Hohn]

Would be great to see more of that kind of info though I suspect the greenest newbies might not use it 

On Wed, Aug 15, 2018 at 10:29 AM Faerthen Felix <faer...@gmail.com> wrote:

Maybe adding a certainty factor based on the number of records used to make the ID would be helpful? For instance, if the AI identifies a common plant in California, you could say "We are 80% confident this is ...", or "10,453 observations suggest that this is ..."

--

You received this message because you are subscribed to the Google Groups "iNaturalist" group.

To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.

To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

[cbgr...@hotmail.com]

Very interesting thread ! Cassi Saari/bouteloua linked me to it.

I only joined iNat recently and I thought the "research grade" vetting was going to be done only by area specialists, like it is for eBird and e-Butterfly.

I learned differently when my tentative id of one species was approved to "research grade" by a (admittedly very talented and motivated) 15-year-old with age-related-life-experience limitations on their exposure to alternative possible species. So I have no confidence that the id is correct. No one will be looking at it any longer though because there are so many observations flooding in as "unknown" that cry out to be dealt with first to avoid discouraging other new members.

I have also seen some mis-id's (such as Cow Parsnip as Giant Hogweed) which I have let go because I don't want to cause friction especially over a species which IS present in the study area even if it is NOT the one in the photo.

I made a couple of mis-ids and have quickly learned to stop even trying to make ids unless it is in my home area and I am 100% sure there are no look alikes etc.

Even then I may be contributing to the problem of mis ids.  (For example, I didn't know till today there was a fungi Abortiporus biennis as well as a Hydnellum peckii in my area. I could easily have mis-id'd someone else's before learning this.) So now I'm often just adding comments suggesting how it seems like a certain id and leaving it for others to make the actual Suggested Id.

 

I really like the auto-id software and I find it a creative and potentially wildly useful way to obtain info about which species are where but there are obviously limitations during this learning phase. The auto id has helped me start narrowing down my potential suspects especially for insects and by using in conjunction with bugguide.net and other resources it has saved me time reaching a tentative id.

I hope something creative can be developed to designate "Beta Phase" ids versus "High Certainty" ids. For example, when the Merlin Bird App was being developed by Cornell, in Beta over the internet, 1000s of us uploaded bird photos to test the software and accepted or rejected its suggested ids. Now the phone app does an amazingly good job of id'ing birds in the more populated areas of North America. In the beginning, though, it would come up with some very humourous suggestions which was part of what motivated people to play with the Beta, deliberately uploading misleading photos and correcting the output.

I don't  have a solution but I hope that the developers find one, as it would be a shame to end up collecting a lot of worthless ids based on false AI. Credibility will be very important for a data collection engine of this magnitude.

[GanCW]

The problem is usually due to lack of data. It used to be very bad at identifying common and easily identifiable Lepidoptera from South East Asia.  We made an effort to upload more observations and now it is very good at identifying most of the common species. 

What will be really useful is to have the ability to allow human to assist the AI identification by narrowing the scope of the identification. 

Specifically by allowinghuman to scope 

- Geographical distribution - by region, country, place 

- Taxonomy by family, subfamily, tribe 

[Whitney Mattila]

At the risk of getting off-topic, I think it's important to add an ID if any of us see the wrong ID being raised to research grade.

If friction is raised, that's okay, because it makes us all sit back, critically think, debate, and do more research. Even if we turn out to be wrong, I'd rather be wrong and learn from my mistakes than go on making the same mistake. :) If someone does get angry, they're not approaching the ID in good faith, which is a lesson in itself.

Also, a lot of us layman sometimes get a bit too helpful in that regard, as we'll sometimes go by experience and new-found knowledge. Roads paved with good intentions and all that. :)

[bouteloua]

Another example similar to Chrysoperla carnea, brought up by mite taxonomist Ray Fisher (@rayfisher):

Trombidium holosericeum: "This species is only known from western Europe and is not able to be differentiated from other members of that genus within its range. Yet on iNat, virtually any photo of a red mite, especially true velvet mites (Trombidiidae), end up at this species-level identification. I try to check the page regularly, but submissions add up quickly. I don't think it is justified to offer species-level identification for this genus given the state of taxonomy for the group."

https://www.inaturalist.org/taxa/58397-Trombidium-holosericeum

cassi

[edanko]

As long as the AI is only trained on species that can be ID'd to research grade, the AI will be intrinsically inclined to ignore the existence of the various other species which cannot or can only rarely be identified beyond family or genus from a photo. Is there any solution except to train the AI on non-research grade photos which nonetheless have a well-defined community ID?

[cassi saari]

Yes, analogous to the Chrysoperla carnea example/recommended fix above, there should be enough "re-identification/correction data" on iNat to train Computer Vision that *only* offering a higher taxon such as Trombidiidae would be advisable.

cassi

[Scott Loarie]

We've actually been working on this problem at the moment. We're
experimenting with training the model on the whole taxonomy (not just
species). This allows us to make use of training data sitting at
coarser nodes (e.g. if there's 1,000 obs at the genus level but very
few at the species level). And also roll up predictions to whatever
point in the tree we're 'confident enough about'. Here's an example of
a photo that is clearly some sort of Cicada species. The model is
99.95 sure its Order Hemiptera, 82% sure its a Cicada, but only 22%
sure its in the genus Cyclochila.

This prompts the question of if we want to offer a single prediction,
how certain should we be? 80%, 90%? Or can we make a cool UI where
people can navigate a hierarchy of suggestions and judge for
themselves


Predicted Node 843 (Cyclochila australasiae)
Node Animalia @ conf 1.000 (max 1.000) (rolling 1.000)
Node Arthropoda @ conf 0.999 (max 0.999) (rolling 0.999)
Node Insecta @ conf 1.000 (max 1.000) (rolling 0.999)
Node Hemiptera @ conf 1.000 (max 1.000) (rolling 0.999)
Node Cicadidae @ conf 0.821 (max 0.821) (rolling 0.820)
Node Cyclochila @ conf 0.267 (max 0.334) (rolling 0.218)
Node Cyclochila australasiae @ conf 1.000 (max 0.714) (rolling 0.218)

> --
> You received this message because you are subscribed to the Google Groups
> "iNaturalist" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to inaturalist...@googlegroups.com.
> To post to this group, send email to inatu...@googlegroups.com.
> Visit this group at https://groups.google.com/group/inaturalist.
> For more options, visit https://groups.google.com/d/optout.



--

--------------------------------------------------
Scott R. Loarie, Ph.D.
Co-director, iNaturalist.org
California Academy of Sciences
55 Music Concourse Dr
San Francisco, CA 94118
--------------------------------------------------

[paloma]

I like the one "where people can navigate a hierarchy of suggestions and judge for themselves." I think I would enjoy it for my own observations and when I try to ID others' observations, plus maybe it would cut down on people just clicking on anything the AI presents and encourage looking into the choices first.

[Charlie Hohn]

seconded on getting as much algorithm info as possible and using that to choose. 

============================
Charlie Hohn
Montpelier, Vermont

[bouteloua]

Would be fun to have a tree of suggestions that are distinguished/displayed in some way based on the model's level of confidence. Also a tooltip on all CV IDs that can show the model's confidence percentage. 

As far as a simpler cutoff otherwise, 80% seems reasonable, though I'm not sure, having not seen it in action with percents attached to the suggestions. What are some current confidence percentages for its IDs of red velvet mite observations in North America, where none *should* be IDed as Trombidium holosericeum? Since the model trained on incorrectly-research grade observations of that species in that place, the percentages might be high (?)

cassi

[paloma]

I'm not sure what a "CV" ID is--do you mean confidence levels showing on all current observations? That would be fantastic, especially if all observations became searchable by confidence level (as well as lack of confidence level, if under whatever the cutoff turns out to be).

[cassi saari]

Yeah, CV IDs --> Computer Vision IDs that have been added to observations.

cassi

--
You received this message because you are subscribed to a topic in the Google Groups "iNaturalist" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/inaturalist/PrjmKO9YvZ0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to inaturalist...@googlegroups.com.

[jdmore]

I strongly concur with making numeric ranks of AI suggestions available to users, great idea @loarie!

If it can be done in a way that makes the information and what it means clear to any casual iNat user, so much the better.  But if not, maybe it can be an option assigned to a button that a user can click while viewing AI suggestions, to "drill down" into the numeric ranks.

One thing we should always be clear about: an AI certainty of 100% does NOT mean the observation is definitely that taxon.  It only means that the AI is 100% certain.  The AI is not infallible.

--Jim Morefield

[jesse rorabaugh]

I would love all the higher nodes to be included in the AI with probabilities included. If the first five items in the list people see are the same as the first five in this list I suspect most people would pick Hemiptera or Cicadidae unless they really know.

[Les Powrie]

I agree that some indication is valuable to show how the ID was derived. Perhaps the iSpot method can be used for the first ID. It gave options like 'I'm as sure as I can be', 'It's likely to be this, but I can't be certain', 'It might be this'. Perhaps add 'iNaturalist computer vision suggestion' as default.

We find in South Africa, with more than 22 000 indigenous seed plants from almost 230 different families; 43565 insect species from 7753 genera and 569 families; 850 recorded bird species; over 350 species of reptiles; 299 mammal species; 4700 amphibian species; 400 species of indigenous fungi; 1750 species of lichen and one variety in 260 genera; and so the list goes on. It would be very helpful for us if California matches were not at the top of the list of suggested matches. I am not ignorant, but I find it a challenge knowing even a very small fraction of our diverse taxa, and usually select order for insects, but I need help in choosing the order rather than selecting from a list of species as first option.

Thanks for a wonderful iNat facility

Les Powrie, Cape Town, South Africa