multiple issues with checklists

[AfriBats]

Hi everyone

It seems the link to checklists is no longer available on project pages. I find these very useful and hope they can be brought back!

Also, I'm struggling to understand how different checklists are working. This project checklist has 260 species listed as observed (which I guess means research grade obs)
http://www.inaturalist.org/lists/94885-Hinterhoftickers-Check-List?q=&taxon=&observed=t&rank=species&taxonomic_status=active
while the corresponding place checklist says "219 of 221 species confirmed" (again probably referring to research grade obs):
http://www.inaturalist.org/places/hinterhof

For instance, common crane (Grus grus) is completely missing from the place checklist although there are 2 RG obs from that place:
http://www.inaturalist.org/observations/1064569
http://www.inaturalist.org/observations/1047678

There's clearly a discrepancy here, and I would appreciate to hear what's causing this. Note that in this case the project and place checklists should be identical.

Thanks, Jakob

[Scott Loarie]

Hi Jakob,

There is now a preference on the project edit page to show the project list or not. Before the recent changes they always showed for non-bioblitz projects and never showed for bioblitz projects to avoid confusion with the list of species that the observations represent (which is what Biobltizes are all about). They now can be displayed or not independently as the project team chooses. 

##begin rambling about listings: 

Checklists have some known bugs. They are very complicated because they generate listings both from manually entered listings and also from observations. And there are definitely some situations where listings aren't getting generated from observations that we haven't had bandwidth to investigate.

We've partially not invested as much recently in checklists because there are in many ways better ways on iNaturalist now to generate and manage a static list of species (guides: http://www.inaturalist.org/guides)  and to generate a list of species directly from a set of observations (e.g. http://www.inaturalist.org/observations?nelat=37.5359&nelng=63.808593900000005&place_id=any&swlat=-47.1313489&swlng=-25.383911000000012&taxon_id=40268&view=species )

We're still struggling to identify the use case that makes the combined functionality Checklists offer (adding and managing manually created listings combined with generating listings automatically from observations) that we should invest in.

Personally, the idea of being able to 'complete a set' has some appeal to me. E.g. there were thought to be 11 species of amphibians on Pepperwood Preserve and for the longest time everything on the list except Black Salamander was checked off (10/11). But this year they succeeded in finding Black Salamander https://twitter.com/TunefulToad/status/696759582022987776

and completed the set (11/11): http://www.inaturalist.org/places/pepperwood-preserve#taxon=20978

And Likewise, I like the identotron functionality of being able to choose an identification from a set of species that includes manually created listings (things that haven't been observed) and listings created from observations.

But I also feel that coming up with a system that can really properly manage manually created listings and the tools needed to curate them so that every time the taxonomy changes we don't accumulate lots of database cruft is going to take a lot of work. And we've really been focusing this last year on observing, identifying, and visualizing the data represented by those identified observations rather than tools for managing manually created listings.

Sorry for rambling. Listings are dear to my heart. But I also have to begrudgingly admit that they are not as core to iNaturalist as observations and identifications which we are still trying to get right.

Scott

--
You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.
To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

--

--------------------------------------------------
Scott R. Loarie, Ph.D.
Co-director, iNaturalist.org
California Academy of Sciences
55 Music Concourse Dr
San Francisco, CA 94118
--------------------------------------------------

[AfriBats]

Thanks for the background, I've now brought back the link to project checklist where we use it (incidentally, the AfriBats checklist is a complete one that we update regularly if needed: http://www.inaturalist.org/lists/57284-AfriBats-Check-List?rank=species)

I understand your concerns about the lists, especially due to the curatorial work that is currently needed to keep them accurate and up-to-date. However, I think they are immensely useful in regions were there are few observations yet, e.g. Africa. I'm constantly adding species to the continental list, or to individual countries, so that the Identotron produces useful results for groups where there are no range maps.

Any idea what is causing the large discrepancy in 'Hinterhof' case above? I haven't manually added any species to the project nor the place checklist, and the species total differs by 40 species (220 vs 260). The whole issue came up when I wanted to see the phenology of common cranes observed from Hinterhof just to find out that this species isn't included at all here (despite 2 RG observations, see above): http://www.inaturalist.org/places/hinterhof

Last but not least I realized that some places can be no longer used as a search mask with the new observation interface - what's the reason for this?
http://www.inaturalist.org/observations?place_id=258261
http://www.inaturalist.org/observations?place_id=199457

Thanks, Jakob

[AfriBats]

 

http://www.inaturalist.org/observations?place_id=258261
http://www.inaturalist.org/observations?place_id=199457

I should add that I used to search with these 2 place previously (which should theoretically correspond to the links above):
http://www.inaturalist.org/places/africa-surrounding-islands
http://www.inaturalist.org/places/europe--5

[Scott Loarie]

Hi Jakob - 

I suspect there's a bug there where they didn't index because they are so big. Next time we do a clean reindex (which should be sometime in the next week or so) they should be fixed.

Along those lines. We've since put a cap on the size of community curated places to ~the size of Texas because the creation of places that encompass many many observations (like continents) triggers a lot of backend processes (the reindexing) that we're realizing aren't maintainable. 

We are currently maintaining countries/states/counties and some parks internally here as 'standard' places. because there seems to be demand for continents from you and others and we're no longer letting people manually create community curated places that large, we're also going to start maintaining these continents (which are derived from the union of the country data we maintain split according to the widely accepted 7-continent model).

Out of curiosity would the standard continents as defined here be suitable for what you're using those community curated continents mentioned above? Existing large places like those will be grandfathered in, but it would be great if people found these new Standard continents useful

--

You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.
To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

[AfriBats]

Hi Scott

I suspect there's a bug there where they didn't index because they are so big. Next time we do a clean reindex (which should be sometime in the next week or so) they should be fixed.

The first issue (large discrepancy of Hinterhof checklists) is certainly not due to being large - quite the contrary. Please advise if & how the discrepancy could be reduced - I don't care about a few species missing here or there, but 40 missing from a total of 260 seems a lot:
http://www.inaturalist.org/lists/94885-Hinterhoftickers-Check-List?q=&taxon=&observed=t&rank=species&taxonomic_status=active
http://www.inaturalist.org/places/hinterhof

 

Out of curiosity would the standard continents as defined here be suitable for what you're using those community curated continents mentioned above? Existing large places like those will be grandfathered in, but it would be great if people found these new Standard continents useful

Looks good as far as I can see from this preview. However, the nice thing about a place like "Africa & surrounding islands" is that it includes the marine zone around it, which I guess would be lost if it's a merge of country boundaries. Also, observations slightly outside the merged country boundaries wouldn't be found with it.

I still don't understand why the 2 places (Europe and Africa plus islands) is no longer working as a search masks in the observations window. I would assume that querying observations with a place filter is independent on the checklists of these places, right?

[Scott Loarie]

I suspect there's a bug there where they didn't index because they are so big. Next time we do a clean reindex (which should be sometime in the next week or so) they should be fixed.

The first issue (large discrepancy of Hinterhof checklists) is certainly not due to being large - quite the contrary. Please advise if & how the discrepancy could be reduced - I don't care about a few species missing here or there, but 40 missing from a total of 260 seems a lot:
http://www.inaturalist.org/lists/94885-Hinterhoftickers-Check-List?q=&taxon=&observed=t&rank=species&taxonomic_status=active
http://www.inaturalist.org/places/hinterhof

This project checklist

http://www.inaturalist.org/lists/94885-Hinterhoftickers-Check-List?q=&taxon=&observed=t&rank=species&taxonomic_status=active

And the place checklist http://www.inaturalist.org/check_lists/94884-Hinterhof-Check-List

that's driving the counts you see here

http://www.inaturalist.org/places/hinterhof

So there's any number of reasons why those lists should have large discrepancies.

Are completely independent and have nothing to do with one another. The project checklist contains any listings you added manually as well as listings automatically generated from observations added to the project (as well as complicated listing cruft that hangs round when observations are deleted or otherwise altered)

Likewise, the place checklist contains any listings you added manually as well as listings automatically generated from observations contained within the place (as well as complicated listing cruft that hangs round when observations are deleted or otherwise altered)

By listing cruft I mean, for example that sometimes if an observation of a species is created in a county then a listing of that species is generated for the place and things propagate around in complicated ways for example a listing for the genus might be generated for the encompassing state. If the observation is deleted, then the listing for the species in the county might be deleted, but maybe the listing for the genus in the county will hang round as cruft. As I said before, at this point, iNat has really good tools for managing observations and identifications. But the tools for managing listings unassociated with observations just really aren't sophisticated enough for showtime IMO. We could invest there, but there's also a strong argument from the Museum community, that listings should always be sourced to occurrences (specimens, iNat obs etc.) because that way if there are changes in taxonomy or the extent of the list (say the geographic extent) then the list can be maintained. Alternatively, you just end up with a lot of names on a list thats not anchored to anything real. Personally, I find that a pretty compelling argument. I completely agree, that that means we need occurrences to drive those lists (for places with no occurrences) but thats a problem we can solve by pushing for more observers/IDers. Anyway, interesting argument which I'm finding value in looking at some of these National Park species lists which are hard to interpret given changes in park boundaries and taxonomy.

 

Out of curiosity would the standard continents as defined here be suitable for what you're using those community curated continents mentioned above? Existing large places like those will be grandfathered in, but it would be great if people found these new Standard continents useful

Looks good as far as I can see from this preview. However, the nice thing about a place like "Africa & surrounding islands" is that it includes the marine zone around it, which I guess would be lost if it's a merge of country boundaries. Also, observations slightly outside the merged country boundaries wouldn't be found with it.

For all standard places we're including a 0.05 degree coastal buffer (which matches the official US census boundaries) - this makes the places look consistent and somewhat uniform. Because the land data we're using includes extremely small islands, this buffered area incorporates nearly 100% of iNat observations. There are a handful of truly pleagic observations made from transatlantic boats etc (like this one http://www.inaturalist.org/observations/2473292) but its something like <0.01% of all observations.

 

I still don't understand why the 2 places (Europe and Africa plus islands) is no longer working as a search masks in the observations window. I would assume that querying observations with a place filter is independent on the checklists of these places, right?

As I mentioned earlier, I suspect something went wrong indexing them. When we launched observation search, we launched a completely separate backend infrastructure for filtering observations thats faster and more scalable. Yes the observation search infrastructure (which is new and awesome) doesn't have anything to do really with checklist infrastructure (which is old and buggy)

 

--
You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.
To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

[Scott Loarie]

One more thing - sorry if I'm misunderstanding your question.

I guess my summary statement would be that we haven't been focusing on how lists are working for over a year and there are some known issues with how listings are generated from observations and destroyed if those observations change.

We have been focusing more recently on how to generate guides (which are static lists that don't consider observations) 

And we have been focusing very recently on how to generate lists directly from observations (e.g. everything is sourced to an observation and the list is generated dynamically). Here are the dynamic species lists generated from the observations in your project:

http://www.inaturalist.org/observations?place_id=any&project_id=1200&verifiable=any&view=species

and the observations in your place:

http://www.inaturalist.org/observations?place_id=58712&view=species

So unless you need to mash up listings sourced to observations and listings manually added, I'd recommend not using lists but rather using guides (for manual listings) and observation search (for listings sourced to observations)

Also, with the new observation search that can do things like this: http://www.inaturalist.org/observations?place_id=58712&view=species there's a less compelling reason to set up a project for a place unless you want:

the advantages (1-5 listed here: http://www.inaturalist.org/pages/bioblitz%252bsetup#project_page) that projects provide

[AfriBats]

Yeah, I think we haven't understood each other re the Hinterhof checklists. First of all the place "Hinterhof" was created by me for the "Hinterhofticker" project, and is used in the latter as a geo rule - observations have to be within that place to qualify for the project. Second, I haven't manually added species to either the project (www.inaturalist.org/lists/94885-Hinterhoftickers-Check-List) or the place checklist (http://www.inaturalist.org/check_lists/94884-Hinterhof-Check-List), hence in theory both should be identical. It might be actually a good test case for you guys to identify some of the checklist issues that seem to be lingering around.

For the specific case above I don't need any static lists (= guides), and I'm actually mostly interested in dynamically created lists derived from observations. If I take your examples from above (and include observations without media), the resulting species number is indeed almost identical (376 vs 374 species):
http://www.inaturalist.org/observations?project_id=1200&view=species&verifiable=any
http://www.inaturalist.org/observations?place_id=58712&subview=table&verifiable=any

Since there are hardly any other user but me contributing to both the project and place, and since the place has no child places, there shouldn't be much "listing cruft" if any at all.

I found the old lists nice as the allowed easy access to phenology of individual species, taxonomic views, leaves etc., most of which seems unavailable in the observation view that you're suggesting as an alternative. But maybe I haven't discovered all options of the observation queries!

Thanks for the additional clarification re Europe & Africa. Buffered "country continents" sounds good, and I would see little use for the manually created continents that I had been using before.

[Scott Loarie]

Great - 

That will be useful when we get back to sleuthing issues with checklists

We're eager to build taxonomic views and the phenology charts into obs/search soon. They are currently displaying leaves.

Great re: continents

--

You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.
To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

[AfriBats]

Great re: continents

One last thing re the new continents: If this is only based on aggregated countries, it doesn't follow the usual delimitation of continents in some areas (e.g. Russia and Turkey being part of both Europe and Asia).

I've put quite a bit of effort into creating the shape of Europe, which should have a pretty accurate boundary where it intersects with Asia:
http://www.inaturalist.org/places/europe--5

Would be nice if this isn't lost but hopefully somehow integrated in the new continental places created for iNat!

[Scott Loarie]

Hi Jakob,

We aggregated the countries to get landmass but split according to the continental delineations. Looks like we're both used the 'modern convention following the crest of the Greater Caucasus, the Ural River, and the Urals range'. So they should be the same if not similar. As you say, was a real pain to find and incorporate that particular boundary which we pulled from http://www.arcgis.com/home/item.html?id=3c4741e22e2e4af2bd4050511b9fc6ad

[AfriBats]

Excellent!

[AfriBats]

Hi Scott

It seems you're rolling out the new continents, which look great
http://www.inaturalist.org/places/europe-84d7a1a6-93a4-4b07-a566-1d4215b0dfe2
http://www.inaturalist.org/places/africa-d6e73131-50b2-458c-bf7c-837f48f7844a

Do you plan to merge other versions with these, for instance
http://www.inaturalist.org/places/europe--5

http://www.inaturalist.org/places/africa
http://www.inaturalist.org/places/africa-surrounding-islands
http://www.inaturalist.org/places/afrotropic-ecozone

If so, I hope something can be done to "rescue" species manually added to the latter places. I've added hundreds if not thousands of species to larger places (especially for 99681 = Afrotropic Ecozone and 199457 = Europe) in cases where species had no associated places at all. Maybe you can filter species listed for only these 2 places (= without any associated observations) and transfer them to the new places (new Africa and Europe, respectively). It has become a very useful resource when using the Identotron and it would be a shame to lose all the work that went into it.

Thanks, Jakob

[Scott Loarie]

Hi Jakob,

Yes - we're rolling out an update to the 'Standard Places' that we're maintaining internally. Things are still processing so search results using them might be a little funky until tomorrow.

We'll send out a description of these standard places and their sources shortly, but they are continents, countries, states, and counties as well as a small set of parks (just US NPS for now). 

Everything else is a 'Community Curated Place' meaning it was created by a user. (Unfortunately we weren't always tracking who created places so some have unknown ownership).

Places can have all sorts of things associated with them:

Projects

Lists & Listings

Guides

Taxon Names

Conservation Statuses

We don't want to merge Community Curated Places that have these kinds of things associated with them because it might screw up someone's use of them (e.g. 'Why did my project boundaries change?!').

But you're welcome to move any of your content (projects, listings, etc.) over to a new Standard Place. And if you need any help doing this programmatically rather than manually, happy to help, just contact me off group

We're hoping to make some changes to bette surface Standard Place alternatives when they are available to discourage people from creating their own duplicate versions of existing places which has happened alot in the past (mostly because it wasn't clear the place they were after already existed). But if people for whatever reason do want their own variant of an existing place they can make a 'community curated' version.

Scott

--

You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.
To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.