Proposal for dealing with the proliferation of observation fields

[Janel Johnson]

There seems to be a lot of confusion and frustration around the use of observation fields and the excessive number of duplicate fields. As a data manager, it's frustrating to try to reconcile all of these fields when extracting the data for uses outside of iNaturalist. It also makes searching of filtering by field data a lot less useful.

I would like to propose some feature requests to deal with the existing duplicate fields and slow down the creation of new ones. 

1. Deprecated fields: Three columns could be added to the underlying field table in the database: "Deprecated" (yes|no), "Replacement" (id of field to use instead), and notes. Deprecated fields wouldn't be deleted but they wouldn't show up in the list of available fields. Project admins should be notified when their fields are marked as deprecated and deprecated fields in an observation page could use a different format to indicate their status. Curators could mark a field as deprecated and designate a replacement, perhaps even require the designation of a replacement field when deprecating fields to avoid orphans. Fields that reference "location", "date", or other information that is already built into the observations should have a special deprecation clause. 
   
   
2. Include description in field search: I recently ran into this problem when searching for an "Interaction:" field to use for a pollinator. "Interaction: Flower visited by" doesn't show up when searching for "pollinator", even though the word pollinator is used in the description.
   
   
3. Additional steps for creating fields: The process of creating a new field should require the user to first search for existing fields and include text stressing the importance of using more general fields to collect data instead of having special fields that only apply to one project or species.
   
   
4. Allow more than one instance of a field per observation: Maybe a butterfly visited more than one species of flower, maybe a photo shows five species of plants. I know this a bit more difficult to program than some of the other items in this list but I think it's very valuable. There are a number of fields like "2nd associated species" that could be deprecated if more than one associated species was allowed.
   
   
5. Allow for field name translations: Projects that cross language borders would be better served by fewer fields with name translations than duplicate fields with English and Spanish names for instance, particularly for taxon lookup fields.  I know that a field with a list of selections is a little more onerous when it come to translations so there are certainly situations where it would be better to have duplicates in different languages.
   
   
6. Add instructions in the project page: Sometimes project administrators want to collect very specific data so they create a new field and add their own instructions. If there was a place for them to instruct users about how to use the existing fields in the context of a project, they are less likely to create extraneous versions of the existing fields.

I know this is a huge list and some of these ideas are much more difficult to implement than others. I have a lot of experience with databases and data architecture but not much with programming UIs, so I will help where I can but don't expect me to create a new branch on GitHub.

Ken-ichi posted a list of fields two years ago and there were maybe two pages worth. The list is now 194 pages long and if it keeps growing at that rate it's going to become completely unmanageable soon.

Thoughts? 

[meu...@landcareresearch.co.nz]

Janel - as I've indicated in previous conversations, I totally support the intent and logic behind your proposal to deal with 'unmanageable' numbers of (replicate) fields. As u imply, it is likely to result in a data crypt rather than a database - when it comes to extracting useful information. Another criterion for deprecating fields would be if they have only ever been used a few times (or not at all in some cases).  the same argument(s) could be used to deal with the proliferation of 'projects' many of which are empty or have only one observation usually because they are so specific and limited in their application or produce nothing different than existing filters can generate. These situations just create confusion and as Janel said, the community loses the ability to recover the vast amount of useful information in the iNaturalist database because the same kind of information has been entered in different ways. it raises the question of how (and who makes the decision) to combine the information in related fields. I think this has been done in the production of the new phenology graphs - which are great. I'm aware for example, that I have (out of habit and what I started with) routinely entered most common phenological condition in plants from a drop down list in the 'plant phenology' field; whereas colleagues enter yes/no/maybe/abundances separately for each phenological stage in their own separate fields. I can see that this is more scientifically valid, since most plants have several stages present at any point in time.  I'd be perfectly happy to have my observations migrated into a more generic phenology field. I prefer drop down boxes and look-up fields (for spp) than free format fields. I guess any combined field would need a longer list of drop down options to accommodate irreconcilable differences in the contributing fields, e.g. 'commonest phenological stages' AND, say, 'also flowers', 'also immature fruits', also 'mature fruits' etc. Go for it!    

[AfriBats]

Just to express my support that Observation Fields need to be better managed. Maybe admins/ developers could briefly sketch their roadmap for that.

Jakob

[lelliott]

I agree with the fact that it's a problem. For instance, for something as basic as number of individuals, there are three possible fields: "Number_of_Individuals", "Number of individuals", and "Number of Individuals". I'm not sure I've used the same one every time I recorded that.

[Janel Johnson]

So, admins, any thoughts?

[Scott Loarie]

Hi Janel et al.,

Thanks for your thoughts on this.

These problems stem from the fact that Observation Fields were designed to support projects that want to do their own thing (e.g. 'we'll use iNaturalist for our effort if we can collect metadata exactly like this'). Almost by definition this leads to proliferation and confusion. 

We'll have a new version of the Observation Detail page hopefully ready to demo and get feedback on by the end of this week. But one of the things this new version will contain is a new feature called 'Annotations' that we hope might be a solution to this problem.

The strategy would be to continue to embrace Observation Fields as a vehicle for letting projects do their own thing as they were designed to do. And maybe take steps to make it clearer that observation fields are associated with projects. But when there are observation fields that aren't attached to a particular project or really have taken on a life of their own and have community support/consensus outside of projects, to roll them over to Annotations. Unlike Observation Fields, Annotations are designed to get everybody on the same page when possible rather than have people doing their own thing. 

We're proposing to launch with 2 Annotations: Life Stage, and Sex. And then to work with the community to bring more Annotations online as they emerge from the community.

The way these annotations work is that (1) they can be subset taxonomically - e.g. we can have an annotation that only applies to insects. (2) they can be mapped to observation fields so that the existing proliferation of observation fields for 'sex' all map to the 'sex' annotation. (3) build some 'power tools' like the Identify tool that makes it more efficient for people to add annotations to observations and (4) they propagate up to the Taxon pages. This is whats allowing us to do things like the like grouping photos or histograms by Life State on the taxon page, e.g. http://www.inaturalist.org/taxa/47916-Actias-luna

Curious to hear your thoughts, but this might make more sense when we can demo the new Observation Page hopefully next week. But in summary the proposal is to:

1) Continue embracing Observation Fields as a vehicle for projects to do their own thing as they were designed to do 

2) Introduce a new feature called Annotations as a vehicle for collecting metadata when the community is on the same page and embrace a strategy for rolling observation fields that fit this roll (e.g. sex etc) over to annotations

Best,

Scott

On Thu, Mar 30, 2017 at 11:08 AM, Janel Johnson <jdjo...@state.nv.us> wrote:

So, admins, any thoughts?

--
You received this message because you are subscribed to the Google Groups "iNaturalist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist+unsubscribe@googlegroups.com.
To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

--

--------------------------------------------------
Scott R. Loarie, Ph.D.
Co-director, iNaturalist.org
California Academy of Sciences
55 Music Concourse Dr
San Francisco, CA 94118
--------------------------------------------------

[Charlie Hohn]

Love the annotations.. any chance we can get 'flowering phenology' in the rollout too? That one is pretty solid in the community, and is very well used in Vermont.  Though it does have the one big confusing point - it needs to either say 'dominant flowering phenology' or let you choose more than one because plants often have two stages at once (ie, flowering AND has fruits/seeds on the same plant). But anyhow, yeah this looks awesome.

On Thursday, March 30, 2017 at 2:33:45 PM UTC-4, Scott Loarie wrote:

Hi Janel et al.,

Thanks for your thoughts on this.

These problems stem from the fact that Observation Fields were designed to support projects that want to do their own thing (e.g. 'we'll use iNaturalist for our effort if we can collect metadata exactly like this'). Almost by definition this leads to proliferation and confusion. 

We'll have a new version of the Observation Detail page hopefully ready to demo and get feedback on by the end of this week. But one of the things this new version will contain is a new feature called 'Annotations' that we hope might be a solution to this problem.

The strategy would be to continue to embrace Observation Fields as a vehicle for letting projects do their own thing as they were designed to do. And maybe take steps to make it clearer that observation fields are associated with projects. But when there are observation fields that aren't attached to a particular project or really have taken on a life of their own and have community support/consensus outside of projects, to roll them over to Annotations. Unlike Observation Fields, Annotations are designed to get everybody on the same page when possible rather than have people doing their own thing. 

We're proposing to launch with 2 Annotations: Life Stage, and Sex. And then to work with the community to bring more Annotations online as they emerge from the community.

The way these annotations work is that (1) they can be subset taxonomically - e.g. we can have an annotation that only applies to insects. (2) they can be mapped to observation fields so that the existing proliferation of observation fields for 'sex' all map to the 'sex' annotation. (3) build some 'power tools' like the Identify tool that makes it more efficient for people to add annotations to observations and (4) they propagate up to the Taxon pages. This is whats allowing us to do things like the like grouping photos or histograms by Life State on the taxon page, e.g. http://www.inaturalist.org/taxa/47916-Actias-luna

Curious to hear your thoughts, but this might make more sense when we can demo the new Observation Page hopefully next week. But in summary the proposal is to:

1) Continue embracing Observation Fields as a vehicle for projects to do their own thing as they were designed to do 

2) Introduce a new feature called Annotations as a vehicle for collecting metadata when the community is on the same page and embrace a strategy for rolling observation fields that fit this roll (e.g. sex etc) over to annotations

Best,

Scott

On Thu, Mar 30, 2017 at 11:08 AM, Janel Johnson <jdjo...@state.nv.us> wrote:

So, admins, any thoughts?

--
You received this message because you are subscribed to the Google Groups "iNaturalist" group.

To unsubscribe from this group and stop receiving emails from it, send an email to inaturalist...@googlegroups.com.

To post to this group, send email to inatu...@googlegroups.com.
Visit this group at https://groups.google.com/group/inaturalist.
For more options, visit https://groups.google.com/d/optout.

[kestrel]

I like this idea a lot - and would love to see an annotation for alive / dead as well - since that can be applied to pretty much anything observed & posted on iNat! Plus, it would be useful for helping to track die-offs in species - e.g., marine birds and mammals that end up washing ashore.

Thanks!
Alison

[Janel Johnson]

Annotations sound like a great idea, particularly to link certain selections to certain taxonomic groups. I get the feeling there will be a flood of requests for new annotation types.

Some questions to consider (don't need to be answered now)

Will the transfer of data from observation fields to annotations be one-time, repeated, or dynamic?

Will the form for creating new observations and projects encourage the use of annotations over obs fields?

How will the observation form adjust the available annotations to match the taxon? Would the user need to enter some minimum level of identification before the annotations are available? This is something I struggled with in designing our own data forms for in-house use.

[Charlie Hohn]

oh yeah one other thing - please make the annotations default to showing on each observation of relevant taxa type (blank if not filled out). For some reason adding fields takes a really long time on my computer and it's a pain. If annotations are just present people will be more likely to fill them out. And the idea of making an interface specifically to add these is awesome.

[David K]

Scott - I also think that this looks like a nice step forward, and you have picked two annotations that seem to have wide application/interest.  Two candidates for next phase may be 'alive/dead' (that defaults to alive) and 'count', as there are just way too many ways to log a count of things.

In the life stage example, will you roll the 'competing' fields "butterfly/moth life stage" and "insect life stage" together?  I can't really see why we need two that overlap so dramatically.  Currently, I need to download both and merge the columns on several projects as you don't know who's using what.  That is the downside of the do-what-you-want approach, I never know if I'm capturing everything.

The screenshots look great and I look forward to seeing it live.

David

[lelliott]

Annotations sounds like a great idea. I would adopt it immediately.