Exporting my Filtered Observations

[BJ Stacey]

When I filter my results and try to export (CSV) I end up getting all of my observations, not just the ones under that filter.

As an example, I am trying to export only my Odonata observations.

Instead of getting only <400 results on the export I am getting a CSV of all 12,457 observations.  This wouldn't be overly bad if the CSV had a way to sort the results to Odonata, but the closest that I can come is Insecta which leaves me with 1,198 results to sort through.  Having more fields (county, state, order, class, etc) could also solve this problem.  The observations are already tagged and able to be filtered on these levels so having them in the exports would hopefully be fairly easy to incorporate.  I'm a data geek and not being able to get my data is driving me nuts.

As of now, is there a way to get an export of my filtered observations?  It would really save me a tremendous amount of time to have this ability.

Thanks for everything that you do already offer.

BJ Stacey

aka Finatic

[Ken-ichi]

No, there isn't, unless you're willing to write some code, which would
require retrieving
http://www.inaturalist.org/observations/finatic.csv?taxon_name=Odonata&page=n
until there are no more results. I'd like to create some kind of data
export tool, though, since people keep running into this problem. What
exactly are you trying to do with the data, anyway?

> --
> You received this message because you are subscribed to the Google Groups
> "iNaturalist" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to inaturalist...@googlegroups.com.
> To post to this group, send email to inatu...@googlegroups.com.
> Visit this group at http://groups.google.com/group/inaturalist.
> For more options, visit https://groups.google.com/groups/opt_out.

[BJ Stacey]

What I plan on doing is using either a spreadsheet (not ideal) or Access database (better option) with the data.  From that I can create lists of what I have seen at a county level.  By knowing what I have already seen I will be better able to target new species for me (on a county level) before I take a trip.

I do county listing for birds, odes and leps here in California.  I am also building lists for Arizona since I travel there at least once a year.  SO I am really only concerned with getting this deep level of data for these two states, but filtering out the other states should be fairly easy.

In a couple of weeks I'm attend the CalOde Blitz and will be championing iNaturalist.  I would love to be able to show this feature off at that time and try to recruit more users.  I know of only one other iNaturalist user that will be attending so there will be many people possibly unaware of this great tool.  Showing how it can be used for listing purposes will truly be a recruitment enhancement.

BJ Stacey

[Kent McFarland]

I might not understand what you need fully, but you don't need to make lists by hand like this. Just use the listing feature on inat combined with places. You make a place and then create a list for that place for your material under your list. 

Kent

____________________________

Kent McFarland
Vermont Center for Ecostudies
PO Box 420 | Norwich, Vermont 05055
802.649.1431 x2

Visit Our Pages:

[Kent McFarland]

Here's a little link to a quick article about place-based lists.

http://www.inaturalist.org/projects/vermont-atlas-of-life/journal/1914

Kent

____________________________

Kent McFarland
Vermont Center for Ecostudies
PO Box 420 | Norwich, Vermont 05055
802.649.1431 x2

Visit Our Pages:

[BJ Stacey]

Hi Kent,

Thanks for the suggestion.  That won't quite give the result that I'm looking for.  I have the lists of possible species, but I want to be able to easily cross reference it with species that I have already seen. 

Unfortunately the listing feature isn't robust enough yet.  Maybe over time it will be but right now I'm finding that is isn't including all species, contains some that are not there and doesn't take in seasonality.  The seasonality is especially important with short lived and/or migrant animals.

By having access to my sighting data of some fields that aren't yet available on export I will be able to do a lot of data mining, some of which I may have have thought of quite or may evolve over time.

BJ Stacey

--
You received this message because you are subscribed to a topic in the Google Groups "iNaturalist" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/inaturalist/oRGMkskv8yY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to inaturalist...@googlegroups.com.

[Lynn Watson]

When I created the plant list for Figueroa Mountain, I combined information from the iNat taxonomic csv and the iNat observations csv, to add order, class and created my own .csv file, which was then used to create this:

http://natureshutterbug.com/wordpress/figmtnflora/?sort=list

The location was also include in the generated .csv file, to allow this to be created:

http://natureshutterbug.com/wordpress/figmtnflora/?sort=place

I had separately generated .csv files for the two options, and they were pre-sorted for use.

For updates, I will just run the .csv files through the same software, and then upload the generated .csv files to my site. 

[Ken-ichi]

Ok guys, I put together the beginnings of an export tool:

http://www.inaturalist.org/observations/export

It's probably still buggy so I haven't linked it anywhere, but please check it out and let me know if this comes close to meeting your needs. It basically accepts all the same filters the normal observation searches do, but it will generate CSV for *all* results matching the query. I also added the ability to add higher level taxonomic columns, though they do kind of slow down the export a bit. Lynn, I know you'll want observation fields, but at the moment it only lets you choose from the most recent 50 fields you've used. Eventually it should let you add any observation field to your export.

Anyway, let me know what you think if you get a chance to try it out.

[Lynn Watson]

Wow, awesome - that was fast. Over the holiday season, I was going to write software to extract the butterfly/plant info I need for SB county, and was dreading it because do not have a good tokenizer (to get past the comments field). 

A question - on what basis did you select the fields that come up - there are lot more. Just the most used ones?

Great stuff.

[Charlie Hohn]

i think Ken-Ichi says it's the last fields you've used.

[BJ Stacey]

A really nice start.  I like being able to select the desired fields. 
Do you plan on adding the two that I most desire, State and County?  If not, I guess I can create custom fields for those so that I can pull it.  Unfortunately that means that I'll have to go back and touch all of my observations to date, quite a feat.

BJ Stacey

[Ken-ichi]

I'll see what I can do. If you're familiar with GIS, though, that should be a pretty straightforward spatial join if you have a shapefile of counties: http://www.census.gov/geo/maps-data/data/cbf/cbf_counties.html

[BJ Stacey]

If it is easy to add those fields (or all places that get auto-populated and show under the photo on the observation page) that would be great.  But if not, I would rather you spend your time on other enhancements or bugs.  I can work with what is now available though it will be a bit of work. 

The GIS would be more work than me adding custom fields to each observation.  I can do a batch tag of past ones (200 at a time) and moving forward just add those fields when entering new observations.  I tested this last night and will work.  It's just all of my previous observations will be a pain.  But, this way I can easily get everything in a CSV and convert it to Excel/Access.

This tool is really going to be a great feature.

BJ Stacey

[Lynn Watson]

Thanks - sometimes my speed reading does not work - I read last 50 obs.

[Lynn Watson]

I have never used 50 fields - probably less than 12.

[Christopher Tracey]

I just wanted to chime in that this export tool is great!  Really useful!

Thanks!

[Ken-ichi]

Hey folks, I put a little more work into this tool (http://www.inaturalist.org/observations/export). For exports that take a while there should now be an option to receive the results via email. There were also some bugs that were causing certain exports to take way too long, which I've fixed (though 10,000+ exports with the taxonomic columns will still take a while).

Let me know if you find any bugs. Just about ready to get this linked.

[Kent McFarland]

Thank for doing this! It seems to work fine if I log back in later and get the file at the bottom of page. But, when I tell it to email me, I don't seem to get an email. I have searched spam boxes to make sure and no sign of it. 

Kent

____________________________

Kent McFarland
Vermont Center for Ecostudies
PO Box 420 | Norwich, Vermont 05055
802.649.1431 x2