Re: feedback for VertNet prototype search API - URGENT
Aaron Steele
Everything you can possibly think of. You're off to a great start.
BTW, here's the test data that the API is currently searching:
http://goo.gl/HZptL
Also, regarding text matching queries, the API is implemented on
Google App Engine where text matching queries like "taeniur%" to find
"taeniura" are not currently supported. But that's coming soon!
http://www.google.com/events/io/2011/sessions/full-text-search.html
Sooooo in the meantime, the prototype API supports querying
case-insensitive *exact* string matches on Darwin Core fields and also
a full text keyword search using the "q" parameter.
> ?limit=999999&georeferencedby=Epperly returns nothing but ?limit=999999
> returns georeferencedby Kevin Epperly (UWBM) - is that in scope at this
> point?
Yeah, so in the test data, "georeferencedby" values are "Kevin Epperly
(UWBM)", not "Epperly", so the following query will return no results:
http://canary.vert-net.appspot.com/api/search?georeferencedby=Epperly
As you found out though, a full text search on "Epperly" returns many results:
http://canary.vert-net.appspot.com/api/search?q=Epperly
Again, this is where text matching queries on App Engine will come into play!
> Matches seem to be at the whole-string level only - just in these few
> records county=alameda%20co returns some stuff and county=alameda returns
> some different stuff and county=alameda%20co. returns even different stuff.
> I don't think that's usable.
Good catch. So the exact string matches on Darwin Core fields are true
to the underlying data. In particular, here are the counts in the test
data for the unique variations of "county" fields containing
"alameda":
41 "alameda co."
59 "alemeda co"
539 "alameda"
Once App Engine rolls out support for text matching queries, all
results could be found with "count=alameda%".
> DWC data are always a heterogeneous mess -
> that's sort of the point - and it's critical to be able to ignore things
> like county abbreviations and gender-changes on scientific names. (That is,
> search "taeniur%" to find "taeniura" and "taeniurus," both of which are
> likely to appear in any two collections.)
+1, and this is where support for text matching queries will
definitely come into play!
> year=1904&stateprovince=california&q=bufo,alameda&limit=1&offset=100
> (example from https://github.com/VertNet/Software/wiki/Prototype-Search-API)
> returns nothing
Good catch! The documentation is intended to be general, so the
examples there may or may not work depending on what data are loaded
(that's why only the request path is included there but not a full
URL). Is this too confusing? Proposed alternatives?
> bb is very slow, at least sometimes.
> ...and adding bb to searches seems to jump the limit to 1001.
Yep, please see issue 33 for what's happening there:
https://github.com/VertNet/Software/issues/33
> Access to all data would be lovely. Removal of the record limit would be
> moreso - VertNet will be most useful in very large quantities, after all.
Well, so one of the things that will make VertNet APIs scale in terms
of QPS (queries per second) is limiting the request time. Basically
our scalability and performance will increase as per-request handling
times decrease. To do that we limit the number of records in a single
response. You can still get all the data by paging across multiple
responses using the standard "limit" and "offset" parameters. We can
totally play with the limits though! What would be better than the
current defaults?
> On that note, a less-verbose JSON return format would shrink return filesize
> by ~50% - maybe much more
Well, the JSON response format remains true to the underlying data.
It's basically just the verbatim row in publishers CSV file. I'm not
sure how we can make that less verbose? We do have some options for
optimizing JSON responses via binary using libraries like bson
(http://bsonspec.org) and MessagePack (http://msgpack.org), but I
don't really like that since a big win for JSON is human readability.
Thoughts?
> given the prevalence of NULL in DWC data and the
> potential for very large recordsets. I'm not sure what the format change is
> called, but....
> [
> {georeferenceprotocol:[
>
>
> 'MaNIS/HerpNet/ORNIS Georeferencing Guidelines, GBIF Best Practices',
> 'MaNIS/HerpNet/ORNIS Georeferencing Guidelines, GBIF Best Practices',
>
>
> 'MaNIS/HerpNet/ORNIS Georeferencing Guidelines, GBIF Best Practices']
> },{stateprovince:[
>
>
> 'California',
> 'California','
>
>
> California']
> }
> ]
In your example above, it's not clear why Darwin Core fields have a
list with repeated values?
> And if anybody's interested, I threw together
> http://arctos.database.museum/CustomPages/vntest.cfm?q=california,aves&genus=limnodromus&county=alameda
> - it just makes the json a little more readable.
Nice! But that looks like an HTML representation, not a JSON
representation, right? What am I missing?
Aaron Steele
> idea of how the service is being utilized and could limit commercial use
> without prior consent of data providers.
Yeah, this is something that we've definitely thought about. IMO, from
an API client perspective, keys are annoying and it affects the API
user experience (e.g., before we can play, we have to sign up for a
key, modify our code to manage the key, we can't pass around query
URLs without a key, etc). This is one of the reasons why APIs like
Google Maps have deprecated keys in favor of server-side ways of
monitoring usage and guarding against abuse. I'm kind of leaning that
way. Can you see a situation where a server-side approach couldn't do
what an API key could?
> 2) Following up on Dusty's comment on JSON. Have you considered offering
> optional formats ("format=JSON/CSV/...") within in your api.
We could definitely support CSV, but there'd be a CPU overhead in
converting JSON to CSV. Right now, to minimize CPU and to maximize
QPS, we pre-render and store JSON results at write-time instead of
rendering them on demand during read-time. So to support CSV in a
similarly efficient way, it would just require additional storage. The
numbers don't look too scary:
1 kilobyte per CSV result * 85 million records = 81 GB storage
81 GB storage * .17 per month = $165 per year
Thoughts?
> In the
> majority of use cases the data will model a table, so why not offer that as
> an option?
Do you mean an HTML table?
> 3)As far as GEOLocate is concerned, we need to be able to harvest all data
> for a given provider. Our current harvest options are DiGIR, CSV and Darwin
> Core Archives. The best option for us would be to have a way to get DwCA
> out of VertNet. Any chance you might be able to stream the data as DwCA?
Maybe. There are some tricky things involved in doing that in an
efficient way, so supporting DwCA might be down the road a ways. In
the meantime though, right now GL could harvest all results in JSON
for any query using the prototype search API, and then transform them
into CSV using an open source C# JSON library
(http://code.google.com/p/jsonsharp).
Would this harvesting workflow be acceptable or is int a non-starter?
I would love to get GL started on some early integration testing!
Thoughts?
John Wieczorek
containing Darwin Core field names has all of the information of a
Darwin Core Archive in a single file without the need for a meta.xml
to describe it - that's completely superfluous overhead in our case
because our model is flat.
Is there a particular use case for the having Darwin Core Archive with
meta.xml? If so, an archive could be constructed trivially from the
csv file header. But, the only reason for meta.xml is to understand
the rows in the csv file, and that information is already in the csv
file - as the header.
Aaron Steele
> containing Darwin Core field names has all of the information of a
> Darwin Core Archive in a single file without the need for a meta.xml
> to describe it - that's completely superfluous overhead in our case
> because our model is flat.
Good point!
Aaron Steele
> method will entail, but if it gets the job done I'm all for it.
Yeah, I don’t really know what our server side methods will entail
either. :) But yeah, I think we should definitely start thinking about
this. I logged an issue:
https://github.com/VertNet/Software/issues/39
> That said, I don't see end users passing around urls to the service.
> Instead they would be passing around urls to client apps that already have
> permission to access the service.
Totally. I mean, API links will definitely still be passed around like
we’re doing here, but yeah, most users will probably be dealing with
app-specific links.
> Regarding CSV:
> If you can support it, great. If not, no biggie.
I logged an issue for this:
https://github.com/VertNet/Software/issues/38
We also have an issue for creating simple clients for the prototype
search API in Python, Java, Perl, and C# if you wanted to start
hacking on a C# client for this:
https://github.com/VertNet/Software/issues/30
> Regarding GEOLocate:
> Consuming JSON is not a problem for us, just though I would shoot for our
> ideal situation (DwCA) first as we already have the plumbing for that.
Definitely good hearing what your ideal situation would be, and this
is exactly the kind of feedback that we’re looking for! I logged this
as an issue:
https://github.com/VertNet/Software/issues/40
Aaron Steele
> now the only option is to get all fields back for any given search. For
> AmphibiaWeb, I won't need all the fields, and getting all of them back will
> slow down the transfer of data.
Definitely. One concern is doing that in a way that doesn't burn CPU.
What if we come up with a streamlined record containing a core set of
fields that can be used by apps who don't want to deal with all
fields? We could then pre-render, store, and serve these very
efficiently. As a starting point for thinking about this, what fields
would you include in the streamlined list?
> Also, how can one get back fields that are not in your example? (ie, order,
> family)
Well, if the publisher provides values for the fields, they will be
available through the API. In our example data, the publisher didn't
provide these fields. Would it be more helpful to populate the test
data with all Darwin Core fields?
> I assume the API will support searches on any darwin core field?
Totally. Actually, right now the API supports searching on any Darwin
Core field or abbreviation (https://gist.github.com/1130903)! But
again, our test data doesn't have all fields, and it seems like that
would actually be pretty useful. Issue submitted:
https://github.com/VertNet/Software/issues/9
> Right now if you pass a garbage request like this
> http://canary.vert-net.appspot.com/api/search?xxx=yyy
> you get back lots of records -- probably you should have some checks for
> valid searches.
Good catch! So right now it's just ignoring garbage input and
returning the first 100 results without any filtering. Instead of
this, are you thinking a 404 might be better?
> To make searches additionally useful it would be good to enable matches of
> strings within specific fields, as Dusty pointed out. I don't actually need
> this for AmphibiaWeb (yet) but for many other applications it would make
> searches more useful.
+1, and this feature is definitely on deck for when the full text
search API surfaces on App Engine.
> I would likely use CSV if you had it (that would be much better for the AWeb
> iPhone app), but if only JSON is available that's ok for using in the AWeb
> cache.
Ok, that's very good to know! FYI we have an issue submitted for this:
https://github.com/VertNet/Software/issues/38
> Otherwise the test searches I did worked fine. I think you should increase
> the limit to 100,000.... 100 is definitely too low.
Yeah, so we'll have to be super careful about how high we jack the
limit. There's a tradeoff between limit size and request time. To
ensure API performance at scale we'll need to keep request times
around 100 ms. We'll need to do some testing to see where the limit
should optimally be. Even if it's lower or higher than 100k, client
apps will still need to handle paging I think. I submitted an issue:
https://github.com/VertNet/Software/issues/41
I mentioned this to Nelson... We have an issue for creating simple
clients for the prototype search API... If you wanted to start hacking
on a Perl client for this it would be amazing!
https://github.com/VertNet/Software/issues/30
Joyce Gross
These are the only fields AmphibiaWeb currently needs in its cache:
ScientificName
DecimalLatitude
DecimalLongitude
InstitutionCode
GlobalUniqueIdentifier
I also store some information about how/when data was loaded so I can
replace data periodically. AmphibiaWeb's list of fields is probably the
shortest anyone would ever need.
For CalBug (which could potentially make use of VertNet at some point),
there's a somewhat longer list of minimal fields we want for
cross-institutional searching -- see the fields in green on this page:
http://calbug.berkeley.edu/data/
I have other potential uses for VertNet in mind but those will depend on
funding and further work. Having all darwin core fields available would
cover all (or most) bases, but still it would be nice to limit the
number of fields returned since there are LOTS of darwin core fields.
The current API seems to return some text (the field name) for each
field whether it's populated with data or not. If someone just wants 5
fields and the API returns 100 fields, that's going to be much slower in
terms of data transfer time.
Regarding an error message about garbage input -- something friendlier
than a 404 message would be better. Maybe something like "Field xxx does
not exist in VertNet's API." Someone could potentially pass a long list
of fields to search on and waste a lot of time figuring out which one is
not supported by the API if there is only a 404 message.
As for populating the test data with all darwin core fields -- the more
the test environment is like what you'll have in the end, the better the
testing will be.
Joyce
--
Joyce Gross
http://bscit.berkeley.edu/
1007 Valley Life Sciences University of California, Berkeley
Aaron Steele
> DecimalLatitude
> DecimalLongitude
> InstitutionCode
> GlobalUniqueIdentifier
Excellent. I submitted an issue for this:
https://github.com/VertNet/Software/issues/42
> The current API
> seems to return some text (the field name) for each field whether it's
> populated with data or not. If someone just wants 5 fields and the API
> returns 100 fields, that's going to be much slower in terms of data transfer
> time.
Good catch! The API should not return fields in a record if there's no
value, and this could potentially save a lot of I/O. Issue submitted:
https://github.com/VertNet/Software/issues/43
> Regarding an error message about garbage input -- something friendlier than
> a 404 message would be better. Maybe something like "Field xxx does not
> exist in VertNet's API." Someone could potentially pass a long list of
> fields to search on and waste a lot of time figuring out which one is not
> supported by the API if there is only a 404 message.
I agree that a friendlier message would be better. I'm thinking that
the low level API should be purely RESTful and return standard HTTP
response codes for things like a resource not found error (404), but
definitely redirect to a custom 404 page with a more informative
error. Good? Submitted an issue:
https://github.com/VertNet/Software/issues/44
> As for populating the test data with all darwin core fields -- the more the
> test environment is like what you'll have in the end, the better the testing
> will be.
+1 and we have an issue open to get a fully populated data set prepared:
https://github.com/VertNet/Software/issues/9
Aaron Steele
> seems to work.
Yeah, it's not immediately clear how Specify would use the search API,
but I do see some other possible integration points.
For example, what do you think about publishing Specify data with
VertNet, similar to how it works with the IPT:
http://specifysoftware.org/sites/specifysoftware.org/files/Publishing%20Specify%20Data%20with%20IPT.pdf
Another possibility might be importing Specify data to VertNet:
http://specifysoftware.org/sites/specifysoftware.org/files/Publishing%20Specify%20Data%20with%20IPT.pdf
Of course, the annotation API might be another interesting integration
point that would allow synchronizing annotations between the two
platforms. Thoughts?
Aaron Steele
> interested in doing it. That's why we included the
> Vertnet schema as one of the pre-defined schema(ta)s available for export
> mappings.
Nice. I wonder if it would make sense to take that a step further and
export directly to VertNet over HTTP? If not, does Specify support
custom data export plugins for stuff like this?
> Which annotation API? Annotations are interesting and synchronization is
> cool but ...
Definitely cool! So the VertNet annotation API will be based on a
pretty simple publisher-subscriber model. Basically you'd subscribe to
VertNet annotations by filter (for example, subscribe to annotations
by record id, or subscribe to annotations of all birds in Madagascar),
and then you'd register a single URL endpoint where all published
annotations would be pushed in near real-time. That's it!
Oh, and publishers and subscribers would be wired up using the
PubSubHubbub protocol:
http://code.google.com/p/pubsubhubbub
What do you think? See any potential here for some collab?
Dusty
On Wednesday, August 17, 2011 12:32:29 PM UTC-5, Aaron Steele wrote:
> What specifically is it that you'd like tested?Everything you can possibly think of. You're off to a great start.
BTW, here's the test data that the API is currently searching:
http://goo.gl/HZptLAlso, regarding text matching queries, the API is implemented on
Google App Engine where text matching queries like "taeniur%" to find
"taeniura" are not currently supported. But that's coming soon!
http://www.google.com/events/io/2011/sessions/full-text-search.html
There must be workarounds - Google itself never responds with "sorry, you haven't exactly matched any full strings, so please get bent."
In any case, if this cannot be overcome I'd say it's overwhelming evidence of using the wrong tools. "It'll work later" is never a very comforting response.
So if Google Apps won't support the types of queries we need, and won't support pulling very large chunks of data into something that will support the types of queries we need.....
Honestly, the limits don't bother me too much - I'll just write a script to fire off a few thousand requests at a time until I get what I need and patch it all back together on my end. I also assume I'm not the target consumer, and that the target user won't have that capacity.
FYI, the average specimen data download in Arctos is 880 records.
> On that note, a less-verbose JSON return format would shrink return filesize
> by ~50% - maybe much moreWell, the JSON response format remains true to the underlying data.
It's basically just the verbatim row in publishers CSV file. I'm not
sure how we can make that less verbose? We do have some options for
optimizing JSON responses via binary using libraries like bson
(http://bsonspec.org) and MessagePack (http://msgpack.org), but I
don't really like that since a big win for JSON is human readability.
Thoughts?> given the prevalence of NULL in DWC data and the
> potential for very large recordsets. I'm not sure what the format change is
> called, but....
> [
> {georeferenceprotocol:[
>
>
> 'MaNIS/HerpNet/ORNIS Georeferencing Guidelines, GBIF Best Practices',
> 'MaNIS/HerpNet/ORNIS Georeferencing Guidelines, GBIF Best Practices',
>
>
> 'MaNIS/HerpNet/ORNIS Georeferencing Guidelines, GBIF Best Practices']
> },{stateprovince:[
>
>
> 'California',
> 'California','
>
>
> California']
> }
> ]In your example above, it's not clear why Darwin Core fields have a
list with repeated values?
It's just a less verbose JSON format - your returns include the headers for each row, mine includes headers only once. It's the same data and you can do the same things with it, mine's just ~50% smaller.
> And if anybody's interested, I threw together
> http://arctos.database.museum/CustomPages/vntest.cfm?q=california,aves&genus=limnodromus&county=alameda
> - it just makes the json a little more readable.Nice! But that looks like an HTML representation, not a JSON
representation, right? What am I missing?
John Wieczorek
Real curious what the distribution looks like. Seems that we should
keep in mind not the mean, but the majority, so something like the
upper limit for 80% of the queries, or 95%, or 99%.
Nelson, do you have access to similar statistics for the FishNet2 portal?
John Wieczorek
> I agree with Dusty's post. We might be barking up the wrong tree if
> searches and their results aren't intuitive and don't have the same feel of
> a typical search engine.
>
> Playing devil's advocate here:
> In the event, google doesn't support what we need, gets expensive, becomes
> evil, decides to dump AppEngine etc, how portable is what you guys are
> building? For example, could it be ported to a virtual server hosted via
> rackspace or amazon or even moved to a standalone server? My understanding
> is that it is difficult to move away from AppEngine. Is that true? Would
> only the data access code need to be re-written?
>
> John, in response to your other message. Yes we have logs we can use to
> compute average number of records downloaded.
> We publish a monly report of activity here:
> http://www.fishnet2.net/reports
> Note: Last month's report has not been posted yet. We will likely do that
> when the report for this month runs and post them both at the same time.
Interesting. If I made the right assumptions, it looks like the
*mean* size of a query result is 6079 records over the month of July.
That's way more than from Arctos, as you would expect based on the
total collections accessed. We don't have easy access to this
information in the other nets because the query statistics are kept by
the providers, not by the portals.
> On a related note: I wonder if there is too much emphasis on queries per
> second. None of our logs indicate that as a major issue. For example,
> FishNet2 might peak at about 300 complete queries per day with a more
> typical rate of 100-150 complete queries per day. What are you seeing in
> Vertnet, Ornis and Manis in terms af true queries?
A quick assessment for the last 6 months (for MaNIS, ORNIS, and
VertNet portals since HerpNet disappeared) shows underestimated
monthly query rates at between 400 and 800. They are underestimated
because they don't include queries that did not include Arctos as a
provider. So, ball-parking here, the mean current monthly query rate
across the vertebrate portals in about 1000 with results sizes
typically in the thousands.
That is the current state, but keep in mind that we're not building
for the status quo. Our goal is to make the data more usable and more
used. Our ideal situation is that people ask for more data than we
think we can possible give them, and then give it to them without
having to blink an eye. Keep in mind that we're opening up an API as
well, not just funneling all requests through a user interface. We
expect that the bulk of the use would come through the API.
>
> Nelson
>
> Nelson E. Rios
> Manager of Collections and Bioinformatics
> Tulane University Museum of Natural History
> Belle Chasse, LA 70037
> E-mail address: nel...@museum.tulane.edu
> Voice: 504-394-1711
> Fax: 504-394-5045
> FYI, the average specimen data download in Arctos is 880 records.
>
Aaron Steele
> haven't exactly matched any full strings, so please get bent."
> In any case, if this cannot be overcome I'd say it's overwhelming evidence
> of using the wrong tools. "It'll work later" is never a very comforting response.
An ongoing challenge with VertNet is finding the right balance between
scalability, sustainability, and functionality. For example, the DiGIR
architecture supports text matching functionality (e.g., searching
"taeniur%" to find "taeniura") but it requires $180k annually to to
maintain server infrastructure (sustainability problem), and it's
getting slower as it grows (scalability problem).
To support text matching functionality right now, we could totally
spin up and load balance multiple Solr servers
(http://lucene.apache.org/solr), and it would be amazing. It would
give us the functionality and scalability that we want, but the
trade-off would be sustainability since it would require long term
infrastructure management. It's all about trade-offs.
So right now our strategy for text matching functionality is to wait a
bit since it's an official feature on deck in the App Engine roadmap:
http://code.google.com/appengine/docs/roadmap.html
If that feature flops, we'll adapt to the situation. Our philosophy is
to keep agile so that we can change our architecture on the fly to
better hit that balance between scalability, sustainability, and
functionality. In the meantime, if you have ideas for text matching
that finds the right balance, we would love to hear about it.
> Honestly, the limits don't bother me too much - I'll just write a script to
> fire off a few thousand requests at a time until I get what I need and patch
> it all back together on my end. I also assume I'm not the target consumer,
> and that the target user won't have that capacity.
You're definitely one of our target consumers. When you say a consumer
wont have the capacity to page using limit/offset parameters, what do
you mean? Unless I'm missing something, if the consumer has capacity
to make a single HTTP request, then they will definitely have capacity
to do that more than once?
> FYI, the average specimen data download in Arctos is 880 records.
Good stat. Definitely interested in John's question about the distribution.
> It's just a less verbose JSON format - your returns include the headers for
> each row, mine includes headers only once. It's the same data and you can do
> the same things with it, mine's just ~50% smaller.
Clever. It's smaller, but the trade-off here is readability, which IMO
is a big win for JSON. Maybe?
Aaron Steele
> On Wed, Sep 14, 2011 at 10:27 AM, Nelson Rios <nel...@museum.tulane.edu> wrote:
> In the event, google doesn't support what we need, gets expensive, becomes
> evil, decides to dump AppEngine etc, how portable is what you guys are
> building? For example, could it be ported to a virtual server hosted via
> rackspace or amazon or even moved to a standalone server? My understanding
> is that it is difficult to move away from AppEngine. Is that true? Would
> only the data access code need to be re-written?
Well, the code is *sorta* portable. App Engine is compatible with the
WSGI standard for Python web application containers
(http://wsgi.org/wsgi/What_is_WSGI), so porting to other containers
like Pylons running on S3 or Linode would be pretty easy. On the other
hand, some of the code definitely depends on App Engine APIs (e.g.,
datastore, memcache, urlfetch). Where possible, we're intentionally
writing code without any App Engine dependencies. Where not possible,
we're using duck typing (http://en.wikipedia.org/wiki/Duck_typing) as
a workaround.
Dave Vieglais
I would also point out that even a single SOLR instance can handle a fairly significant load and volume of content. THis has been demonstrated in practice by a number of installations such as NASA's DAAC and the NBII clearing house which have also been quite successful and cost effective.
The main crux of the problem for vertnet really lies in the collation of content, and providing that capability is kept functionally distinct from the search and retrieval services, it should be relatively straight forward to investigate different search engines to determine the most cost effective long term solutions. This is especially important when considering the project reliance upon a commercial service which has already significantly altered its provision agreements since the project started. There are also significant rumblings within Google of tightening down on "research" topics that do not directly lead to revenue, Earth Engine being one example. It would be prudent to keep such considerations in mind and develop the infrastructure accordingly.
regards,
Dave V.
Aaron Steele
+1. IMO, I think effective indexing provides a better
search/cost/performance trade-off at scale when compared with partial
string matching.
> I would also point out that even a single SOLR instance can handle a fairly significant load and volume of content. THis has been demonstrated in practice by a number of installations such as NASA's DAAC and the NBII clearing house which have also been quite successful and cost effective.
Great point. On the other hand, where possible, we might want to focus
our limited developer resources on building software tools rather than
hardware infrastructure. Maybe?
> The main crux of the problem for vertnet really lies in the collation of content and providing that capability is kept functionally distinct from the search and retrieval services, it should be relatively straight forward to investigate different search engines to determine the most cost effective long term solutions.
When you say "collation of content" do you mean storage?
> This is especially important when considering the project reliance upon a commercial service which has already significantly altered its provision agreements since the project started.
Well, FWIW, there wasn't actually a provision agreement in place since
it was just a limited preview. Now that it's an official Google
service with an SLA and support, they have an official provision
agreement. But your point is well taken.
> There are also significant rumblings within Google of tightening down on "research" topics that do not directly lead to revenue, Earth Engine being one example. It would be prudent to keep such considerations in mind and develop the infrastructure accordingly.
+1
Dave Vieglais
On 15Sep, 2011, at 14:06 , Aaron Steele wrote:
…
>
>> The main crux of the problem for vertnet really lies in the collation of content and providing that capability is kept functionally distinct from the search and retrieval services, it should be relatively straight forward to investigate different search engines to determine the most cost effective long term solutions.
>
> When you say "collation of content" do you mean storage?
Implementing search is easy, almost trivial with all the great tools available now, once you have content in some well defined structure. My argument is that (at least from my perspective) a great contribution that can be made by the Vertnet project is a well engineered mechanism for placing all the vert content in a single (virtual or otherwise) location where it can be accessed easily.
So yes storage is what I mean by collation, but a little more than just that - storage in an orderly fashion which facilitates progressive refinement of the content or augmentation by other resources (the classic being combination of environmental attributes with specimens through a natural join on geo-temporal location).
Aaron Steele
+1
> So yes storage is what I mean by collation, but a little more than just that - storage in an orderly fashion which facilitates progressive refinement of the content or augmentation by other resources (the classic being combination of environmental attributes with specimens through a natural join on geo-temporal location).
Thanks for extra details on collation. Totally agree on all points.
Dusty
On Wednesday, September 14, 2011 4:09:31 PM UTC-5, Aaron Steele wrote:
> Honestly, the limits don't bother me too much - I'll just write a script to
> fire off a few thousand requests at a time until I get what I need and patch
> it all back together on my end. I also assume I'm not the target consumer,
> and that the target user won't have that capacity.You're definitely one of our target consumers. When you say a consumer
wont have the capacity to page using limit/offset parameters, what do
you mean? Unless I'm missing something, if the consumer has capacity
to make a single HTTP request, then they will definitely have capacity
to do that more than once?
I mean that with 100-record limits the "average" consumer would have to click the "next page" button 14,000 times just to get what's in Arctos - a tiny little slice of what will be in VertNet. Nobody will do that, ever. Some of us have the tools and knowledge to cheat, but that's beside the point.
> FYI, the average specimen data download in Arctos is 880 records.
Good stat. Definitely interested in John's question about the distribution.
> It's just a less verbose JSON format - your returns include the headers for
> each row, mine includes headers only once. It's the same data and you can do
> the same things with it, mine's just ~50% smaller.Clever. It's smaller, but the trade-off here is readability, which IMO
is a big win for JSON. Maybe?
Both are JSON. JSON isn't all that readable to the we mere mortals, BTW - that's why I threw that JSON-->table thing together.
Dusty
For July, the average (of 110,000 queries) query returned 150 records, with 8500 of those returning >100 records, 1125 over 1,000, and 144 over 10,000.
I suspect that part of the disparity between Arctos and FishNet recordcounts is attributable to the relative homogeneity of the data. All the guppies collected by Bob Jones is a straightforward query in Arctos - there's probably one taxon name, there are clear searchable synonomies if not, and you can filter on _the_ Bob Jones you're interested in. FishNet (and VertNet) are less unified data and more mashups, meaning that you'll have to search for every namestring that anyone MIGHT have called a guppy, and you'll have to search for everyone that MIGHT be your Bob Jones, (Jones, B. Jones, Robert Jones, Bob J. Jones, etc.) and then sort out the union of those to get what you want.
More is explained by visibility. FishNet simply has a larger audience than Arctos.
VertNet will greatly amplify both of those problems.
Aaron Steele
> click the "next page" button 14,000 times just to get what's in Arctos - a
> tiny little slice of what will be in VertNet. Nobody will do that, ever.
> Some of us have the tools and knowledge to cheat, but that's beside the
> point.
Yeah, but I wonder if the average consumer might just add more filters
to their search to reduce the number of results (e.g., "puma concolor
california 1920" instead of just "puma concolor")?
> Both are JSON. JSON isn't all that readable to the we mere mortals, BTW -
> that's why I threw that JSON-->table thing together.
Ha, good point about readability, but I was thinking more about
readability for *developers* who are definitely not mere mortals. For
users, yeah, HTML is definitely the way to go!
Carla Cicero
> I mean that with 100-record limits the "average" consumer would have toYeah, but I wonder if the average consumer might just add more filters
> click the "next page" button 14,000 times just to get what's in Arctos - a
> tiny little slice of what will be in VertNet. Nobody will do that, ever.
> Some of us have the tools and knowledge to cheat, but that's beside the
> point.
to their search to reduce the number of results (e.g., "puma concolor
california 1920" instead of just "puma concolor")?