Re: [grinnellplans-dev] Digest for grinnellplans-development@googlegroups.com - 2 Messages in 1 Topic

[CM Lubinski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've had good experience with Solr. Easy to set up and maintain, though
you do need a servlet container like Tomcat or Jetty.

CM Lubinski

On Thu, 19 Jul 2012 04:20:29 +0000
grinnellplan...@googlegroups.com wrote:

> =============================================================================
> Today's Topic Summary
> =============================================================================
>
> Group: grinnellplan...@googlegroups.com
> Url: http://groups.google.com/group/grinnellplans-development/topics
>
> - Quick question about Plans-on-Rails database [2 Updates]
> http://groups.google.com/group/grinnellplans-development/t/7cb9edd9043a6bc6
>
>
> =============================================================================
> Topic: Quick question about Plans-on-Rails database
> Url:
> http://groups.google.com/group/grinnellplans-development/t/7cb9edd9043a6bc6
> =============================================================================
>
> ---------- 1 of 2 ----------
> From: Shitanshu Aggarwal <saggar...@gmail.com>
> Date: Jul 18 02:54AM -0700
> Url:
> http://groups.google.com/group/grinnellplans-development/msg/a95390d8e31f99a3
>
> I cannot imagine a situation where we would fully need to move away
> from a relational database. A relational schema should be able to
> support at least twenty more generations of grinnellians without too
> much trouble; we might need to move to a small host in a couple years
> and a large host in 15 years. What do you think?
>
> A case could be made for using a data store that specializes in
> search by way of MapReduce. However, this is an expensive solution
> because of the initial engineering work and the operational overhead
> of managing multiple packages later. I think we might be able to fix
> the problem by optimizing our MySQL configuration more, and looking
> into building an inverted index for quicklove. Since RDS does MySQL
> 5.1 and above only, we have stored procedures. So something like (
> http://code.google.com/p/inverted-index/wiki/Introduction) could be a
> fun experiment.
>
> Do we document the specifications of our production fleet somewhere?
> -- I'm specifically interested in the database.
>
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAlAHk44ACgkQfzi1OiZiJLAunACfSGXqtUIl7xlQupr0vBVijQbt
NWgAn1h1JsidAtBDRScrxeiacdm+AWFB
=Yssh
-----END PGP SIGNATURE-----

[Shitanshu Aggarwal]

I briefly looked at the stored procedure implementation of inverted indices on MySQL, and it doesn't look pretty.
One concern with using anything outside our database is consistency. How do we push data to Solr? What is the intended deployment configuration; is it single instance or multi-instance? I'm assuming we want to minimize the number of machines, which makes the MapReduce capable data stores less useful.

Shitanshu

--
You received this message because you are subscribed to the Google Groups "GrinnellPlans Development" group.
To post to this group, send email to grinnellplan...@googlegroups.com.
To unsubscribe from this group, send email to grinnellplans-deve...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/grinnellplans-development?hl=en.

--
Shitanshu Aggarwal

[[michaelh]]

When it comes to Ruby and Solr I've had good results with Sunspot - http://sunspot.github.com/ .  It is not without it's quirks and we've altered it a little bit for our objects, but overall it makes it easy to configure and push data.  It does come with some very basic solr server management and support which is great for development mode.  It's what we're currently using for and according to the admin pages it's indexed 418,457 documents w/o any trouble.  With sunspot the concept is that you'd configure the searching behavior on each of the Ruby classes and it provides after save/destroy hooks to push the changes up to Solr.  There are also rake tasks to wipe and re-index all documents. 

Thinking Sphinx from http://freelancing-god.github.com/ts/en/ operates similar, but not being web based it uses some smarts to get the config you specify in the ruby classes and write out a series of select statements for building the sphinx configuration.  One problem we had with it was keeping changes current, there is the concept of a core index and a delta index for any changes, but managing those indexes and changes was more than we wanted to do.

On Thursday, July 19, 2012 3:42:24 AM UTC-4, [aggarwal] wrote:

I briefly looked at the stored procedure implementation of inverted indices on MySQL, and it doesn't look pretty.
One concern with using anything outside our database is consistency. How do we push data to Solr? What is the intended deployment configuration; is it single instance or multi-instance? I'm assuming we want to minimize the number of machines, which makes the MapReduce capable data stores less useful.

Shitanshu

On Wed, Jul 18, 2012 at 9:56 PM, CM Lubinski <cmc3...@gmail.com> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've had good experience with Solr. Easy to set up and maintain, though
you do need a servlet container like Tomcat or Jetty.

CM Lubinski

On Thu, 19 Jul 2012 04:20:29 +0000
grinnellplans-development@googlegroups.com wrote:

> =============================================================================
> Today's Topic Summary
> =============================================================================
>

> Group: grinnellplans-development@googlegroups.com

To post to this group, send email to grinnellplans-development@googlegroups.com.
To unsubscribe from this group, send email to grinnellplans-development+unsub...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/grinnellplans-development?hl=en.

--
Shitanshu Aggarwal