FW: ASF Foresight Speaker Database - Tech Question

[Bino Gopal]

Hey guys, some of you may remember John Smart from the LA Future Salons (which he founded in 2001 and ran until 2007 when he moved to the Bay Area).  He has a technical DB question here and I figured I’d turn to the experts at BCLA for some input…you can send any replies directly to me or cc: the list too; just looking to gather some responses and give John some options/input; thanks so much for any help!!

 

                                                                                                                BINO

 

 

From: John Smart [mailto:john...@gmail.com]
Sent: Friday, December 11, 2009 12:08 PM
To: Bino Gopal
Subject: ASF Foresight Speaker Database - Tech Question

 

Can I ask a Technical Question?

Our nonprofit, ASF, needs to do develop an online database for foresight professionals, folks who like to speak about or research the future of sci, tech, business, and social change. This is a resource that our Salon leaders can use to find or add local potential speakers, as well as a general public resource (a free foresight speaker database) people can blunder into and add themselves to.

We are expecting this database to get up to maybe 100,000 records eventually (and take a long time getting there). It will have 10 required and 23 more optional fields per speaker. We want it online and mostly open access, available to anyone to search and add to, on GlobalForesight.org. Anyone should be able to update their own record, and decide whether they want their email accessible on it. Part of the database (private email address, personal notes) should be viewable only by Salon leaders.

We would like it to be as quick as possible to serve records, able to serve them in a full page format (required info on top, optional available on scrolldown) and also single line format for multiple record returns. It should be as cheap as possible to run and create.

I am wondering if doing it in the cloud, using Amazon's Web Services (perhaps SimpleDB, or RDS, http://aws.amazon.com/windows/), would be best for long term operation, scaling, etc. Could I do it there for an affordable monthly rate? Or should we ask our web hosting company (which gives us a very low hosting rate) do to it in something like Drupal+SQL? Would that scale to a 100,000 records and still be fast?

Sorry to be so detailed but I know you techies like specs well defined :)

Hope you are all having a great end of the year. 2010's here we come!!!

Cheers,

--
John M. Smart,
____________________________________
Co-Founder, Evo Devo Universe Community
www.evodevouniverse.com
President, Acceleration Studies Foundation
216 Mtn View Ave, Mtn View, CA 94041
310 831-4191 | www.accelerating.org

[Dan Tentler]

Hi Bino, John,

  100,000 records is not all that much. If you want absurdly fast results, consider using memcached - which takes all of those records and makes them available using RAM instead of disk.

I'd say start with 2 servers, side by side in a rack somewhere. One with apache and mysql, the other with very little disk space and as much ram as you can possibly shove in it. The first server is the one with all the actual data on it, the second is purely a memcached server - which does nothing but serve data out of ram.

This can be done with whatever front end you choose, and mysql as a back end. I know for a fact wordpress has a memcached plugin you can simply download and it should 'just work' and I presume that drupal has the same thing - I'd be surprised if it didn't.

I'm not super-familiar with ec2, but its definitely a possibility. If this is going to be a long term solution though, you're probably going to want a dedicated host - all of the cloud-based solutions charge you on a per-cpu-cycle basis, so if your tool becomes REALLY popular it's going to be more expensive to use cloud than it would be to have dedicated hardware.

Just my 2c!

-Dan
atenlabs.com

--
BarCampLA Wiki: http://barcamp.org/BarCampLosAngeles
BarCampLA Blog: http://www.barcampla.org/
BarCampLA Group: http://groups.google.com/group/BarcampLA?hl=en

[Darius Clarke]

Hi Bino & John,

You might try DabbleDB http://dabbledb.com/ to get started quickly. It's hosted, but is extremely flexible for adapting to continious changing data structures by the users of their service. It's easy to slice and dice the data for analysis (like pivot tables). It has innovation and adaptability in its soul, so it might be a good fit for Global Foresight at a philosophical level. So, if Global Foresight's clients have data they want to integrate together with Global Foresight and each other, your database can evolve in real time to accommodate that, including geolocation map data. 

Case Study: How the Battery Hen Welfare Trust is Using Dabble DB

http://blog.dabbledb.com/2009/12/as-a-customer-for-almost-two-years-the-battery-hen-welfare-trust-bhwt-is-a-great-example-of-how-people-are-using-dabble-db.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+DabbleDB+(Dabble+DB+Blog)

Battery Hen Welfare Trust in the UK use it and are happy with it. It seems they keep track of over 180,000 hens and their new homes. The only thing I'm not sure about is maintaining User ID's and pw. A shell application might be able to handle that, only giving access to one filtered view of the data in a form. 

Most of the data is cached in memory, I believe. It's free to try for a month. 

Cheers,

Darius Clarke

metaverseograher.com

__________________________

--

[rick tait]

+1. Dan nailed it. 100,000 is nothing in the grand scheme of things - especially for the type of niche traffic you’re talking about, and even if you *do* end up getting crazy traffic out of the blue, Dan’s addition of memcached is adding you a nice “holdover” layer. The only thing I might add is that if you start getting*really* crazy amounts of (read-only, not read-write) traffic, stick a couple of reverse proxying httpd’s (nginx anyone?) in front to offload some of the more popular/smaller http requests to let your webserver handle more of the big ones.

 

This could all be absolutely done on EC2, but it won’t be cheap, and since you have an ISP hook-up it doesn’t make good strategic sense for you to lose that while adding more recurring (and increasing!) costs from AMZN.

 

Stick with your ISP, get a good PHP dev/systems guy and you’ll be golden.

 

Hope this helps,

RMT.