What's Foursquare built on?

472 views
Skip to first unread message

David Veksler

unread,
Jan 8, 2013, 5:23:56 AM1/8/13
to Mises.org Development

I'm kind of a Foursquare addict. Here is their about page talking about how it's built. I'm NOT suggesting that Mises.org do this because Foursquare is solving different kinds of problems - but it's interesting.  Some of these technologies are a good idea for any large-ish site.  

---

I'm really into technology. What's Foursquare built on?

Because of our pretty insane growth, we’ve designed our entire software stack to be as flexible and scalable as possible. It’s allowed us to go from a few thousand users to over fifteen million, and also to easily add more amazing engineers and get them contributing right away.

Foursquare is currently hosted within Amazon's EC2 service, using hundreds of servers running a bare bones version of CentOS Linux. We use NGINX to route requests and serve static content, andHAProxy to load balance web and API requests across many machines.

Then we get to the fun part. Moving up the stack, the live site data is stored in MongoDB (though we use Memcache to cache a small set of expensive calculations). For offline data analysis we regularly snapshot our live data and import it into a Hadoop cluster. We have some custom MapReduce jobs, but mostly rely on Hive's simple query syntax and a custom built job scheduler for regular calculations. We use Solr and Elasticsearch for powering venues, tips, users, and events search. Our search geo-indexing uses Google's s2 library to store cellids within our search index. We use PostGISand the wonderful geonames.org dataset to reverse geocode addresses into coordinates, which allows us to place venues on a map and make them available for location-based search. Kestrel is our queue for asynchronous tasks that we wish to perform out of band of users' requests. User generated photos are stored on Amazon S3 with content delivery through Akamai. There is a bit more complexity if you dig deeper, but that’s the heart of it.

Almost all of the code for the web site, API, and batch processing is written in Scala. The web and API are built on top of the Lift web framework. We also use a good bit of Python and Bash scripting for automating build, deployment, and operations tasks. Finally, the dynamic content on the web site is written in javascript with a mix of jQueryBackbone.js for object models, and Soy for templating.

We use beautiful maps by MapBox created using data provided by the wonderful © OpenStreetMapand contributors, and licensed Open Data Commons Open Database License. The interactive maps are generated using the open-source library Leaflet.

Chris Folsom

unread,
Jan 8, 2013, 9:22:06 AM1/8/13
to mise...@googlegroups.com



I thought you didn't like Java.  Are you still planning on moving Mises.org off of .NET?

David Veksler

unread,
Jan 8, 2013, 9:30:35 AM1/8/13
to mise...@googlegroups.com
Distributed rich-client no-SQL architecture (is that a thing?) is about moving away from server-generated UI, so it is more platform agnostic.

I already use some of these technologies (memcached) with .Net.

Scala runs on .Net though: http://www.scala-lang.org/node/10299

---
Regards,
David Veksler
+86 186 168 73407

Chris Folsom

unread,
Jan 8, 2013, 10:29:41 AM1/8/13
to mise...@googlegroups.com


Interesting. I didn't know that Scala was ported to .NET. I never thought that No-SQL was about moving away from traditional web frameworks. I've spent quite a bit of time working with Hadoop and Hive and the reason is because they are the only things that can solve the types of problems that they are addressing. I don't think I would choose mongo for the backend of anything just do to the performance problems I have seen with it.


To me, the mises.org site seems pretty good as it is and if I had any time to contribute, it would be around SEO. After all, getting the message out is what this about isn't it?
Reply all
Reply to author
Forward
0 new messages