Concrete Example using swarm

[Daniel Garcia]

Ian,

The concept behind this project is fascinating. I'm trying to wrap my
head around this problem.

1. Can you give a concrete example of an algorithm or problem that
would benefit from using swarm?

2. In determining what data should be moved, would the swarm
optimization happen at runtime, compile time, or both?


Regards,
Daniel

[Ian Clarke]

On Mon, Oct 12, 2009 at 2:41 PM, Daniel Garcia <dan...@danielgarcia.info> wrote:

1. Can you give a concrete example of an algorithm or problem that
would benefit from using swarm?

Hmm, good question.  I'll give an example that was one of the inspirations for Swarm in the first place:

Let's say you run a dating website, and you have millions of users.  You have an algorithm that can take data about two users, and predict how well they will match.

The problem is that there is too much data to keep it all on one computer, and you need to make predictions very quickly, so you can't be retrieving data from other computers every time you make a prediction.

With Swarm, the data would be distributed across multiple computers automatically.  If you need a prediction for a user or users with data on a remote computer, then Swarm will transparently transfer the continuation to the remote computers.

But clearly this would be very inefficient if it happened for every prediction, so the Swarm loadbalancer would  identify which users tend to get tested against each-other frequently (perhaps because they are in a similar geographic area), and automatically try to ensure that their data is stored on the same computer.

Through this mechanism, the dating website's data is automatically and intelligently clustered so that it can be distributed across multiple computers, without an excessive communication overhead.

Does that make sense?

2. In determining what data should be moved, would the swarm
optimization happen at runtime, compile time, or both?

Runtime.  It would be a background supervisor process similar to "just in time" compilation or garbage collection.

Ian.

--
Ian Clarke
CEO, Uprizer Labs
Email: i...@uprizer.com
Ph: +1 512 422 3588
Fax: +1 512 276 6674

[Patrick Wright]

An interesting area of related work is Project Darkstar, which is a
Sun/open source project to build a platform for large online games.
One of the problems they are trying to solve is to collocate data
within the server cluster according to usage patterns. Jim Waldo's
blog is great to follow; here's one piece (a worthwhile read) about
the collocation problem
http://blogs.sun.com/scalinggames/entry/miracle_1

Patrick

[Rick Richardson]

Wow. Now I'm really not going to get any work done today.