Bigdata Wars

[Calmera]

Hello everyone,

It seems all great idea's (at least mine) manifest themselves in a bathroom, and so when showering yesterdag I came up with the following:

Wouldn't it be great to have something like a bigdata wars event. Ok before everyone starts thinking about tools of modern warfare, let me explain:

Several teams may participate in an event to decide which bigdata technology stack is the "best" (<- mind the quotes). Every team may choose their stack and try to build the best solution to the problem we will throw at them within a limited amount of time. The solutions will be judged by their accurancy , performance, stability, implementation speed, learning curve, ... all to end up with the "best solution".

The dataset will be provided by us, and should be well documented. Maybe we can invite a jury to judge the results?

What do you think? Would this be feasable? Any feedback is greatly appreciated.

Daan.

[Eric Charles]

A bit like topcoder's algorithm contests (http://community.topcoder.com/tc) ?

I expressed a similar idea during our last meetup, but more like a 1 day Hackathon to benefit from the .be locality.
The result wouldn't be a "best one", but more the identification of weakness (indexing and search, transaction, locking,...) and the found solutions to overcome them.

Will bigdata.be integrate some "hands-on", or will it be more "ideas/experience exchange/presentations"?

Thx,
Eric

[Daan Gerits]

Hey Eric,

Sorry for the very late reply. I certainly would like something like a hackathon to do some nifty stuff with bigdata and to learn from one another. We will address this in the meetup next week, make sure to remind me if I forget (putting a big cross on his hand).

At one side I would like to hear from the community how they are tackling problems with bigdata, how do they get their data into hadoop for example. Do they only use MapReduce for batch analysis or are there any people out there using some kind of realtime alternative? Do they export their data to a relational system or build their applications to connect to the bigdata solution?

On the other hand, I think we would all benefit from some hands-on experience so everyone can learn from eachother. I certainly want to have a look at organizing something like this, but I don't have a lot of experience with organizing hackathons, so any help is welcome.

As for the bigdata wars, The idea was to organize a multi-day event and attract teams to build a solution in a semi-competitive way. I think it will be very interesting to see what some people can come up with when handling loads of data.

Daan.

[Wim Van Leuven]

Hey Eric,

Of course bigdata.be can be leveraged for hands-on sessions and hackatons. I even dare to say that it must be leveraged for that purpose!!! 

So it is great to hear you would like to organize such an event. So send us your idea and topic for your hackaton and we can use meetup.com to schedule a get-together. We can make you the host so you can organize it how you'd like.

Looking forward to that!
-wim

[Nathan Bijnens]

Wikipedia announced the Wiki challenge:

Wikipedia's Participation Challenge

This competition challenges data-mining experts to build a predictive model that predicts the number of edits an editor will make in the five months after the end date of the training dataset. The dataset is randomly sampled from the English Wikipedia dataset from the period January 2001 - August 2010.

The objective of this competition is to quantitively understand what factors determine editing behavior. We hope to be able to answer questions, using these predictive models, why people stop editing or increase their pace of editing.

Contestants are expected to build a predictive model that can be reused by the Wikimedia Foundation to forecast long term trends in the number of edits that we can expect.

http://www.kaggle.com/c/wikichallenge

N.

---
nat...@nathan.gs : http://nathan.gs

http://twitter.com/nathan_gs : http://linkedin.com/in/nbijnens

[Eric Charles Gmail]

Hi Wim,

Thx for the suggestion and confidence, I can take this on :)

I'm leaving on 29 Aug. for 2 months to San Jose, CA, going after to ApacheCon NA 2011 in Vancouver.
I should be operational again in Belgium as from 15 Nov. with plenty of ideas I hope.

Let's add 'Hackaton' to the agenda of our next meetup on 24 Aug.

Thx.
- Eric