Scaling an Ensemble

[Rohit Manohar]

Hi

How do you scale a currently running ensemble from N clusters to N+1
cluster without losing data?

I had a ensemble running of 2 clusters (each with 1 master and 1
server)
I tried the following, but could not retain all the data (I had just 1
bucket):

1. Export backup of the bucket of both the servers.
2. Restart the servers using the update cluster.json files. So now I
have a ensemble of 3 cluster (the third cluster is also 1 master and 1
server)
3. Import the backup on both the servers.
4. When I try to GET all data, it shows 404 for some of the records.

I even tried cleaning the masters before joining the servers in the
new ensemble, but faced the same issue.

Is there a way to scale ensembles?

Regards,

[Sergio Bossa]

On Wed, Jun 8, 2011 at 8:52 PM, Rohit Manohar <roh...@gmail.com> wrote:

> I had a ensemble running of 2 clusters (each with 1 master and 1
> server)
> I tried the following, but could not retain all the data (I had just 1
> bucket):

I just noticed there is a bug with the current backup implementation,
which doesn't correctly deals with ensembles: do you mind to open an
issue about it?
I'll fix it ASAP for 0.8.2.

Thanks,
Cheers,

Sergio B.

--
Sergio Bossa
http://www.linkedin.com/in/sergiob

[Rohit Manohar]

Hi Sergio,

I am really looking forward for 0.8.2. Terrastore shows a lot of
promise to evolve into a really nice distributed solution.

Isn't the only way to fix this bug is perform re-partitioning on the
import of backup data, so that even the new cluster gets its share?.

Now if such an operation could be supported during normal operations,
it could support the case for dynamic scaling up of the ensemble.
Sure, it would incur an substantial overhead, but dynamic scaling up
would be a really cool feature.

Regards,
Rohit

On Jun 9, 1:57 am, Sergio Bossa <sergio.bo...@gmail.com> wrote:

[Sergio Bossa]

On Fri, Jun 10, 2011 at 7:53 PM, Rohit Manohar <roh...@gmail.com> wrote:

> I am really looking forward for 0.8.2. Terrastore shows a lot of
> promise to evolve into a really nice distributed solution.

Thanks for your kind words!
Unfortunately, Terrastore has no support from external "vendors", and
still very little community, so it is pretty difficult to make it
evolve quickly.

> Isn't the only way to fix this bug is perform re-partitioning on the
> import of backup data, so that even the new cluster gets its share?.

Exactly, that's the idea.

> Now if such an operation could be supported during normal operations,
> it could support the case for dynamic scaling up of the ensemble.
> Sure, it would incur an substantial overhead, but dynamic scaling up
> would be a really cool feature.

That's something we discussed about with other committers too: we had
some plans to implement dynamic ensemble scaling in the future, but it
requires lots of time and work, so we have no real schedule right now
:(