Concept of partition

[Matt]

Hi,

Could someone explain to me the concept of partition a little bit more
please.

For example,
How should I choose the value for partition.count ?
What happen when the last partition is full ?
Should I have many small partitions of just 2-3 big ones ?
If I have many partition, is the data written on the first one until
it's full and then the second one and so on ?
Should I use the same start and count for the element store and the
connection store ?
Is it possible to monitor the partition to know how many entries there
is on each partition ?

Thanks
Mathieu

[Matt]

Anyone ?

[Jingwei]

Hi Matt,

Here is what I think.

Cleo makes a simple assumption that elements have unique IDs (i.e.
integer numbers starting from 0). This assumption applies in many
cases such as users, companies etc.

> > How should I choose the value for partition.count ?

The partition.count and start specify range-based partition, which
matches well with users/members. As new members join your site, you
create a new partition.

We typically have range partition of size 1,000,000 and colocate
multiple partitions within one instance for performance.

> > What happen when the last partition is full ?

You need to have logic/code in place to allocate a new partition. Each
partition only handles updates in the related range.

> > Should I have many small partitions of just 2-3 big ones ?

You could, but your routing logic will be more complicated. We have
partitions of the same size.

> > If I have many partition, is the data written on the first one until
> > it's full and then the second one and so on ?

No. The data goes to its owning partition directly. Say if you have an
update for company 1,000,005 and the update should go to its owning
partition (start = 1,000,000 and count = 1,000,000).

> > Should I use the same start and count for the element store and the
> > connection store ?

You should have something like the following assuming that you have
partitions of size 1,000,000.

partition.start partition.count
connectionsStoreCapacity
0 1000000 1000000
1000000 1000000 1000000
2000000 1000000 1000000
....

> > Is it possible to monitor the partition to know how many entries there
> > is on each partition ?

The cleo package does not provide any monitoring support.