rebalance_table_shards is not working

[shankar]

Hi,

In my setup there is one master node, two worker nodes in RDS postgresql setup. I created two more RDS instances and added DNS names in pg_worker_list.conf on master node.

I want to re-balance the shards to new worker nodes.

I used the document in this page - https://www.citusdata.com/documentation/citusdb-documentation/admin_guide/scaling_out_your_cluster.html#adding-worker-node

After updating pg_worker_list.conf file i reloaded config file using -- select pg_reload_conf();

New worker nodes (total 4) are showing in the result of -- select * from master_get_active_worker_nodes();

Created extension -- CREATE EXTENSION shard_rebalancer;

When i use this function to re-balance shards to 4 worker nodes, i am getting error.

select rebalance_table_shards('github_events');

Error is :

CONTEXT:  SQL statement "SELECT update_shard_placement(placement_update_data, relation,

                                       responsive_node_list)"

PL/pgSQL function replicate_table_shards(regclass,integer,integer,bigint[]) line 60 at PERFORM

NOTICE:  Copying shard 103380 from worker1.t56ye4r5t6y7.us-east-1.rds.amazonaws.com:5432 to worker3.t56ye4r5t6y7.us-east-1.rds.amazonaws.com:5432 ...

WARNING:  could not receive query results from worker3.t56ye4r5t6y7.us-east-1.rds.amazonaws.com:5432

DETAIL:  Client error: function worker_fetch_regular_table(unknown, integer, unknown, unknown) does not exist

CONTEXT:  SQL statement "SELECT update_shard_placement(placement_update_data, relation,

                                       responsive_node_list)"

PL/pgSQL function replicate_table_shards(regclass,integer,integer,bigint[]) line 60 at PERFORM

WARNING:  could not copy 103380 from worker1.t56ye4r5t6y7.us-east-1.rds.amazonaws.com:5432 to worker3.t56ye4r5t6y7.us-east-1.rds.amazonaws.com:5432

CONTEXT:  SQL statement "SELECT update_shard_placement(placement_update_data, relation,

                                       responsive_node_list)"

PL/pgSQL function replicate_table_shards(regclass,integer,integer,bigint[]) line 60 at PERFORM

NOTICE:  Copying shard 103381 from worker2.t56ye4r5t6y7.us-east-1.rds.amazonaws.com:5432 to worker4.t56ye4r5t6y7.us-east-1.rds.amazonaws.com:5432 ...

WARNING:  could not receive query results from worker4.t56ye4r5t6y7.us-east-1.rds.amazonaws.com:5432

DETAIL:  Client error: function worker_fetch_regular_table(unknown, integer, unknown, unknown) does not exist

CONTEXT:  SQL statement "SELECT update_shard_placement(placement_update_data, relation,

                                       responsive_node_list)"

PL/pgSQL function replicate_table_shards(regclass,integer,integer,bigint[]) line 60 at PERFORM

WARNING:  could not copy 103381 from worker2.t56ye4r5t6y7.us-east-1.rds.amazonaws.com:5432 to worker4.t56ye4r5t6y7.us-east-1.rds.amazonaws.com:5432

When i check the table structure in first two existing worker nodes, it looks like this.

           List of schemas

           Name            |  Owner   

---------------------------+----------

 pgs_distribution_metadata | postgres

 public                    | postgres

(2 rows)

When i check the table structure in new worker nodes, it looks like this.

           List of schemas

           Name            |  Owner   

---------------------------+----------

 public                    | postgres

(2 rows)

Do we need to perform any other step to work the rebalance on shard. I set up the cluster using these instructions in citusdata installation.

https://www.citusdata.com/documentation/citusdb-documentation/installation/multi_node_deb.html#multi-node-deb

Appreciate you help on this.

Thanks

Shankar.

[shankar]

The function worker_fetch_regular_table is not available in master node functions list.

[Onder Kalaci]

Hi shankar,

Shard rebalancer extension is required on both master node and the workers nodes. And, I don't think that you can install&create external extensions on RDS, such as shard rebalancer.

[shankar]

Thank you Onder. I will try with all the nodes with EC2 instances

Thanks

Shankar

On Monday, November 23, 2015 at 6:26:32 PM UTC+5:30, shankar wrote:

[shankar]

Hi Onder,

I created distributed table with replication factor 2 as given below.

SELECT master_create_worker_shards('github_events', 16, 2);

Then i populated table with some data. After some days i want to increase the replication factor to 3.

Can we achieve this using the UDF replicate_table_shards as given in the page 

https://www.citusdata.com/documentation/citusdb-documentation/reference/user_defined_functions.html#replicate_table_shards ?

Do we need shard rebalance extension to be installed on master and worker nodes to use replicate_table_shards UDF ?

Thanks

Shankar

On Monday, November 23, 2015 at 6:26:32 PM UTC+5:30, shankar wrote:

[shankar]

If i change the configuration parameter  shard_replication_factor (integer) in master node as given in https://www.citusdata.com/documentation/citusdb-documentation/reference/configuration.html#Data Loading will it affect all the tables?

If i want to do it for only one particular distributed table how can i do this?

Thanks

Shankar

On Monday, November 23, 2015 at 6:26:32 PM UTC+5:30, shankar wrote:

[Onder Kalaci]

Hi,

Are you creating a hash-partitioned or range-partitioned distributed tables? I am asking this because, for hash-partitioned tables, the replication factor is determined on the "master_create_worker_shards()" as discussed in the README on github.

shard_replication_factor(integer) parameter is effective for range partitioned tables. Whenever you create a shard via staging data, CitusDB creates  shard_replication_factor number of placements for that shard. So, you can change the parameter whenever you want to create different number of shards for different tables.

Is this useful information for you?

Thanks,

Onder

[shankar]

Hi,

I created hash-partitioned table and replication factor determined as 2 with "master_create_worker_shards()" function.

If i want to increase the replication factor, like 3. How can we do this?

Thanks

Shankar

On Monday, November 23, 2015 at 6:26:32 PM UTC+5:30, shankar wrote:

[Onder Kalaci]

Hi shankar,

You can do it easily with shard rebalancer. Assuming that you create the rebalancer extension on all nodes:

SELECT replicate_table_shards('dist_table', shard_replication_factor:=3);

Thanks,

Onder

[shankar]

Thank you Onder.

Regards

Shankar

On Monday, November 23, 2015 at 6:26:32 PM UTC+5:30, shankar wrote:

[shankar]

Hi Onder,

I setup cluster with all EC2 instances. 1 Master and 3 worker nodes and all are installed with pg_shard and shard_rebalancer extenstion.

I stopped 3rd worker node and removed the node entry in pg_worker_list.conf file and reloaded the config using SELECT pg_reload_conf(); 

when i checked with UDF select * from master_get_active_worker_nodes(); it shows only first two worker nodes.

I ran SELECT rebalance_table_shards('github_events'); to rebalance the shards among two active worker nodes (1st and 2nd) .

Result given below:

 rebalance_table_shards 

------------------------

 

(1 row)

But i dont see the shards are rebalancing. Please help me on this.

Thanks

Shankar

On Monday, November 23, 2015 at 6:26:32 PM UTC+5:30, shankar wrote: