Joining two distributed tables

112 views
Skip to first unread message

maxim...@gmail.com

unread,
May 21, 2016, 9:00:26 PM5/21/16
to citus-users
Hello.

I have tables t_user and t_address. Each of them is hash-distributed into 16 shards between two worker nodes of my AWS EC2 cluster.
I try to execute join-query, and here is what I got:
postgres=# begin;
set client_min_messages to debug4;
set citus.large_table_shard_count to 2;
set citus.task_executor_type TO 'task-tracker';
alter sequence pg_catalog.pg_dist_jobid_seq restart 1250;
select * from t_user tusr join t_address tadr on tusr.user_address_id = tadr.address_id;
end;


BEGIN
Time: 0.100 ms
DEBUG:  CommitTransactionCommand
SET
Time: 0.101 ms
DEBUG:  StartTransactionCommand
DEBUG:  ProcessUtility
DEBUG:  CommitTransactionCommand
SET
Time: 0.093 ms
DEBUG:  StartTransactionCommand
DEBUG:  ProcessUtility
DEBUG:  CommitTransactionCommand
SET
Time: 0.084 ms
DEBUG:  StartTransactionCommand
DEBUG:  ProcessUtility
DEBUG:  EventTriggerInvoke 16388
DEBUG:  CommitTransactionCommand
ALTER SEQUENCE
Time: 0.129 ms
DEBUG:  StartTransactionCommand
DEBUG:  generated sql query for job 1250 and task 2
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102680 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 4
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102681 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 6
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102682 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 8
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102683 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 10
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102684 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 12
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102685 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 14
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102686 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 16
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102687 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 18
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102688 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 20
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102689 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 22
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102690 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 24
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102691 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 26
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102692 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 28
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102693 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 30
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102694 tusr WHERE true"
DEBUG:  generated sql query for job 1250 and task 32
DETAIL:  query string: "SELECT user_id, user_addr, user_nickname, user_billing, user_address_id FROM t_user_102695 tusr WHERE true"
DEBUG:  assigned task 2 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 4 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 6 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 8 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 10 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 12 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 14 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 16 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 18 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 20 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 22 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 24 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 26 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 28 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 30 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 32 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  generated sql query for job 1251 and task 2
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102648 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 4
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102649 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 6
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102650 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 8
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102651 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 10
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102652 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 12
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102653 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 14
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102654 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 16
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102655 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 18
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102656 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 20
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102657 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 22
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102658 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 24
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102659 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 26
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102660 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 28
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102661 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 30
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102662 tadr WHERE true"
DEBUG:  generated sql query for job 1251 and task 32
DETAIL:  query string: "SELECT address_id, full_address, first, second, zip, city, country_code FROM t_address_102663 tadr WHERE true"
DEBUG:  assigned task 2 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 4 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 6 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 8 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 10 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 12 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 14 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 16 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 18 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 20 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 22 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 24 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 26 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 28 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 30 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 32 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  join prunable for task partitionId 0 and 1
DEBUG:  join prunable for task partitionId 0 and 2
DEBUG:  join prunable for task partitionId 0 and 3
DEBUG:  join prunable for task partitionId 0 and 4
DEBUG:  join prunable for task partitionId 0 and 5
DEBUG:  join prunable for task partitionId 0 and 6
DEBUG:  join prunable for task partitionId 0 and 7
DEBUG:  join prunable for task partitionId 1 and 0
DEBUG:  join prunable for task partitionId 1 and 2
DEBUG:  join prunable for task partitionId 1 and 3
DEBUG:  join prunable for task partitionId 1 and 4
DEBUG:  join prunable for task partitionId 1 and 5
DEBUG:  join prunable for task partitionId 1 and 6
DEBUG:  join prunable for task partitionId 1 and 7
DEBUG:  join prunable for task partitionId 2 and 0
DEBUG:  join prunable for task partitionId 2 and 1
DEBUG:  join prunable for task partitionId 2 and 3
DEBUG:  join prunable for task partitionId 2 and 4
DEBUG:  join prunable for task partitionId 2 and 5
DEBUG:  join prunable for task partitionId 2 and 6
DEBUG:  join prunable for task partitionId 2 and 7
DEBUG:  join prunable for task partitionId 3 and 0
DEBUG:  join prunable for task partitionId 3 and 1
DEBUG:  join prunable for task partitionId 3 and 2
DEBUG:  join prunable for task partitionId 3 and 4
DEBUG:  join prunable for task partitionId 3 and 5
DEBUG:  join prunable for task partitionId 3 and 6
DEBUG:  join prunable for task partitionId 3 and 7
DEBUG:  join prunable for task partitionId 4 and 0
DEBUG:  join prunable for task partitionId 4 and 1
DEBUG:  join prunable for task partitionId 4 and 2
DEBUG:  join prunable for task partitionId 4 and 3
DEBUG:  join prunable for task partitionId 4 and 5
DEBUG:  join prunable for task partitionId 4 and 6
DEBUG:  join prunable for task partitionId 4 and 7
DEBUG:  join prunable for task partitionId 5 and 0
DEBUG:  join prunable for task partitionId 5 and 1
DEBUG:  join prunable for task partitionId 5 and 2
DEBUG:  join prunable for task partitionId 5 and 3
DEBUG:  join prunable for task partitionId 5 and 4
DEBUG:  join prunable for task partitionId 5 and 6
DEBUG:  join prunable for task partitionId 5 and 7
DEBUG:  join prunable for task partitionId 6 and 0
DEBUG:  join prunable for task partitionId 6 and 1
DEBUG:  join prunable for task partitionId 6 and 2
DEBUG:  join prunable for task partitionId 6 and 3
DEBUG:  join prunable for task partitionId 6 and 4
DEBUG:  join prunable for task partitionId 6 and 5
DEBUG:  join prunable for task partitionId 6 and 7
DEBUG:  join prunable for task partitionId 7 and 0
DEBUG:  join prunable for task partitionId 7 and 1
DEBUG:  join prunable for task partitionId 7 and 2
DEBUG:  join prunable for task partitionId 7 and 3
DEBUG:  join prunable for task partitionId 7 and 4
DEBUG:  join prunable for task partitionId 7 and 5
DEBUG:  join prunable for task partitionId 7 and 6
DEBUG:  generated sql query for job 1252 and task 3
DETAIL:  query string: "SELECT "pg_merge_job_1250.task_000033".intermediate_column_1250_0 AS user_id, "pg_merge_job_1250.task_000033".intermediate_column_1250_1 AS user_addr, "pg_merge_job_1250.task_000033".intermediate_column_1250_2 AS user_nickname, "pg_merge_job_1250.task_000033".intermediate_column_1250_3 AS user_billing, "pg_merge_job_1250.task_000033".intermediate_column_1250_4 AS user_address_id, "pg_merge_job_1251.task_000033".intermediate_column_1251_0 AS address_id, "pg_merge_job_1251.task_000033".intermediate_column_1251_1 AS full_address, "pg_merge_job_1251.task_000033".intermediate_column_1251_2 AS first, "pg_merge_job_1251.task_000033".intermediate_column_1251_3 AS second, "pg_merge_job_1251.task_000033".intermediate_column_1251_4 AS zip, "pg_merge_job_1251.task_000033".intermediate_column_1251_5 AS city, "pg_merge_job_1251.task_000033".intermediate_column_1251_6 AS country_code FROM (pg_merge_job_1250.task_000033 "pg_merge_job_1250.task_000033" JOIN pg_merge_job_1251.task_000033 "pg_merge_job_1251.task_000033" ON (("pg_merge_job_1250.task_000033".intermediate_column_1250_4 = "pg_merge_job_1251.task_000033".intermediate_column_1251_0))) WHERE true"
DEBUG:  generated sql query for job 1252 and task 6
DETAIL:  query string: "SELECT "pg_merge_job_1250.task_000050".intermediate_column_1250_0 AS user_id, "pg_merge_job_1250.task_000050".intermediate_column_1250_1 AS user_addr, "pg_merge_job_1250.task_000050".intermediate_column_1250_2 AS user_nickname, "pg_merge_job_1250.task_000050".intermediate_column_1250_3 AS user_billing, "pg_merge_job_1250.task_000050".intermediate_column_1250_4 AS user_address_id, "pg_merge_job_1251.task_000050".intermediate_column_1251_0 AS address_id, "pg_merge_job_1251.task_000050".intermediate_column_1251_1 AS full_address, "pg_merge_job_1251.task_000050".intermediate_column_1251_2 AS first, "pg_merge_job_1251.task_000050".intermediate_column_1251_3 AS second, "pg_merge_job_1251.task_000050".intermediate_column_1251_4 AS zip, "pg_merge_job_1251.task_000050".intermediate_column_1251_5 AS city, "pg_merge_job_1251.task_000050".intermediate_column_1251_6 AS country_code FROM (pg_merge_job_1250.task_000050 "pg_merge_job_1250.task_000050" JOIN pg_merge_job_1251.task_000050 "pg_merge_job_1251.task_000050" ON (("pg_merge_job_1250.task_000050".intermediate_column_1250_4 = "pg_merge_job_1251.task_000050".intermediate_column_1251_0))) WHERE true"
DEBUG:  generated sql query for job 1252 and task 9
DETAIL:  query string: "SELECT "pg_merge_job_1250.task_000067".intermediate_column_1250_0 AS user_id, "pg_merge_job_1250.task_000067".intermediate_column_1250_1 AS user_addr, "pg_merge_job_1250.task_000067".intermediate_column_1250_2 AS user_nickname, "pg_merge_job_1250.task_000067".intermediate_column_1250_3 AS user_billing, "pg_merge_job_1250.task_000067".intermediate_column_1250_4 AS user_address_id, "pg_merge_job_1251.task_000067".intermediate_column_1251_0 AS address_id, "pg_merge_job_1251.task_000067".intermediate_column_1251_1 AS full_address, "pg_merge_job_1251.task_000067".intermediate_column_1251_2 AS first, "pg_merge_job_1251.task_000067".intermediate_column_1251_3 AS second, "pg_merge_job_1251.task_000067".intermediate_column_1251_4 AS zip, "pg_merge_job_1251.task_000067".intermediate_column_1251_5 AS city, "pg_merge_job_1251.task_000067".intermediate_column_1251_6 AS country_code FROM (pg_merge_job_1250.task_000067 "pg_merge_job_1250.task_000067" JOIN pg_merge_job_1251.task_000067 "pg_merge_job_1251.task_000067" ON (("pg_merge_job_1250.task_000067".intermediate_column_1250_4 = "pg_merge_job_1251.task_000067".intermediate_column_1251_0))) WHERE true"
DEBUG:  generated sql query for job 1252 and task 12
DETAIL:  query string: "SELECT "pg_merge_job_1250.task_000084".intermediate_column_1250_0 AS user_id, "pg_merge_job_1250.task_000084".intermediate_column_1250_1 AS user_addr, "pg_merge_job_1250.task_000084".intermediate_column_1250_2 AS user_nickname, "pg_merge_job_1250.task_000084".intermediate_column_1250_3 AS user_billing, "pg_merge_job_1250.task_000084".intermediate_column_1250_4 AS user_address_id, "pg_merge_job_1251.task_000084".intermediate_column_1251_0 AS address_id, "pg_merge_job_1251.task_000084".intermediate_column_1251_1 AS full_address, "pg_merge_job_1251.task_000084".intermediate_column_1251_2 AS first, "pg_merge_job_1251.task_000084".intermediate_column_1251_3 AS second, "pg_merge_job_1251.task_000084".intermediate_column_1251_4 AS zip, "pg_merge_job_1251.task_000084".intermediate_column_1251_5 AS city, "pg_merge_job_1251.task_000084".intermediate_column_1251_6 AS country_code FROM (pg_merge_job_1250.task_000084 "pg_merge_job_1250.task_000084" JOIN pg_merge_job_1251.task_000084 "pg_merge_job_1251.task_000084" ON (("pg_merge_job_1250.task_000084".intermediate_column_1250_4 = "pg_merge_job_1251.task_000084".intermediate_column_1251_0))) WHERE true"
DEBUG:  generated sql query for job 1252 and task 15
DETAIL:  query string: "SELECT "pg_merge_job_1250.task_000101".intermediate_column_1250_0 AS user_id, "pg_merge_job_1250.task_000101".intermediate_column_1250_1 AS user_addr, "pg_merge_job_1250.task_000101".intermediate_column_1250_2 AS user_nickname, "pg_merge_job_1250.task_000101".intermediate_column_1250_3 AS user_billing, "pg_merge_job_1250.task_000101".intermediate_column_1250_4 AS user_address_id, "pg_merge_job_1251.task_000101".intermediate_column_1251_0 AS address_id, "pg_merge_job_1251.task_000101".intermediate_column_1251_1 AS full_address, "pg_merge_job_1251.task_000101".intermediate_column_1251_2 AS first, "pg_merge_job_1251.task_000101".intermediate_column_1251_3 AS second, "pg_merge_job_1251.task_000101".intermediate_column_1251_4 AS zip, "pg_merge_job_1251.task_000101".intermediate_column_1251_5 AS city, "pg_merge_job_1251.task_000101".intermediate_column_1251_6 AS country_code FROM (pg_merge_job_1250.task_000101 "pg_merge_job_1250.task_000101" JOIN pg_merge_job_1251.task_000101 "pg_merge_job_1251.task_000101" ON (("pg_merge_job_1250.task_000101".intermediate_column_1250_4 = "pg_merge_job_1251.task_000101".intermediate_column_1251_0))) WHERE true"
DEBUG:  generated sql query for job 1252 and task 18
DETAIL:  query string: "SELECT "pg_merge_job_1250.task_000118".intermediate_column_1250_0 AS user_id, "pg_merge_job_1250.task_000118".intermediate_column_1250_1 AS user_addr, "pg_merge_job_1250.task_000118".intermediate_column_1250_2 AS user_nickname, "pg_merge_job_1250.task_000118".intermediate_column_1250_3 AS user_billing, "pg_merge_job_1250.task_000118".intermediate_column_1250_4 AS user_address_id, "pg_merge_job_1251.task_000118".intermediate_column_1251_0 AS address_id, "pg_merge_job_1251.task_000118".intermediate_column_1251_1 AS full_address, "pg_merge_job_1251.task_000118".intermediate_column_1251_2 AS first, "pg_merge_job_1251.task_000118".intermediate_column_1251_3 AS second, "pg_merge_job_1251.task_000118".intermediate_column_1251_4 AS zip, "pg_merge_job_1251.task_000118".intermediate_column_1251_5 AS city, "pg_merge_job_1251.task_000118".intermediate_column_1251_6 AS country_code FROM (pg_merge_job_1250.task_000118 "pg_merge_job_1250.task_000118" JOIN pg_merge_job_1251.task_000118 "pg_merge_job_1251.task_000118" ON (("pg_merge_job_1250.task_000118".intermediate_column_1250_4 = "pg_merge_job_1251.task_000118".intermediate_column_1251_0))) WHERE true"
DEBUG:  generated sql query for job 1252 and task 21
DETAIL:  query string: "SELECT "pg_merge_job_1250.task_000135".intermediate_column_1250_0 AS user_id, "pg_merge_job_1250.task_000135".intermediate_column_1250_1 AS user_addr, "pg_merge_job_1250.task_000135".intermediate_column_1250_2 AS user_nickname, "pg_merge_job_1250.task_000135".intermediate_column_1250_3 AS user_billing, "pg_merge_job_1250.task_000135".intermediate_column_1250_4 AS user_address_id, "pg_merge_job_1251.task_000135".intermediate_column_1251_0 AS address_id, "pg_merge_job_1251.task_000135".intermediate_column_1251_1 AS full_address, "pg_merge_job_1251.task_000135".intermediate_column_1251_2 AS first, "pg_merge_job_1251.task_000135".intermediate_column_1251_3 AS second, "pg_merge_job_1251.task_000135".intermediate_column_1251_4 AS zip, "pg_merge_job_1251.task_000135".intermediate_column_1251_5 AS city, "pg_merge_job_1251.task_000135".intermediate_column_1251_6 AS country_code FROM (pg_merge_job_1250.task_000135 "pg_merge_job_1250.task_000135" JOIN pg_merge_job_1251.task_000135 "pg_merge_job_1251.task_000135" ON (("pg_merge_job_1250.task_000135".intermediate_column_1250_4 = "pg_merge_job_1251.task_000135".intermediate_column_1251_0))) WHERE true"
DEBUG:  generated sql query for job 1252 and task 24
DETAIL:  query string: "SELECT "pg_merge_job_1250.task_000152".intermediate_column_1250_0 AS user_id, "pg_merge_job_1250.task_000152".intermediate_column_1250_1 AS user_addr, "pg_merge_job_1250.task_000152".intermediate_column_1250_2 AS user_nickname, "pg_merge_job_1250.task_000152".intermediate_column_1250_3 AS user_billing, "pg_merge_job_1250.task_000152".intermediate_column_1250_4 AS user_address_id, "pg_merge_job_1251.task_000152".intermediate_column_1251_0 AS address_id, "pg_merge_job_1251.task_000152".intermediate_column_1251_1 AS full_address, "pg_merge_job_1251.task_000152".intermediate_column_1251_2 AS first, "pg_merge_job_1251.task_000152".intermediate_column_1251_3 AS second, "pg_merge_job_1251.task_000152".intermediate_column_1251_4 AS zip, "pg_merge_job_1251.task_000152".intermediate_column_1251_5 AS city, "pg_merge_job_1251.task_000152".intermediate_column_1251_6 AS country_code FROM (pg_merge_job_1250.task_000152 "pg_merge_job_1250.task_000152" JOIN pg_merge_job_1251.task_000152 "pg_merge_job_1251.task_000152" ON (("pg_merge_job_1250.task_000152".intermediate_column_1250_4 = "pg_merge_job_1251.task_000152".intermediate_column_1251_0))) WHERE true"
DEBUG:  pruning merge fetch taskId 1
DETAIL:  Creating dependency on merge taskId 33
DEBUG:  pruning merge fetch taskId 2
DETAIL:  Creating dependency on merge taskId 33
DEBUG:  pruning merge fetch taskId 4
DETAIL:  Creating dependency on merge taskId 50
DEBUG:  pruning merge fetch taskId 5
DETAIL:  Creating dependency on merge taskId 50
DEBUG:  pruning merge fetch taskId 7
DETAIL:  Creating dependency on merge taskId 67
DEBUG:  pruning merge fetch taskId 8
DETAIL:  Creating dependency on merge taskId 67
DEBUG:  pruning merge fetch taskId 10
DETAIL:  Creating dependency on merge taskId 84
DEBUG:  pruning merge fetch taskId 11
DETAIL:  Creating dependency on merge taskId 84
DEBUG:  pruning merge fetch taskId 13
DETAIL:  Creating dependency on merge taskId 101
DEBUG:  pruning merge fetch taskId 14
DETAIL:  Creating dependency on merge taskId 101
DEBUG:  pruning merge fetch taskId 16
DETAIL:  Creating dependency on merge taskId 118
DEBUG:  pruning merge fetch taskId 17
DETAIL:  Creating dependency on merge taskId 118
DEBUG:  pruning merge fetch taskId 19
DETAIL:  Creating dependency on merge taskId 135
DEBUG:  pruning merge fetch taskId 20
DETAIL:  Creating dependency on merge taskId 135
DEBUG:  pruning merge fetch taskId 22
DETAIL:  Creating dependency on merge taskId 152
DEBUG:  pruning merge fetch taskId 23
DETAIL:  Creating dependency on merge taskId 152
DEBUG:  assigned task 3 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 6 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 9 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 12 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 15 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 18 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  assigned task 21 to node ip-10-192-0-225.eu-central-1.compute.internal:5432
DEBUG:  assigned task 24 to node ip-10-192-1-242.eu-central-1.compute.internal:5432
DEBUG:  multiple task state entries for job 1251 and task 160
DEBUG:  multiple task state entries for job 1250 and task 94
DEBUG:  multiple task state entries for job 1251 and task 167
DEBUG:  multiple task state entries for job 1250 and task 80
DEBUG:  completed cleanup query for job 1252 on node "ip-10-192-0-225.eu-central-1.compute.internal:5432"
DEBUG:  completed cleanup query for job 1252 on node "ip-10-192-1-242.eu-central-1.compute.internal:5432"
DEBUG:  completed cleanup query for job 1250 on node "ip-10-192-0-225.eu-central-1.compute.internal:5432"
DEBUG:  completed cleanup query for job 1250 on node "ip-10-192-1-242.eu-central-1.compute.internal:5432"
DEBUG:  completed cleanup query for job 1251 on node "ip-10-192-0-225.eu-central-1.compute.internal:5432"
DEBUG:  completed cleanup query for job 1251 on node "ip-10-192-1-242.eu-central-1.compute.internal:5432"
ERROR:  failed to execute job 1252
DETAIL:  Failure due to failed task 21
Time: 3062.832 ms
ROLLBACK
Time: 0.464 ms


I made the change, but it doesn't seem to work for me (although I could make a mistake somewhere). 
Is there anything else I can do to solve this problem? 


  

Sai Srirampur

unread,
May 23, 2016, 9:04:18 PM5/23/16
to Макс Левин, citus-users
Hi Maxim,

Thanks for reaching out to us.

Just to confirm, did you modify authentication settings in pg_hba.conf on all the workers ?

Is there a specific reason that you altered the job id sequence ?  Citus uses this sequence internally to generate job ids. You shouldn't need to modify the sequence to run task tracker queries.

To help us to debug better, can you send the error logs on the worker ip-10-192-0-225.eu-central-1.compute.internal, because the task failure seems to be on this node ?

Sai

--
You received this message because you are subscribed to the Google Groups "citus-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to citus-users...@googlegroups.com.
To post to this group, send email to citus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/citus-users/17762c45-906f-4f02-951e-cf97a75623a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Max Levin

unread,
May 24, 2016, 11:54:11 AM5/24/16
to citus-users
Hello, Sai.

I modified pg_hba.conf on all the workers and did a restart. 
About the sequence: I'm not very familiar with the subject, I just saw a few examples of how a join should be done, I thought it's a necessity of some kind. 
Here is a part of the error log of the worker ip-10-192-0-32.eu-central-1.compute.internal (don't pay attention to ip change, I made a new cluster):
...
STATEMENT:  COPY "base/pgsql_job_cache/job_1275/task_000002/p_00005" TO STDOUT WITH (format 'transmit')
WARNING:  could not receive query results from ip-10-192-0-32.eu-central-1.compute.internal:5432
DETAIL:  Client error: relation "base/pgsql_job_cache/job_1275/task_000010/p_00000" does not exist
WARNING:  could not receive query results from localhost:5432
DETAIL:  Client error: could not receive file "base/pgsql_job_cache/job_1275/task_000010/p_00000" from ip-10-192-0-32.eu-central-1.compute.internal:5432
WARNING:  could not receive query results from ip-10-192-0-32.eu-central-1.compute.internal:5432
DETAIL:  Client error: relation "base/pgsql_job_cache/job_1275/task_000018/p_00006" does not exist
WARNING:  could not receive query results from localhost:5432
DETAIL:  Client error: could not receive file "base/pgsql_job_cache/job_1275/task_000018/p_00006" from ip-10-192-0-32.eu-central-1.compute.internal:5432
WARNING:  could not receive query results from ip-10-192-0-33.eu-central-1.compute.internal:5432
DETAIL:  Client error: relation "base/pgsql_job_cache/job_1274/task_000012/p_00006" does not exist
WARNING:  could not delete file "base/pgsql_job_cache/job_1274/task_000135/task_000012_1387658420.attempt": No such file or directory
WARNING:  could not receive query results from ip-10-192-0-32.eu-central-1.compute.internal:5432
DETAIL:  Client error: relation "base/pgsql_job_cache/job_1274/task_000010/p_00000" does not exist
WARNING:  could not receive query results from ip-10-192-0-32.eu-central-1.compute.internal:5432
DETAIL:  Client error: relation "base/pgsql_job_cache/job_1274/task_000026/p_00006" does not exist
WARNING:  could not delete file "base/pgsql_job_cache/job_1274/task_000135/task_000026_1434543459.attempt": No such file or directory
WARNING:  could not receive query results from ip-10-192-0-33.eu-central-1.compute.internal:5432
DETAIL:  Client error: relation "base/pgsql_job_cache/job_1275/task_000004/p_00006" does not exist
...

but at the same time, here is part of the log from ip-10-192-0-33.eu-central-1.compute.internal :

...
LOG:  statement: SELECT task_tracker_task_status(1274, 156)
LOG:  statement: SELECT task_tracker_task_status(1274, 88)
LOG:  statement: SELECT task_tracker_task_status(1274, 54)
LOG:  connection received: host=ip-10-192-0-32.eu-central-1.compute.internal port=44372
LOG:  connection authorized: user=ec2-user database=postgres
LOG:  statement: COPY "base/pgsql_job_cache/job_1274/task_000032/p_00002" TO STDOUT WITH (format 'transmit')
ERROR:  relation "base/pgsql_job_cache/job_1274/task_000032/p_00002" does not exist
...

FYI, I just used this template for the cluster organisation:

вторник, 24 мая 2016 г., 4:04:18 UTC+3 пользователь Sai Srirampur написал:
Reply all
Reply to author
Forward
0 new messages