Help: Hang when running "COPY sub_table FROM ..." for a partition table in parallel

60 views

Skip to first unread message

Ming Li

unread,

Nov 6, 2019, 3:52:50 AM11/6/19

to Greenplum Developers

Hi all,

"COPY sub_table FROM ...." will trigger "vacuum()->analyze_rel()" on the parent of the partition table. So if we process different subtables for the same partition table at the same time in parallel, it may hang forever.

I wonder is there any solution to fix this problem? Or any GUC setting to workaround? Many thanks in advance!

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Below are the details:

1. The table "sales" is a partition table which has some sub tables:

2. When running "gpcopy --parallelize-leaf-partitions=true", it will process 2 sub tables at the same time, the log is:

20711 20191106:14:14:52 gpcopy:gpadmin:gpdb:102368-[DEBUG]:-[Worker 0] COPY command of sending data: COPY "public"."sales_1_prt_feb17" TO PROGRAM 'gpcopy_help er --seg-id <SEGID> --host 192.168.2.12,192.168.2.12,192.168.2.12 --port 21634,36245,27914' ON SEGMENT CSV IGNORE EXTERNAL PARTITIONS

20713 20191106:14:14:52 gpcopy:gpadmin:gpdb:102368-[DEBUG]:-[Worker 1] COPY command of sending data: COPY "public"."sales_1_prt_apr17" TO PROGRAM 'gpcopy_help er --seg-id <SEGID> --host 192.168.2.12,192.168.2.12,192.168.2.12 --port 34817,33720,31060' ON SEGMENT CSV IGNORE EXTERNAL PARTITIONS

3. Then the ps has belows output:

gpadmin 66665 87875 0 14:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(62826) con6925 cmd16 COPY

gpadmin 66673 87843 0 14:37 ? 00:00:00 postgres: 6002, gpadmin testdb 192.168.2.12(47272) con6925 seg0 cmd16 SELECT waiting
gpadmin 66674 87844 0 14:37 ? 00:00:00 postgres: 6003, gpadmin testdb 192.168.2.12(51614) con6925 seg1 cmd16 SELECT waiting
gpadmin 66675 87845 0 14:37 ? 00:00:00 postgres: 6004, gpadmin testdb 192.168.2.12(52646) con6925 seg2 cmd16 SELECT waiting

gpadmin 66667 87875 0 14:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(62828) con6926 cmd10 COPY waiting

4. The related bt of the process is:

(1) One hanging query

gpadmin 66665 87875 0 14:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(62826) con6925 cmd16 COPY

(gdb) bt
#0 0x00007f121a55ea3d in poll () from /lib64/libc.so.6
#1 0x0000000000b88524 in checkDispatchResult (ds=0x1ee03b8, wait=1 '\001') at cdbdisp_async.c:486
#2 0x0000000000b8817b in cdbdisp_checkDispatchResult_async (ds=0x1ee03b8, waitMode=DISPATCH_WAIT_NONE) at cdbdisp_async.c:333
#3 0x0000000000b87007 in cdbdisp_checkDispatchResult (ds=0x1ee03b8, waitMode=DISPATCH_WAIT_NONE) at cdbdisp.c:108
#4 0x0000000000b8c028 in cdbdisp_dispatchCommandInternal (pQueryParms=0x1f4a808, flags=4, segments=0x1f5b8a0, cdb_pgresults=0x7ffd6d9b6e80)
at cdbdisp_query.c:446
#5 0x0000000000b8be59 in CdbDispatchCommandToSegments (
strCommand=0x1f5b8d8 "select * from pg_catalog.gp_acquire_sample_rows(97607, 10000, 't') as (totalrows pg_catalog.float8, totaldeadrows pg_catalog.float8, oversized_cols_bitmap pg_catalog.text, id integer, date date, amt n"..., flags=4, segments=0x1f5b8a0, cdb_pgresults=0x7ffd6d9b6e80) at cdbdisp_query.c:366
#6 0x0000000000b8bda8 in CdbDispatchCommand (
strCommand=0x1f5b8d8 "select * from pg_catalog.gp_acquire_sample_rows(97607, 10000, 't') as (totalrows pg_catalog.float8, totaldeadrows pg_catalog.float8, oversized_cols_bitmap pg_catalog.text, id integer, date date, amt n"..., flags=4, cdb_pgresults=0x7ffd6d9b6e80) at cdbdisp_query.c:338
#7 0x0000000000672e1e in acquire_sample_rows_dispatcher (onerel=0x7f121c2087c8, inh=1 '\001', elevel=13, rows=0x1f7fbf8, targrows=30000,
totalrows=0x7ffd6d9b7168, totaldeadrows=0x7ffd6d9b7160) at analyze.c:2442
#8 0x0000000000671f8a in acquire_inherited_sample_rows (onerel=0x7f121c2087c8, elevel=13, rows=0x1f7fbf8, targrows=30000, totalrows=0x7ffd6d9b7168,
totaldeadrows=0x7ffd6d9b7160) at analyze.c:2014
#9 0x000000000066f49f in do_analyze_rel (onerel=0x7f121c2087c8, vacstmt=0x1f560c0, acquirefunc=0x6716ff <acquire_sample_rows>, relpages=0, inh=1 '\001',
in_outer_xact=1 '\001', elevel=13) at analyze.c:679
#10 0x000000000066eade in analyze_rel_internal (relid=97607, vacstmt=0x1f560c0, in_outer_xact=1 '\001', bstrategy=0x1f37308) at analyze.c:405
#11 0x000000000066e48e in analyze_rel (relid=97607, vacstmt=0x1f560c0, in_outer_xact=1 '\001', bstrategy=0x1f37308) at analyze.c:223
#12 0x000000000071a144 in vacuum (vacstmt=0x1f560c0, relid=0, do_toast=0 '\000', bstrategy=0x1f37308, for_wraparound=0 '\000', isTopLevel=0 '\000')
at vacuum.c:387
#13 0x00000000008e07b9 in autostats_issue_analyze (relationOid=97615) at autostats.c:74
#14 0x00000000008e0d3b in auto_stats (cmdType=AUTOSTATS_CMDTYPE_COPY, relationOid=97615, ntuples=1, inFunction=0 '\000') at autostats.c:324
#15 0x0000000000683f37 in DoCopy (stmt=0x1e35540,
queryString=0x1e34678 "COPY \"public\".\"sales_1_prt_feb17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 9659f97f-03d9-4e1c-b6d4-68d47662f058' ON SEGMENT CSV", processed=0x7ffd6d9b7890) at copy.c:1173
#16 0x000000000096dd01 in standard_ProcessUtility (parsetree=0x1e35540,
queryString=0x1e34678 "COPY \"public\".\"sales_1_prt_feb17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 9659f97f-03d9-4e1c-b6d4-68d47662f058' ON SEGMENT CSV", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0x1e35928, completionTag=0x7ffd6d9b7be0 "") at utility.c:634
#17 0x000000000096d677 in ProcessUtility (parsetree=0x1e35540,
queryString=0x1e34678 "COPY \"public\".\"sales_1_prt_feb17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 9659f97f-03d9-4e1c-b6d4-68d47662f058' ON SEGMENT CSV", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0x1e35928, completionTag=0x7ffd6d9b7be0 "") at utility.c:373
#18 0x000000000096c44a in PortalRunUtility (portal=0x1e3c238, utilityStmt=0x1e35540, isTopLevel=1 '\001', dest=0x1e35928, completionTag=0x7ffd6d9b7be0 "")
at pquery.c:1379
#19 0x000000000096c6db in PortalRunMulti (portal=0x1e3c238, isTopLevel=1 '\001', dest=0x1e35928, altdest=0x1e35928, completionTag=0x7ffd6d9b7be0 "")
at pquery.c:1510
#20 0x000000000096bbd2 in PortalRun (portal=0x1e3c238, count=9223372036854775807, isTopLevel=1 '\001', dest=0x1e35928, altdest=0x1e35928,
completionTag=0x7ffd6d9b7be0 "") at pquery.c:1016
#21 0x00000000009640d2 in exec_simple_query (
query_string=0x1e34678 "COPY \"public\".\"sales_1_prt_feb17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 9659f97f-03d9-4e1c-b6d4-68d47662f058' ON SEGMENT CSV") at postgres.c:1798
#22 0x0000000000968e8c in PostgresMain (argc=1, argv=0x1e14228, dbname=0x1e13fd0 "testdb", username=0x1e14168 "gpadmin") at postgres.c:5215
#23 0x00000000008d7c40 in BackendRun (port=0x1e3c040) at postmaster.c:4803
#24 0x00000000008d72e0 in BackendStartup (port=0x1e3c040) at postmaster.c:4460
#25 0x00000000008d32ab in ServerLoop () at postmaster.c:1945
#26 0x00000000008d27fd in PostmasterMain (argc=6, argv=0x1e12020) at postmaster.c:1515
#27 0x00000000007cdc32 in main (argc=6, argv=0x1e12020) at main.c:245

It waiting for the result of below QEs:
gpadmin 66673 87843 0 14:37 ? 00:00:00 postgres: 6002, gpadmin testdb 192.168.2.12(47272) con6925 seg0 cmd16 SELECT waiting
gpadmin 66674 87844 0 14:37 ? 00:00:00 postgres: 6003, gpadmin testdb 192.168.2.12(51614) con6925 seg1 cmd16 SELECT waiting
gpadmin 66675 87845 0 14:37 ? 00:00:00 postgres: 6004, gpadmin testdb 192.168.2.12(52646) con6925 seg2 cmd16 SELECT waiting

gdb -p 66673
(gdb) bt
#0 0x00007f3d9b5c2b47 in semop () from /lib64/libc.so.6
#1 0x00000000008bec73 in PGSemaphoreLock (sema=0x7f3d8e6b7380, interruptOK=1 '\001') at pg_sema.c:422
#2 0x000000000094e901 in ProcSleep (locallock=0x20d9f10, lockMethodTable=0xdc0560 <default_lockmethod>) at proc.c:1338
#3 0x00000000009477cf in WaitOnLock (locallock=0x20d9f10, owner=0x20d4a88) at lock.c:1849
#4 0x000000000094654e in LockAcquireExtended (locktag=0x7ffff3878600, lockmode=1, sessionLock=0 '\000', dontWait=0 '\000', reportMemoryError=1 '\001',
locallockp=0x7ffff38785f8) at lock.c:1151
#5 0x00000000009438d5 in LockRelationOid (relid=97623, lockmode=1) at lmgr.c:102
#6 0x00000000005e1a85 in find_inheritance_children (parentrelId=97607, lockmode=1) at pg_inherits.c:121
#7 0x00000000005e1c4b in find_all_inheritors (parentrelId=97607, lockmode=1, numparents=0x0) at pg_inherits.c:211
#8 0x0000000000671faa in acquire_inherited_sample_rows (onerel=0x7f3d9d1d4a98, elevel=14, rows=0x217b838, targrows=10000, totalrows=0x7ffff3878860,
totaldeadrows=0x7ffff3878858) at analyze.c:2024
#9 0x00000000007259c8 in gp_acquire_sample_rows (fcinfo=0x7ffff3878a90) at analyzefuncs.c:202
#10 0x0000000000743002 in ExecMakeTableFunctionResult (funcexpr=0x2170660, econtext=0x216ff10, argContext=0x20d5230, expectedDesc=0x2170fd0,
randomAccess=0 '\000', operatorMemKB=32768) at execQual.c:2337
#11 0x000000000077531b in FunctionNext_guts (node=0x216faa0) at nodeFunctionscan.c:103
#12 0x0000000000775925 in FunctionNext (node=0x216faa0) at nodeFunctionscan.c:296
#13 0x000000000074cf1f in ExecScanFetch (node=0x216faa0, accessMtd=0x77590d <FunctionNext>, recheckMtd=0x77595a <FunctionRecheck>) at execScan.c:84
#14 0x000000000074cf8a in ExecScan (node=0x216faa0, accessMtd=0x77590d <FunctionNext>, recheckMtd=0x77595a <FunctionRecheck>) at execScan.c:132
#15 0x000000000077598f in ExecFunctionScan (node=0x216faa0) at nodeFunctionscan.c:326
#16 0x000000000073e12c in ExecProcNode (node=0x216faa0) at execProcnode.c:1043
#17 0x00000000007387d2 in ExecutePlan (estate=0x216f7b8, planstate=0x216faa0, operation=CMD_SELECT, sendTuples=1 '\001', numberTuples=0,
direction=ForwardScanDirection, dest=0x216b020) at execMain.c:2979
#18 0x0000000000735049 in standard_ExecutorRun (queryDesc=0x215aab8, direction=ForwardScanDirection, count=0) at execMain.c:968
#19 0x0000000000734cc1 in ExecutorRun (queryDesc=0x215aab8, direction=ForwardScanDirection, count=0) at execMain.c:834
#20 0x000000000096be92 in PortalRunSelect (portal=0x21486a8, forward=1 '\001', count=0, dest=0x216b020) at pquery.c:1149
#21 0x000000000096bb08 in PortalRun (portal=0x21486a8, count=9223372036854775807, isTopLevel=1 '\001', dest=0x216b020, altdest=0x216b020,
completionTag=0x7ffff38795a0 "") at pquery.c:990
#22 0x00000000009640d2 in exec_simple_query (
query_string=0x213dd94 "select * from pg_catalog.gp_acquire_sample_rows(97607, 10000, 't') as (totalrows pg_catalog.float8, totaldeadrows pg_catalog.float8, oversized_cols_bitmap pg_catalog.text, id integer, date date, amt n"...) at postgres.c:1798
#23 0x00000000009693b7 in PostgresMain (argc=1, argv=0x20ac968, dbname=0x20ac840 "testdb", username=0x20ac820 "gpadmin") at postgres.c:5357
#24 0x00000000008d7c40 in BackendRun (port=0x20dcaf0) at postmaster.c:4803
#25 0x00000000008d72e0 in BackendStartup (port=0x20dcaf0) at postmaster.c:4460
#26 0x00000000008d32ab in ServerLoop () at postmaster.c:1945
#27 0x00000000008d27fd in PostmasterMain (argc=5, argv=0x20aa890) at postmaster.c:1515
#28 0x00000000007cdc32 in main (argc=5, argv=0x20aa890) at main.c:245

(2) The other hanging query

gpadmin 66667 87875 0 14:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(62828) con6926 cmd10 COPY waiting

(gdb) bt
#0 0x00007f121a56ab47 in semop () from /lib64/libc.so.6
#1 0x00000000008bec73 in PGSemaphoreLock (sema=0x7f120ec28860, interruptOK=1 '\001') at pg_sema.c:422
#2 0x000000000094e901 in ProcSleep (locallock=0x1e41900, lockMethodTable=0xdc0560 <default_lockmethod>) at proc.c:1338
#3 0x00000000009477cf in WaitOnLock (locallock=0x1e41900, owner=0x1e8be90) at lock.c:1849
#4 0x000000000094654e in LockAcquireExtended (locktag=0x7ffd6d9b7260, lockmode=4, sessionLock=0 '\000', dontWait=0 '\000', reportMemoryError=1 '\001',
locallockp=0x7ffd6d9b7258) at lock.c:1151
#5 0x00000000009438d5 in LockRelationOid (relid=97607, lockmode=4) at lmgr.c:102
#6 0x00000000004df9ab in try_relation_open (relationId=97607, lockmode=4, noWait=0 '\000') at heapam.c:1126
#7 0x000000000066e603 in analyze_rel_internal (relid=97607, vacstmt=0x1e3f340, in_outer_xact=1 '\001', bstrategy=0x1f092f8) at analyze.c:269
#8 0x000000000066e48e in analyze_rel (relid=97607, vacstmt=0x1e3f340, in_outer_xact=1 '\001', bstrategy=0x1f092f8) at analyze.c:223
#9 0x000000000071a144 in vacuum (vacstmt=0x1e3f340, relid=0, do_toast=0 '\000', bstrategy=0x1f092f8, for_wraparound=0 '\000', isTopLevel=0 '\000')
at vacuum.c:387
#10 0x00000000008e07b9 in autostats_issue_analyze (relationOid=97623) at autostats.c:74
#11 0x00000000008e0d3b in auto_stats (cmdType=AUTOSTATS_CMDTYPE_COPY, relationOid=97623, ntuples=1, inFunction=0 '\000') at autostats.c:324
#12 0x0000000000683f37 in DoCopy (stmt=0x1e35540,
queryString=0x1e34678 "COPY \"public\".\"sales_1_prt_apr17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id e6252549-c329-4b88-a203-b80ac9ff80dd' ON SEGMENT CSV", processed=0x7ffd6d9b7890) at copy.c:1173
#13 0x000000000096dd01 in standard_ProcessUtility (parsetree=0x1e35540,
queryString=0x1e34678 "COPY \"public\".\"sales_1_prt_apr17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id e6252549-c329-4b88-a203-b80ac9ff80dd' ON SEGMENT CSV", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0x1e35928, completionTag=0x7ffd6d9b7be0 "") at utility.c:634
#14 0x000000000096d677 in ProcessUtility (parsetree=0x1e35540,
queryString=0x1e34678 "COPY \"public\".\"sales_1_prt_apr17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id e6252549-c329-4b88-a203-b80ac9ff80dd' ON SEGMENT CSV", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0x1e35928, completionTag=0x7ffd6d9b7be0 "") at utility.c:373
#15 0x000000000096c44a in PortalRunUtility (portal=0x1edfb08, utilityStmt=0x1e35540, isTopLevel=1 '\001', dest=0x1e35928, completionTag=0x7ffd6d9b7be0 "")
at pquery.c:1379
#16 0x000000000096c6db in PortalRunMulti (portal=0x1edfb08, isTopLevel=1 '\001', dest=0x1e35928, altdest=0x1e35928, completionTag=0x7ffd6d9b7be0 "")
at pquery.c:1510
#17 0x000000000096bbd2 in PortalRun (portal=0x1edfb08, count=9223372036854775807, isTopLevel=1 '\001', dest=0x1e35928, altdest=0x1e35928,
completionTag=0x7ffd6d9b7be0 "") at pquery.c:1016
#18 0x00000000009640d2 in exec_simple_query (
query_string=0x1e34678 "COPY \"public\".\"sales_1_prt_apr17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id e6252549-c329-4b88-a203-b80ac9ff80dd' ON SEGMENT CSV") at postgres.c:1798
#19 0x0000000000968e8c in PostgresMain (argc=1, argv=0x1e14228, dbname=0x1e140c8 "testdb", username=0x1e13fd0 "gpadmin") at postgres.c:5215
#20 0x00000000008d7c40 in BackendRun (port=0x1e3c040) at postmaster.c:4803
#21 0x00000000008d72e0 in BackendStartup (port=0x1e3c040) at postmaster.c:4460
#22 0x00000000008d32ab in ServerLoop () at postmaster.c:1945
#23 0x00000000008d27fd in PostmasterMain (argc=6, argv=0x1e12020) at postmaster.c:1515
#24 0x00000000007cdc32 in main (argc=6, argv=0x1e12020) at main.c:245

5. gpcopy continue if I cancel one of the hang query:

kill -s INT 66667

Zhenghua Lyu

unread,

Nov 6, 2019, 4:04:29 AM11/6/19

to Ming Li, Greenplum Developers

Try this GUC to disable auto stats: gp_autostats_mode

QE 66673 is waiting for when it tries to hold AccessShare Lock, which means some other processes hold Access ExclusiveLock.

Which process hold it?

Best Regards,

Zhenghua Lyu

Ming Li

unread,

Nov 6, 2019, 4:07:18 AM11/6/19

to Zhenghua Lyu, Greenplum Developers

Thanks Zhenghua,

It should be the process with 66667.

Zhenghua Lyu

unread,

Nov 6, 2019, 4:10:20 AM11/6/19

to Ming Li, Greenplum Developers

Why a COPY statement holds Access Exclusive Lock (the highest lock level, often catalog changes need to hold such lock mode).

I recommend to GDB the process to find when holding the lock.

Best Regards,

Zhenghua Lyu

Ming Li

unread,

Nov 6, 2019, 11:00:24 PM11/6/19

to Zhenghua Lyu, Greenplum Developers

Hi Zhenghua,

Good question!

After some investigation, I found that the cause of the Access Exclusive Lock held by the write gang QE process of the QD pid 66667 is:

before execute

> COPY \"public\".\"sales_1_prt_apr17\" FROM PROGRAM ...

it will firstly execute:

> TRUNCATE ""public"".""sales_1_prt_apr17"

which will call ExecuteTruncate() and request AccessExclusiveLock.

However whether or not gpcopy need to call TRUNCATE, it is depend on the customer, gpcopy support 4 parameters for DDL.

--drop

--truncate

--append

--skip-existing

It seems the first 2 parameters have this hang problem.

Thanks.

Ming Li

unread,

Nov 8, 2019, 4:58:56 AM11/8/19

to Zhenghua Lyu, Greenplum Developers

Hi Zhenghua,

Sorry, the previous reply was wrong.

The Access Exclusive Lock was held by the QE of the other query execution. (pls refer to below "# Conclusion:" section).

However even after Global Deadlock Detector enabled as below step, it still hangs. It seems that the waiting dependency (query dispatcher waiting for the QE) is not considered in the Global Deadlock Detector.

Thanks.

Below is some info collected when reproducing this problem.
------------------------------------------------------------------------

# Conclusion:

(1) 117818 QD waiting:
Finish processing "COPY \"public\".\"sales_1_prt_apr17\" FROM PROGRAM ...
But the transaction is not commit
auto trigger analyze on "sales" table
want to request ShareUpdateExclusiveLock on "sales" table.

(2) --> (waiting for) 117816 QD
Finish Executing ""COPY \"public\".\"sales_1_prt_feb17\" FROM PROGRAM ..."
auto trigger analyze on "sales" table
Dispatching query "select * from pg_catalog.gp_acquire_sample_rows(178296, 10000, 't') ...
And waiting for the QE to return

(3) The Seg0 QE 117824 request AccessShareLock on table sales_1_prt_apr17 when processing query "select * from pg_catalog.gp_acquire_sample_rows(178296, 10000, 't') ...

(4) --> (waiting for) 117830 which is the seg0 QE of the con2862
gpadmin 117818 33176 0 15:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(21208) con2862 cmd10 COPY waiting

And this QE has finished processing "COPY \"public\".\"sales_1_prt_apr17\" FROM PROGRAM ... ON SEGMENT but not COMMIT (which means the held locks have not been released)

So this is a deadlock, which hangs forever.

# Some details:

testdb=# select oid, relname from pg_class where relname like 'sales%';
oid | relname

--------+-------------------
178296 | sales
178299 | sales_1_prt_jan17
178308 | sales_1_prt_mar17
178304 | sales_1_prt_feb17
178312 | sales_1_prt_apr17
178316 | sales_1_prt_may17
178320 | sales_1_prt_jun17
178328 | sales_1_prt_aug17
178324 | sales_1_prt_jul17
178332 | sales_1_prt_sep17
178336 | sales_1_prt_oct17
178340 | sales_1_prt_nov17
178344 | sales_1_prt_dec17
(13 rows)

gpadmin@gpdb-2 ~ $ pp
gpadmin 33138 1 0 15:18 ? 00:00:11 /home/gpadmin/workspace/install/gpdb/bin/postgres -D /home/gpadmin/workspace/gpdb/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0 -p 6002
gpadmin 33139 1 0 15:18 ? 00:00:12 /home/gpadmin/workspace/install/gpdb/bin/postgres -D /home/gpadmin/workspace/gpdb/gpAux/gpdemo/datadirs/dbfast2/demoDataDir1 -p 6003
gpadmin 33140 1 0 15:18 ? 00:00:12 /home/gpadmin/workspace/install/gpdb/bin/postgres -D /home/gpadmin/workspace/gpdb/gpAux/gpdemo/datadirs/dbfast3/demoDataDir2 -p 6004
gpadmin 33141 33138 0 15:18 ? 00:00:00 postgres: 6002, logger process
gpadmin 33146 33138 0 15:18 ? 00:00:00 postgres: 6002, checkpointer process
gpadmin 33147 33138 0 15:18 ? 00:00:00 postgres: 6002, writer process
gpadmin 33148 33138 0 15:18 ? 00:00:00 postgres: 6002, wal writer process
gpadmin 33149 33138 0 15:18 ? 00:00:00 postgres: 6002, stats collector process
gpadmin 33150 33138 0 15:18 ? 00:00:00 postgres: 6002, bgworker: sweeper process
gpadmin 33152 33139 0 15:18 ? 00:00:00 postgres: 6003, logger process
gpadmin 33156 33140 0 15:18 ? 00:00:00 postgres: 6004, logger process
gpadmin 33161 33139 0 15:18 ? 00:00:00 postgres: 6003, checkpointer process
gpadmin 33162 33139 0 15:18 ? 00:00:00 postgres: 6003, writer process
gpadmin 33163 33139 0 15:18 ? 00:00:02 postgres: 6003, wal writer process
gpadmin 33164 33139 0 15:18 ? 00:00:00 postgres: 6003, stats collector process
gpadmin 33165 33139 0 15:18 ? 00:00:00 postgres: 6003, bgworker: sweeper process
gpadmin 33167 33140 0 15:18 ? 00:00:00 postgres: 6004, checkpointer process
gpadmin 33168 33140 0 15:18 ? 00:00:00 postgres: 6004, writer process
gpadmin 33169 33140 0 15:18 ? 00:00:00 postgres: 6004, wal writer process
gpadmin 33170 33140 0 15:18 ? 00:00:00 postgres: 6004, stats collector process
gpadmin 33171 33140 0 15:18 ? 00:00:00 postgres: 6004, bgworker: sweeper process
gpadmin 33176 1 0 15:18 ? 00:00:10 /home/gpadmin/workspace/install/gpdb/bin/postgres -D /home/gpadmin/workspace/gpdb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 -p 6000 -E
gpadmin 33177 33176 0 15:19 ? 00:00:04 postgres: 6000, master logger process
gpadmin 33181 33176 0 15:19 ? 00:00:00 postgres: 6000, checkpointer process
gpadmin 33182 33176 0 15:19 ? 00:00:00 postgres: 6000, writer process
gpadmin 33183 33176 0 15:19 ? 00:00:00 postgres: 6000, wal writer process
gpadmin 33184 33176 0 15:19 ? 00:00:00 postgres: 6000, stats collector process
gpadmin 33186 33176 0 15:19 ? 00:00:00 postgres: 6000, bgworker: ftsprobe process
gpadmin 33199 33176 0 15:19 ? 00:00:00 postgres: 6000, bgworker: sweeper process
gpadmin 33209 33176 0 15:19 ? 00:00:06 postgres: 6000, gpadmin postgres 192.168.2.11(40338) con7 cmd25 idle
gpadmin 117750 33176 0 15:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(21136) con2855 cmd4 idle
gpadmin 117752 33138 0 15:37 ? 00:00:00 postgres: 6002, gpadmin testdb 192.168.2.12(46228) con2855 seg0 idle
gpadmin 117753 33139 0 15:37 ? 00:00:00 postgres: 6003, gpadmin testdb 192.168.2.12(50570) con2855 seg1 idle
gpadmin 117754 33140 0 15:37 ? 00:00:00 postgres: 6004, gpadmin testdb 192.168.2.12(51602) con2855 seg2 idle
gpadmin 117758 33176 0 15:37 ? 00:00:00 postgres: 6000, gpadmin postgres 192.168.2.11(21140) con2856 cmd2 idle
gpadmin 117792 33176 0 15:37 ? 00:00:00 postgres: 6000, gpadmin postgres 192.168.2.11(21152) con2858 cmd15 idle
gpadmin 117794 33138 0 15:37 ? 00:00:00 postgres: 6002, gpadmin postgres 192.168.2.12(46240) con2858 seg0 idle
gpadmin 117795 33139 0 15:37 ? 00:00:00 postgres: 6003, gpadmin postgres 192.168.2.12(50582) con2858 seg1 idle
gpadmin 117796 33140 0 15:37 ? 00:00:00 postgres: 6004, gpadmin postgres 192.168.2.12(51614) con2858 seg2 idle
gpadmin 117816 33176 0 15:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(21206) con2861 cmd16 COPY
gpadmin 117818 33176 0 15:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(21208) con2862 cmd10 COPY waiting
gpadmin 117820 33176 0 15:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(21210) con2863 cmd6 idle
gpadmin 117822 33176 0 15:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(21212) con2864 cmd6 idle
gpadmin 117824 33138 0 15:37 ? 00:00:00 postgres: 6002, gpadmin testdb 192.168.2.12(46258) con2861 seg0 cmd16 SELECT waiting
gpadmin 117825 33139 0 15:37 ? 00:00:00 postgres: 6003, gpadmin testdb 192.168.2.12(50600) con2861 seg1 cmd16 SELECT waiting
gpadmin 117826 33140 0 15:37 ? 00:00:00 postgres: 6004, gpadmin testdb 192.168.2.12(51632) con2861 seg2 cmd16 SELECT waiting
gpadmin 117830 33138 0 15:37 ? 00:00:00 postgres: 6002, gpadmin testdb 192.168.2.12(46264) con2862 seg0 idle in transaction
gpadmin 117831 33139 0 15:37 ? 00:00:00 postgres: 6003, gpadmin testdb 192.168.2.12(50606) con2862 seg1 idle in transaction
gpadmin 117832 33140 0 15:37 ? 00:00:00 postgres: 6004, gpadmin testdb 192.168.2.12(51638) con2862 seg2 idle in transaction
gpadmin 117836 33138 0 15:37 ? 00:00:00 postgres: 6002, gpadmin testdb 192.168.2.12(46270) con2863 seg0 idle
gpadmin 117837 33140 0 15:37 ? 00:00:00 postgres: 6004, gpadmin testdb 192.168.2.12(51644) con2863 seg2 idle
gpadmin 117838 33139 0 15:37 ? 00:00:00 postgres: 6003, gpadmin testdb 192.168.2.12(50612) con2863 seg1 idle
gpadmin 117842 33138 0 15:37 ? 00:00:00 postgres: 6002, gpadmin testdb 192.168.2.12(46276) con2864 seg0 idle
gpadmin 117843 33139 0 15:37 ? 00:00:00 postgres: 6003, gpadmin testdb 192.168.2.12(50618) con2864 seg1 idle
gpadmin 117844 33140 0 15:37 ? 00:00:00 postgres: 6004, gpadmin testdb 192.168.2.12(51650) con2864 seg2 idle

postgres=# select * from pg_locks;
locktype | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid | mode | granted | fastpath | mppsessionid | mppiswriter | gp_segment_id
---------------+----------+----------+------+-------+------------+---------------+---------+-------+----------+--------------------+--------+--------------------------+---------+----------+--------------+-------------+---------------
virtualxid | | | | | 5/7953 | | | | | 5/7953 | 117816 | ExclusiveLock | t | t | 2861 | t | -1
relation | 12810 | 11667 | | | | | | | | 10/88 | 118067 | AccessShareLock | t | t | 2868 | t | -1
virtualxid | | | | | 10/88 | | | | | 10/88 | 118067 | ExclusiveLock | t | t | 2868 | t | -1
virtualxid | | | | | 7/7539 | | | | | 7/7539 | 117818 | ExclusiveLock | t | t | 2862 | t | -1
relation | 178295 | 178312 | | | | | | | | 7/7539 | 117818 | RowExclusiveLock | t | f | 2862 | t | -1
relation | 178295 | 178312 | | | | | | | | 7/7539 | 117818 | ShareUpdateExclusiveLock | t | f | 2862 | t | -1
relation | 178295 | 178312 | | | | | | | | 7/7539 | 117818 | ShareLock | t | f | 2862 | t | -1
relation | 178295 | 178312 | | | | | | | | 7/7539 | 117818 | AccessExclusiveLock | t | f | 2862 | t | -1
transactionid | | | | | | 28842 | | | | 5/7953 | 117816 | ExclusiveLock | t | f | 2861 | t | -1
transactionid | | | | | | 28843 | | | | 7/7539 | 117818 | ExclusiveLock | t | f | 2862 | t | -1
relation | 178295 | 178304 | | | | | | | | 5/7953 | 117816 | RowExclusiveLock | t | f | 2861 | t | -1
relation | 178295 | 178304 | | | | | | | | 5/7953 | 117816 | ShareUpdateExclusiveLock | t | f | 2861 | t | -1
relation | 178295 | 178304 | | | | | | | | 5/7953 | 117816 | ShareLock | t | f | 2861 | t | -1
relation | 178295 | 178304 | | | | | | | | 5/7953 | 117816 | AccessExclusiveLock | t | f | 2861 | t | -1
relation | 178295 | 178296 | | | | | | | | 5/7953 | 117816 | ShareUpdateExclusiveLock | t | f | 2861 | t | -1
relation | 178295 | 178296 | | | | | | | | 7/7539 | 117818 | ShareUpdateExclusiveLock | f | f | 2862 | t | -1
userlock | 57005 | | | | | | 57005 | 57005 | 221 | 7/7539 | 117818 | AccessShareLock | t | f | 2862 | t | -1
userlock | 57005 | | | | | | 57005 | 57005 | 221 | 5/7953 | 117816 | AccessShareLock | t | f | 2861 | t | -1
virtualxid | | | | | 4/9852 | | | | | 4/9852 | 117830 | ExclusiveLock | t | t | 2862 | t | 0
virtualxid | | | | | 1/3825 | | | | | 1/3825 | 118069 | ExclusiveLock | t | t | 2868 | t | 0
relation | 178295 | 178308 | | | | | | | | 2/11685 | 117824 | AccessShareLock | t | t | 2861 | t | 0
relation | 178295 | 178299 | | | | | | | | 2/11685 | 117824 | AccessShareLock | t | t | 2861 | t | 0
relation | 178295 | 178296 | | | | | | | | 2/11685 | 117824 | AccessShareLock | t | t | 2861 | t | 0
virtualxid | | | | | 2/11685 | | | | | 2/11685 | 117824 | ExclusiveLock | t | t | 2861 | t | 0
relation | 178295 | 178304 | | | | | | | | 2/11685 | 117824 | AccessShareLock | t | f | 2861 | t | 0
relation | 178295 | 178304 | | | | | | | | 2/11685 | 117824 | RowExclusiveLock | t | f | 2861 | t | 0
relation | 178295 | 178304 | | | | | | | | 2/11685 | 117824 | ShareLock | t | f | 2861 | t | 0
relation | 178295 | 178304 | | | | | | | | 2/11685 | 117824 | AccessExclusiveLock | t | f | 2861 | t | 0
relation | 178295 | 178312 | | | | | | | | 2/11685 | 117824 | AccessShareLock | f | f | 2861 | t | 0
relation | 178295 | 178312 | | | | | | | | 4/9852 | 117830 | RowExclusiveLock | t | f | 2862 | t | 0
relation | 178295 | 178312 | | | | | | | | 4/9852 | 117830 | ShareLock | t | f | 2862 | t | 0
relation | 178295 | 178312 | | | | | | | | 4/9852 | 117830 | AccessExclusiveLock | t | f | 2862 | t | 0
transactionid | | | | | | 41656 | | | | 2/11685 | 117824 | ExclusiveLock | t | f | 2861 | t | 0
transactionid | | | | | | 41655 | | | | 4/9852 | 117830 | ExclusiveLock | t | f | 2862 | t | 0
relation | 178295 | 178308 | | | | | | | | 2/9610 | 117825 | AccessShareLock | t | t | 2861 | t | 1
relation | 178295 | 178299 | | | | | | | | 2/9610 | 117825 | AccessShareLock | t | t | 2861 | t | 1
relation | 178295 | 178296 | | | | | | | | 2/9610 | 117825 | AccessShareLock | t | t | 2861 | t | 1
virtualxid | | | | | 2/9610 | | | | | 2/9610 | 117825 | ExclusiveLock | t | t | 2861 | t | 1
virtualxid | | | | | 4/9525 | | | | | 4/9525 | 117831 | ExclusiveLock | t | t | 2862 | t | 1
virtualxid | | | | | 1/5397 | | | | | 1/5397 | 118071 | ExclusiveLock | t | t | 2868 | t | 1
transactionid | | | | | | 41687 | | | | 2/9610 | 117825 | ExclusiveLock | t | f | 2861 | t | 1
relation | 178295 | 178312 | | | | | | | | 2/9610 | 117825 | AccessShareLock | f | f | 2861 | t | 1
transactionid | | | | | | 41686 | | | | 4/9525 | 117831 | ExclusiveLock | t | f | 2862 | t | 1
relation | 178295 | 178304 | | | | | | | | 2/9610 | 117825 | AccessShareLock | t | f | 2861 | t | 1
relation | 178295 | 178304 | | | | | | | | 2/9610 | 117825 | RowExclusiveLock | t | f | 2861 | t | 1
relation | 178295 | 178304 | | | | | | | | 2/9610 | 117825 | ShareLock | t | f | 2861 | t | 1
relation | 178295 | 178304 | | | | | | | | 2/9610 | 117825 | AccessExclusiveLock | t | f | 2861 | t | 1
relation | 178295 | 178312 | | | | | | | | 4/9525 | 117831 | RowExclusiveLock | t | f | 2862 | t | 1
relation | 178295 | 178312 | | | | | | | | 4/9525 | 117831 | ShareLock | t | f | 2862 | t | 1
relation | 178295 | 178312 | | | | | | | | 4/9525 | 117831 | AccessExclusiveLock | t | f | 2862 | t | 1
virtualxid | | | | | 1/5393 | | | | | 1/5393 | 118070 | ExclusiveLock | t | t | 2868 | t | 2
virtualxid | | | | | 4/9526 | | | | | 4/9526 | 117832 | ExclusiveLock | t | t | 2862 | t | 2
relation | 178295 | 178308 | | | | | | | | 2/9543 | 117826 | AccessShareLock | t | t | 2861 | t | 2
relation | 178295 | 178299 | | | | | | | | 2/9543 | 117826 | AccessShareLock | t | t | 2861 | t | 2
relation | 178295 | 178296 | | | | | | | | 2/9543 | 117826 | AccessShareLock | t | t | 2861 | t | 2
virtualxid | | | | | 2/9543 | | | | | 2/9543 | 117826 | ExclusiveLock | t | t | 2861 | t | 2
relation | 178295 | 178312 | | | | | | | | 2/9543 | 117826 | AccessShareLock | f | f | 2861 | t | 2
transactionid | | | | | | 41855 | | | | 4/9526 | 117832 | ExclusiveLock | t | f | 2862 | t | 2
relation | 178295 | 178304 | | | | | | | | 2/9543 | 117826 | AccessShareLock | t | f | 2861 | t | 2
relation | 178295 | 178304 | | | | | | | | 2/9543 | 117826 | RowExclusiveLock | t | f | 2861 | t | 2
relation | 178295 | 178304 | | | | | | | | 2/9543 | 117826 | ShareLock | t | f | 2861 | t | 2
relation | 178295 | 178304 | | | | | | | | 2/9543 | 117826 | AccessExclusiveLock | t | f | 2861 | t | 2
transactionid | | | | | | 41854 | | | | 2/9543 | 117826 | ExclusiveLock | t | f | 2861 | t | 2
relation | 178295 | 178312 | | | | | | | | 4/9526 | 117832 | RowExclusiveLock | t | f | 2862 | t | 2
relation | 178295 | 178312 | | | | | | | | 4/9526 | 117832 | ShareLock | t | f | 2862 | t | 2
relation | 178295 | 178312 | | | | | | | | 4/9526 | 117832 | AccessExclusiveLock | t | f | 2862 | t | 2
(66 rows)

gpadmin 117818 33176 0 15:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(21208) con2862 cmd10 COPY waiting

(gdb) bt
#0 0x00007f58559cfb47 in semop () from /lib64/libc.so.6
#1 0x00000000008bec73 in PGSemaphoreLock (sema=0x7f584a08d240, interruptOK=1 '\001') at pg_sema.c:422
#2 0x000000000094e901 in ProcSleep (locallock=0x2e95920, lockMethodTable=0xdc0560 <default_lockmethod>) at proc.c:1338
#3 0x00000000009477cf in WaitOnLock (locallock=0x2e95920, owner=0x2edfeb0) at lock.c:1849
#4 0x000000000094654e in LockAcquireExtended (locktag=0x7ffc56136400, lockmode=4, sessionLock=0 '\000', dontWait=0 '\000', reportMemoryError=1 '\001',
locallockp=0x7ffc561363f8) at lock.c:1151
#5 0x00000000009438d5 in LockRelationOid (relid=178296, lockmode=4) at lmgr.c:102
#6 0x00000000004df9ab in try_relation_open (relationId=178296, lockmode=4, noWait=0 '\000') at heapam.c:1126
#7 0x000000000066e603 in analyze_rel_internal (relid=178296, vacstmt=0x2f21390, in_outer_xact=1 '\001', bstrategy=0x2f5d318) at analyze.c:269
#8 0x000000000066e48e in analyze_rel (relid=178296, vacstmt=0x2f21390, in_outer_xact=1 '\001', bstrategy=0x2f5d318) at analyze.c:223
#9 0x000000000071a144 in vacuum (vacstmt=0x2f21390, relid=0, do_toast=0 '\000', bstrategy=0x2f5d318, for_wraparound=0 '\000', isTopLevel=0 '\000') at vacuum.c:387
#10 0x00000000008e07b9 in autostats_issue_analyze (relationOid=178312) at autostats.c:74
#11 0x00000000008e0d3b in auto_stats (cmdType=AUTOSTATS_CMDTYPE_COPY, relationOid=178312, ntuples=1, inFunction=0 '\000') at autostats.c:324
#12 0x0000000000683f37 in DoCopy (stmt=0x2e89560,
queryString=0x2e88698 "COPY \"public\".\"sales_1_prt_apr17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 7fdd09be-a80f-46d0-8c2c-e0d771e81bbd' ON SEGMENT CSV", processed=0x7ffc56136a30) at copy.c:1173
#13 0x000000000096dd01 in standard_ProcessUtility (parsetree=0x2e89560,
queryString=0x2e88698 "COPY \"public\".\"sales_1_prt_apr17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 7fdd09be-a80f-46d0-8c2c-e0d771e81bbd' ON SEGMENT CSV", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0x2e89948, completionTag=0x7ffc56136d80 "") at utility.c:634
#14 0x000000000096d677 in ProcessUtility (parsetree=0x2e89560,
queryString=0x2e88698 "COPY \"public\".\"sales_1_prt_apr17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 7fdd09be-a80f-46d0-8c2c-e0d771e81bbd' ON SEGMENT CSV", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0x2e89948, completionTag=0x7ffc56136d80 "") at utility.c:373
#15 0x000000000096c44a in PortalRunUtility (portal=0x2e90318, utilityStmt=0x2e89560, isTopLevel=1 '\001', dest=0x2e89948, completionTag=0x7ffc56136d80 "")
at pquery.c:1379
#16 0x000000000096c6db in PortalRunMulti (portal=0x2e90318, isTopLevel=1 '\001', dest=0x2e89948, altdest=0x2e89948, completionTag=0x7ffc56136d80 "")
at pquery.c:1510
#17 0x000000000096bbd2 in PortalRun (portal=0x2e90318, count=9223372036854775807, isTopLevel=1 '\001', dest=0x2e89948, altdest=0x2e89948,
completionTag=0x7ffc56136d80 "") at pquery.c:1016

#18 0x00000000009640d2 in exec_simple_query (

query_string=0x2e88698 "COPY \"public\".\"sales_1_prt_apr17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 7fdd09be-a80f-46d0-8c2c-e0d771e81bbd' ON SEGMENT CSV") at postgres.c:1798
#19 0x0000000000968e8c in PostgresMain (argc=1, argv=0x2e68248, dbname=0x2e67ff0 "testdb", username=0x2e68190 "gpadmin") at postgres.c:5215
#20 0x00000000008d7c40 in BackendRun (port=0x2e90040) at postmaster.c:4803
#21 0x00000000008d72e0 in BackendStartup (port=0x2e90040) at postmaster.c:4460

#22 0x00000000008d32ab in ServerLoop () at postmaster.c:1945

#23 0x00000000008d27fd in PostmasterMain (argc=6, argv=0x2e66040) at postmaster.c:1515
#24 0x00000000007cdc32 in main (argc=6, argv=0x2e66040) at main.c:245

postgres=# select locktype,relation,pid,mode,granted from pg_locks where pid = 117818;
locktype | relation | pid | mode | granted
---------------+----------+--------+--------------------------+---------
virtualxid | | 117818 | ExclusiveLock | t
relation | 178312 | 117818 | RowExclusiveLock | t
relation | 178312 | 117818 | ShareUpdateExclusiveLock | t
relation | 178312 | 117818 | ShareLock | t
relation | 178312 | 117818 | AccessExclusiveLock | t
transactionid | | 117818 | ExclusiveLock | t
relation | 178296 | 117818 | ShareUpdateExclusiveLock | f
userlock | | 117818 | AccessShareLock | t
(8 rows)

ShareUpdateExclusiveLock On sales (which is caused by vacuume()/analyze())

testdb=# select locktype,relation,pid,mode,granted,gp_segment_id from pg_locks where relation =178296 ;
locktype | relation | pid | mode | granted | gp_segment_id
----------+----------+--------+--------------------------+---------+---------------
relation | 178296 | 117816 | ShareUpdateExclusiveLock | t | -1
relation | 178296 | 117818 | ShareUpdateExclusiveLock | f | -1
relation | 178296 | 117824 | AccessShareLock | t | 0
relation | 178296 | 117825 | AccessShareLock | t | 1
relation | 178296 | 117826 | AccessShareLock | t | 2
(5 rows)

gpadmin 117816 33176 0 15:37 ? 00:00:00 postgres: 6000, gpadmin testdb 192.168.2.11(21206) con2861 cmd16 COPY

(gdb) bt
#0 0x00007f58559c3a3d in poll () from /lib64/libc.so.6
#1 0x0000000000b88524 in checkDispatchResult (ds=0x2f1e318, wait=1 '\001') at cdbdisp_async.c:486
#2 0x0000000000b8817b in cdbdisp_checkDispatchResult_async (ds=0x2f1e318, waitMode=DISPATCH_WAIT_NONE) at cdbdisp_async.c:333
#3 0x0000000000b87007 in cdbdisp_checkDispatchResult (ds=0x2f1e318, waitMode=DISPATCH_WAIT_NONE) at cdbdisp.c:108
#4 0x0000000000b8c028 in cdbdisp_dispatchCommandInternal (pQueryParms=0x2f9ef48, flags=4, segments=0x2faffe0, cdb_pgresults=0x7ffc56136020) at cdbdisp_query.c:446

#5 0x0000000000b8be59 in CdbDispatchCommandToSegments (

#6 0x0000000000b8bda8 in CdbDispatchCommand (

strCommand=0x2fb0018 "select * from pg_catalog.gp_acquire_sample_rows(178296, 10000, 't') as (totalrows pg_catalog.float8, totaldeadrows pg_catalog.float8, oversized_cols_bitmap pg_catalog.text, id integer, date date, amt "..., flags=4, cdb_pgresults=0x7ffc56136020) at cdbdisp_query.c:338
#7 0x0000000000672e1e in acquire_sample_rows_dispatcher (onerel=0x7f585766d7c8, inh=1 '\001', elevel=13, rows=0x2fd4338, targrows=30000, totalrows=0x7ffc56136308,
totaldeadrows=0x7ffc56136300) at analyze.c:2442
#8 0x0000000000671f8a in acquire_inherited_sample_rows (onerel=0x7f585766d7c8, elevel=13, rows=0x2fd4338, targrows=30000, totalrows=0x7ffc56136308,
totaldeadrows=0x7ffc56136300) at analyze.c:2014
#9 0x000000000066f49f in do_analyze_rel (onerel=0x7f585766d7c8, vacstmt=0x2faa800, acquirefunc=0x6716ff <acquire_sample_rows>, relpages=0, inh=1 '\001',

in_outer_xact=1 '\001', elevel=13) at analyze.c:679

#10 0x000000000066eade in analyze_rel_internal (relid=178296, vacstmt=0x2faa800, in_outer_xact=1 '\001', bstrategy=0x2f8ba48) at analyze.c:405
#11 0x000000000066e48e in analyze_rel (relid=178296, vacstmt=0x2faa800, in_outer_xact=1 '\001', bstrategy=0x2f8ba48) at analyze.c:223
#12 0x000000000071a144 in vacuum (vacstmt=0x2faa800, relid=0, do_toast=0 '\000', bstrategy=0x2f8ba48, for_wraparound=0 '\000', isTopLevel=0 '\000') at vacuum.c:387
#13 0x00000000008e07b9 in autostats_issue_analyze (relationOid=178304) at autostats.c:74
#14 0x00000000008e0d3b in auto_stats (cmdType=AUTOSTATS_CMDTYPE_COPY, relationOid=178304, ntuples=1, inFunction=0 '\000') at autostats.c:324
#15 0x0000000000683f37 in DoCopy (stmt=0x2e89560,
queryString=0x2e88698 "COPY \"public\".\"sales_1_prt_feb17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 3fef6efb-af36-4f6e-8adc-ec103e110a26' ON SEGMENT CSV", processed=0x7ffc56136a30) at copy.c:1173
#16 0x000000000096dd01 in standard_ProcessUtility (parsetree=0x2e89560,
queryString=0x2e88698 "COPY \"public\".\"sales_1_prt_feb17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 3fef6efb-af36-4f6e-8adc-ec103e110a26' ON SEGMENT CSV", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0x2e89948, completionTag=0x7ffc56136d80 "") at utility.c:634
#17 0x000000000096d677 in ProcessUtility (parsetree=0x2e89560,
queryString=0x2e88698 "COPY \"public\".\"sales_1_prt_feb17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 3fef6efb-af36-4f6e-8adc-ec103e110a26' ON SEGMENT CSV", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, dest=0x2e89948, completionTag=0x7ffc56136d80 "") at utility.c:373
#18 0x000000000096c44a in PortalRunUtility (portal=0x2f33b28, utilityStmt=0x2e89560, isTopLevel=1 '\001', dest=0x2e89948, completionTag=0x7ffc56136d80 "")
at pquery.c:1379
#19 0x000000000096c6db in PortalRunMulti (portal=0x2f33b28, isTopLevel=1 '\001', dest=0x2e89948, altdest=0x2e89948, completionTag=0x7ffc56136d80 "")
at pquery.c:1510
#20 0x000000000096bbd2 in PortalRun (portal=0x2f33b28, count=9223372036854775807, isTopLevel=1 '\001', dest=0x2e89948, altdest=0x2e89948,
completionTag=0x7ffc56136d80 "") at pquery.c:1016

#21 0x00000000009640d2 in exec_simple_query (

query_string=0x2e88698 "COPY \"public\".\"sales_1_prt_feb17\" FROM PROGRAM 'gpcopy_helper --listen --seg-id <SEGID> --cmd-id 3fef6efb-af36-4f6e-8adc-ec103e110a26' ON SEGMENT CSV") at postgres.c:1798
#22 0x0000000000968e8c in PostgresMain (argc=1, argv=0x2e68248, dbname=0x2e680f0 "testdb", username=0x2e680d0 "gpadmin") at postgres.c:5215
#23 0x00000000008d7c40 in BackendRun (port=0x2e90040) at postmaster.c:4803
#24 0x00000000008d72e0 in BackendStartup (port=0x2e90040) at postmaster.c:4460

#25 0x00000000008d32ab in ServerLoop () at postmaster.c:1945

#26 0x00000000008d27fd in PostmasterMain (argc=6, argv=0x2e66040) at postmaster.c:1515
#27 0x00000000007cdc32 in main (argc=6, argv=0x2e66040) at main.c:245

gpadmin 117824 33138 0 15:37 ? 00:00:00 postgres: 6002, gpadmin testdb 192.168.2.12(46258) con2861 seg0 cmd16 SELECT waiting

(gdb) bt
#0 0x00007ff57bcf5b47 in semop () from /lib64/libc.so.6
#1 0x00000000008bec73 in PGSemaphoreLock (sema=0x7ff56edeb5e0, interruptOK=1 '\001') at pg_sema.c:422
#2 0x000000000094e901 in ProcSleep (locallock=0x19c5f60, lockMethodTable=0xdc0560 <default_lockmethod>) at proc.c:1338
#3 0x00000000009477cf in WaitOnLock (locallock=0x19c5f60, owner=0x19c0ad8) at lock.c:1849
#4 0x000000000094654e in LockAcquireExtended (locktag=0x7ffc6498b8d0, lockmode=1, sessionLock=0 '\000', dontWait=0 '\000', reportMemoryError=1 '\001',
locallockp=0x7ffc6498b8c8) at lock.c:1151
#5 0x00000000009438d5 in LockRelationOid (relid=178312, lockmode=1) at lmgr.c:102
#6 0x00000000005e1a85 in find_inheritance_children (parentrelId=178296, lockmode=1) at pg_inherits.c:121
#7 0x00000000005e1c4b in find_all_inheritors (parentrelId=178296, lockmode=1, numparents=0x0) at pg_inherits.c:211
#8 0x0000000000671faa in acquire_inherited_sample_rows (onerel=0x1a55508, elevel=14, rows=0x1a878a8, targrows=10000, totalrows=0x7ffc6498bb30,
totaldeadrows=0x7ffc6498bb28) at analyze.c:2024
#9 0x00000000007259c8 in gp_acquire_sample_rows (fcinfo=0x7ffc6498bd60) at analyzefuncs.c:202
#10 0x0000000000743002 in ExecMakeTableFunctionResult (funcexpr=0x1a7c6d0, econtext=0x1a7bf80, argContext=0x19c1280, expectedDesc=0x1a7d040, randomAccess=0 '\000',
operatorMemKB=32768) at execQual.c:2337
#11 0x000000000077531b in FunctionNext_guts (node=0x1a7bb10) at nodeFunctionscan.c:103
#12 0x0000000000775925 in FunctionNext (node=0x1a7bb10) at nodeFunctionscan.c:296
#13 0x000000000074cf1f in ExecScanFetch (node=0x1a7bb10, accessMtd=0x77590d <FunctionNext>, recheckMtd=0x77595a <FunctionRecheck>) at execScan.c:84
#14 0x000000000074cf8a in ExecScan (node=0x1a7bb10, accessMtd=0x77590d <FunctionNext>, recheckMtd=0x77595a <FunctionRecheck>) at execScan.c:132
#15 0x000000000077598f in ExecFunctionScan (node=0x1a7bb10) at nodeFunctionscan.c:326
#16 0x000000000073e12c in ExecProcNode (node=0x1a7bb10) at execProcnode.c:1043
#17 0x00000000007387d2 in ExecutePlan (estate=0x1a7b828, planstate=0x1a7bb10, operation=CMD_SELECT, sendTuples=1 '\001', numberTuples=0,
direction=ForwardScanDirection, dest=0x1a77090) at execMain.c:2979
#18 0x0000000000735049 in standard_ExecutorRun (queryDesc=0x1a66b28, direction=ForwardScanDirection, count=0) at execMain.c:968
#19 0x0000000000734cc1 in ExecutorRun (queryDesc=0x1a66b28, direction=ForwardScanDirection, count=0) at execMain.c:834
#20 0x000000000096be92 in PortalRunSelect (portal=0x1a346f8, forward=1 '\001', count=0, dest=0x1a77090) at pquery.c:1149
#21 0x000000000096bb08 in PortalRun (portal=0x1a346f8, count=9223372036854775807, isTopLevel=1 '\001', dest=0x1a77090, altdest=0x1a77090,
completionTag=0x7ffc6498c870 "") at pquery.c:990

#22 0x00000000009640d2 in exec_simple_query (

query_string=0x1a29de4 "select * from pg_catalog.gp_acquire_sample_rows(178296, 10000, 't') as (totalrows pg_catalog.float8, totaldeadrows pg_catalog.float8, oversized_cols_bitmap pg_catalog.text, id integer, date date, amt "...) at postgres.c:1798
#23 0x00000000009693b7 in PostgresMain (argc=1, argv=0x19989b8, dbname=0x1998890 "testdb", username=0x1998870 "gpadmin") at postgres.c:5357
#24 0x00000000008d7c40 in BackendRun (port=0x19c8b40) at postmaster.c:4803
#25 0x00000000008d72e0 in BackendStartup (port=0x19c8b40) at postmaster.c:4460

#26 0x00000000008d32ab in ServerLoop () at postmaster.c:1945

#27 0x00000000008d27fd in PostmasterMain (argc=5, argv=0x19968e0) at postmaster.c:1515
#28 0x00000000007cdc32 in main (argc=5, argv=0x19968e0) at main.c:245

postgres=# select locktype,relation,pid,mode,granted,gp_segment_id from pg_locks where relation =178312 ;
locktype | relation | pid | mode | granted | gp_segment_id
----------+----------+--------+--------------------------+---------+---------------
relation | 178312 | 117818 | RowExclusiveLock | t | -1
relation | 178312 | 117818 | ShareUpdateExclusiveLock | t | -1
relation | 178312 | 117818 | ShareLock | t | -1
relation | 178312 | 117818 | AccessExclusiveLock | t | -1
relation | 178312 | 117824 | AccessShareLock | f | 0
relation | 178312 | 117830 | RowExclusiveLock | t | 0
relation | 178312 | 117830 | ShareLock | t | 0
relation | 178312 | 117830 | AccessExclusiveLock | t | 0
relation | 178312 | 117825 | AccessShareLock | f | 1
relation | 178312 | 117831 | RowExclusiveLock | t | 1
relation | 178312 | 117831 | ShareLock | t | 1
relation | 178312 | 117831 | AccessExclusiveLock | t | 1
relation | 178312 | 117826 | AccessShareLock | f | 2
relation | 178312 | 117832 | RowExclusiveLock | t | 2
relation | 178312 | 117832 | ShareLock | t | 2
relation | 178312 | 117832 | AccessExclusiveLock | t | 2
(16 rows)

gpadmin 117830 33138 0 15:37 ? 00:00:00 postgres: 6002, gpadmin testdb 192.168.2.12(46264) con2862 seg0 idle in transaction

1) This process is idle, the bt output is useless.
2) search con2862 in the pg_log on the QD, it just executed:

Reply all

Reply to author

Forward

0 new messages