Bulk Load
7 views
Skip to first unread message
Nirmal Ramalingam
Jul 29, 2011, 6:35:45 PM7/29/11
to denver-infor...@googlegroups.com
'Wanted to see if bulk load actually gives better performance and apparently it's not much or for at least the test I ran. I don't know what I am doing wrong. There are no keys or indexes on the table. The mapping has just the source to a dummy exp to the target. Adjusted my commit interval too.
Source Table: Stage.LATIS_NME_HDR
Rows: 5,254,504
Columns: 102
Test 1:
Normal Load:
Commit Interval: 10,000 rows
Total run time : 4 mins 27 secs
Test 2:
Bulk Load:
Commit Interval: 10,000 rows
Total run time : 27 mins 39secs
Thread [READER_1_1_1] created for [the read stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [1539.169077] secs
Total Idle Time = [1376.145748] secs
Busy Percentage = [10.591645]
Thread [TRANSF_1_1_1] created for [the transformation stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [1538.764069] secs
Total Idle Time = [1524.835837] secs
Busy Percentage = [0.905157]
Transformation-specific statistics for this thread were not accurate enough to report.
Thread [WRITER_1_*_1] created for [the write stage] of partition point [LATIS_NME_HDR_TEST] has completed.
Total Run Time = [1619.397935] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
Total Run Time = [1539.169077] secs
Total Idle Time = [1376.145748] secs
Busy Percentage = [10.591645]
Thread [TRANSF_1_1_1] created for [the transformation stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [1538.764069] secs
Total Idle Time = [1524.835837] secs
Busy Percentage = [0.905157]
Transformation-specific statistics for this thread were not accurate enough to report.
Thread [WRITER_1_*_1] created for [the write stage] of partition point [LATIS_NME_HDR_TEST] has completed.
Total Run Time = [1619.397935] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
Test 3:
Bulk Load:
Commit Interval: 250,000 rows
Total run time : 17 mins 8 secs
Thread [READER_1_1_1] created for [the read stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [1059.255310] secs
Total Idle Time = [888.538447] secs
Busy Percentage = [16.116687]
Thread [TRANSF_1_1_1] created for [the transformation stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [1059.158673] secs
Total Idle Time = [1044.036955] secs
Busy Percentage = [1.427710]
Transformation-specific statistics for this thread were not accurate enough to report.
Thread [WRITER_1_*_1] created for [the write stage] of partition point [LATIS_NME_HDR_TEST] has completed.
Total Run Time = [1069.631083] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
Total Run Time = [1059.255310] secs
Total Idle Time = [888.538447] secs
Busy Percentage = [16.116687]
Thread [TRANSF_1_1_1] created for [the transformation stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [1059.158673] secs
Total Idle Time = [1044.036955] secs
Busy Percentage = [1.427710]
Transformation-specific statistics for this thread were not accurate enough to report.
Thread [WRITER_1_*_1] created for [the write stage] of partition point [LATIS_NME_HDR_TEST] has completed.
Total Run Time = [1069.631083] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
Test 4:
Bulk Load:
Commit Interval: 1,000,000 rows
Total run time : 3 mins 35 secs
***** RUN INFO FOR TGT LOAD ORDER GROUP [1], CONCURRENT SET [1] *****
Thread [READER_1_1_1] created for [the read stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [178.921643] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
Thread [TRANSF_1_1_1] created for [the transformation stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [178.903216] secs
Total Idle Time = [163.736273] secs
Busy Percentage = [8.477736]
Transformation-specific statistics for this thread were not accurate enough to report.
Thread [WRITER_1_*_1] created for [the write stage] of partition point [LATIS_NME_HDR_TEST] has completed.
Total Run Time = [205.680358] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
Thread [READER_1_1_1] created for [the read stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [178.921643] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
Thread [TRANSF_1_1_1] created for [the transformation stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [178.903216] secs
Total Idle Time = [163.736273] secs
Busy Percentage = [8.477736]
Transformation-specific statistics for this thread were not accurate enough to report.
Thread [WRITER_1_*_1] created for [the write stage] of partition point [LATIS_NME_HDR_TEST] has completed.
Total Run Time = [205.680358] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
Test 5:
Bulk Load:
Commit Interval: 2,000,000 rows
Total run time : 3 mins 33 secs
INFO 7/29/2011 4:21:29 PM node_01 MANAGER PETL_24031
***** RUN INFO FOR TGT LOAD ORDER GROUP [1], CONCURRENT SET [1] *****
Thread [READER_1_1_1] created for [the read stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [185.219079] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
Thread [TRANSF_1_1_1] created for [the transformation stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [185.202346] secs
Total Idle Time = [170.115423] secs
Busy Percentage = [8.146183]
Transformation-specific statistics for this thread were not accurate enough to report.
Thread [WRITER_1_*_1] created for [the write stage] of partition point [LATIS_NME_HDR_TEST] has completed.
Total Run Time = [205.935426] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
***** RUN INFO FOR TGT LOAD ORDER GROUP [1], CONCURRENT SET [1] *****
Thread [READER_1_1_1] created for [the read stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [185.219079] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
Thread [TRANSF_1_1_1] created for [the transformation stage] of partition point [SQ_Shortcut_to_LATIS_NME_HDR] has completed.
Total Run Time = [185.202346] secs
Total Idle Time = [170.115423] secs
Busy Percentage = [8.146183]
Transformation-specific statistics for this thread were not accurate enough to report.
Thread [WRITER_1_*_1] created for [the write stage] of partition point [LATIS_NME_HDR_TEST] has completed.
Total Run Time = [205.935426] secs
Total Idle Time = [0.000000] secs
Busy Percentage = [100.000000]
We are getting about 1 minute less run time if we tweak the commit interval. Seems like that is not much of savings.
Reply all
Reply to author
Forward
0 new messages