New to MR3
kauts shukla
Sungwoo Park
If on K8s, what does the command print (where you should use your DAGAppMaster Pod name)?
kubectl log -n hivemr3 mr3master-8498-0-fftrj -f | grep -e Scale -e Scaling -e average kauts shukla
kubectl logs -n hivemr3 mr3master-4925-0 -f --tail 100 | grep -e Scale -e Scaling -e average2020-07-19T10:07:59,742 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (1024MB / 4096MB)2020-07-19T10:07:59,742 INFO [All-In-One] AMHostTracker: Checking for Scale-in: 1 false 3 1 52020-07-19T10:08:09,742 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (1024MB / 4096MB)2020-07-19T10:08:09,742 INFO [All-In-One] AMHostTracker: Checking for Scale-in: 1 false 3 1 52020-07-19T10:08:19,742 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (1024MB / 4096MB)2020-07-19T10:08:19,742 INFO [All-In-One] AMHostTracker: Checking for Scale-in: 1 false 3 1 52020-07-19T10:08:29,742 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (1024MB / 4096MB)2020-07-19T10:08:29,742 INFO [All-In-One] AMHostTracker: Checking for Scale-in: 1 false 3 1 52020-07-19T10:08:39,742 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (1024MB / 4096MB)2020-07-19T10:08:39,742 INFO [All-In-One] AMHostTracker: Checking for Scale-in: 1 false 3 1 5
kauts shukla
kubectl -n hivemr3 get podsNAME READY STATUS RESTARTS AGEhivemr3-hiveserver2-pntj6 1/1 Running 0 47mhivemr3-metastore-0 1/1 Running 0 48mmr3master-4925-0 1/1 Running 0 45mmr3worker-60ce-1 1/1 Running 0 45m
Sungwoo Park
Vikas Rana
These are our observation:
On setting mr3.enable.auto.scaling to false. We are seeing new PODs were getting launch by the MR3_Master.
Vikass-MacBook-Pro:hive-mr3 vrana$ kubectl -n hivemr3 logs -f mr3master-4718-0 | grep -e Scale -e Scaling -e average
2020-07-20T16:54:50,066 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:55:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:55:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:55:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:55:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:55:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:55:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:56:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:56:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:56:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:56:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:56:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-20T16:56:50,064 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-20T16:57:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-20T16:57:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-20T16:57:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-20T16:57:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-20T16:57:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-20T16:57:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-20T16:58:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-20T16:58:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-20T16:58:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-20T16:58:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-20T16:58:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3328MB / 13312MB)
2020-07-20T16:58:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3584MB / 14336MB)
2020-07-20T16:59:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3840MB / 15360MB)
2020-07-20T16:59:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (4096MB / 16384MB)
2020-07-20T16:59:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 23.5% (4096MB / 17408MB)
2020-07-20T16:59:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 22.2% (4096MB / 18432MB)
2020-07-20T16:59:40,064 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 21.1% (4096MB / 19456MB)
2020-07-20T16:59:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 20.0% (4096MB / 20480MB)
2020-07-20T17:00:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 19.0% (4096MB / 21504MB)
2020-07-20T17:00:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 18.2% (4096MB / 22528MB)
2020-07-20T17:00:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 17.4% (4096MB / 23552MB)
2020-07-20T17:00:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 16.7% (4096MB / 24576MB)
2020-07-20T17:00:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 15.6% (3840MB / 24576MB)
2020-07-20T17:00:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 15.3% (3754MB / 24576MB)
2020-07-20T17:01:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 14.9% (3669MB / 24576MB)
2020-07-20T17:01:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 14.6% (3584MB / 24576MB)
2020-07-20T17:01:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 16.0% (4096MB / 25600MB)
2020-07-20T17:01:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 17.3% (4608MB / 26624MB)
2020-07-20T17:01:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 18.5% (5120MB / 27648MB)
2020-07-20T17:01:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 19.3% (5546MB / 28672MB)
2020-07-20T17:02:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 20.1% (5973MB / 29696MB)
2020-07-20T17:02:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 21.1% (6485MB / 30720MB)
2020-07-20T17:02:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 20.4% (6485MB / 31744MB)
2020-07-20T17:02:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 19.8% (6485MB / 32768MB)
2020-07-20T17:02:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 19.2% (6485MB / 33792MB)
2020-07-20T17:02:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 18.1% (6314MB / 34816MB)
2020-07-20T17:03:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 17.1% (6144MB / 35840MB)
2020-07-20T17:03:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 16.2% (5973MB / 36864MB)
2020-07-20T17:03:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 15.1% (5717MB / 37888MB)
2020-07-20T17:03:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 14.0% (5461MB / 38912MB)
2020-07-20T17:03:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 13.0% (5205MB / 39936MB)
2020-07-20T17:03:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 12.3% (5034MB / 40960MB)
2020-07-20T17:04:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 11.6% (4864MB / 41984MB)
2020-07-20T17:04:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 10.7% (4608MB / 43008MB)
2020-07-20T17:04:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 12.0% (5290MB / 44032MB)
2020-07-20T17:04:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 13.3% (5973MB / 45056MB)
2020-07-20T17:04:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 14.4% (6656MB / 46080MB)
2020-07-20T17:04:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 13.8% (6485MB / 47104MB)
2020-07-20T17:05:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 13.1% (6314MB / 48128MB)
2020-07-20T17:05:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 12.5% (6144MB / 49152MB)
2020-07-20T17:05:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 11.6% (5717MB / 49152MB)
2020-07-20T17:05:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 11.1% (5546MB / 50176MB)
2020-07-20T17:05:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 10.5% (5376MB / 51200MB)
2020-07-20T17:05:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 10.0% (5205MB / 52224MB)
2020-07-20T17:06:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 9.5% (5034MB / 53248MB)
2020-07-20T17:06:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 9.0% (4864MB / 54272MB)
2020-07-20T17:06:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 7.7% (4266MB / 55296MB)
2020-07-20T17:06:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 6.5% (3669MB / 56320MB)
2020-07-20T17:06:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 5.7% (3242MB / 57344MB)
2020-07-20T17:06:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 7.2% (4181MB / 58368MB)
2020-07-20T17:07:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 9.1% (5376MB / 59392MB)
2020-07-20T17:07:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 11.0% (6741MB / 61098MB)
2020-07-20T17:07:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 13.0% (8192MB / 63146MB)
2020-07-20T17:07:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 12.8% (8192MB / 64170MB)
2020-07-20T17:07:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 12.6% (8192MB / 65194MB)
2020-07-20T17:07:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 12.4% (8192MB / 66218MB)
2020-07-20T17:08:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 12.2% (8192MB / 67242MB)
2020-07-20T17:08:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 12.0% (8192MB / 68266MB)
2020-07-20T17:08:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 11.8% (8192MB / 69290MB)
2020-07-20T17:08:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 11.8% (8448MB / 71338MB)
2020-07-20T17:08:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 11.6% (8533MB / 73386MB)
2020-07-20T17:08:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 10.7% (8106MB / 75434MB)
2020-07-20T17:09:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 9.6% (7424MB / 77482MB)
2020-07-20T17:09:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 8.3% (6570MB / 78848MB)
2020-07-20T17:09:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 7.3% (5802MB / 79872MB)
2020-07-20T17:09:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 8.5% (6912MB / 80896MB)
2020-07-20T17:09:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 10.2% (8362MB / 81920MB)
2020-07-20T17:09:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 11.8% (9813MB / 82944MB)
2020-07-20T17:10:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 13.4% (11264MB / 83968MB)
2020-07-20T17:10:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 15.0% (12714MB / 84992MB)
2020-07-20T17:10:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 16.5% (14165MB / 86016MB)
2020-07-20T17:10:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 17.9% (15360MB / 86016MB)
2020-07-20T17:10:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 19.2% (16554MB / 86016MB)
2020-07-20T17:10:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 20.6% (17749MB / 86016MB)
2020-07-20T17:11:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 22.0% (18944MB / 86016MB)
2020-07-20T17:11:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 23.4% (20394MB / 87040MB)
2020-07-20T17:11:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 24.6% (21674MB / 88064MB)
2020-07-20T17:11:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (22272MB / 89088MB)
2020-07-20T17:11:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (22528MB / 90112MB)
2020-07-20T17:11:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 24.7% (22528MB / 91136MB)
2020-07-20T17:12:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 22.5% (20736MB / 92160MB)
2020-07-20T17:12:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 20.3% (18944MB / 93184MB)
2020-07-20T17:12:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 18.2% (17152MB / 94208MB)
2020-07-20T17:12:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 16.1% (15360MB / 95232MB)
2020-07-20T17:12:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 14.1% (13568MB / 96256MB)
2020-07-20T17:12:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 12.1% (11776MB / 97280MB)
2020-07-20T17:13:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 10.2% (9984MB / 98304MB)
2020-07-20T17:13:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 8.0% (7936MB / 99328MB)
2020-07-20T17:13:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 5.9% (5888MB / 100352MB)
2020-07-20T17:13:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 3.8% (3840MB / 101376MB)
2020-07-20T17:13:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 1.8% (1792MB / 102400MB)
2020-07-20T17:13:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 103424MB)
2020-07-20T17:14:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 104448MB)
2020-07-20T17:14:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 105472MB)
2020-07-20T17:14:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 106496MB)
2020-07-20T17:14:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 107520MB)
2020-07-20T17:14:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 108544MB)
2020-07-20T17:14:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 109568MB)
2020-07-20T17:15:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:15:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:15:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:15:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:15:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:15:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:16:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:16:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:16:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:16:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:16:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:16:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:17:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:17:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:17:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:17:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:17:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:17:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:18:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:18:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:18:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:18:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:18:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:18:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:19:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:19:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:19:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:19:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:19:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:19:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:20:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T17:20:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
So we have following open questions:
- We are seeing PODs and Node [via cluster_autoscale] getting added to the cluster. But when cluster is ideal for 30+ mins, these newly added pods were not removed
and due to which nodes not getting SCALED_IN. - Can you explain the logs, which show below: Does this represent cluster usage by all MAP and Reduce task in total ?
All-In-One average memory usage = 25.0% (3328MB / 13312MB)
And if I'm correct I don't see cluster usages is not above 25%. Is this optimal? - Is there a way I can scrape MR3 metrics into PROMETHEUS ?
- What does mr3.am.min.cluster.resource.memory.mb and mr3.am.min.cluster.resource.cpu.cores represent?
Previously we haven't set these values and cluster wasn't able to scale beyond 36864MBmr3.am.min.cluster.resource.memory.mb 40960 Min size of memory in MB that DAGAppMaster assumes as the cluster resource when initializing Map Tasks mr3.am.min.cluster.resource.cpu.cores 40 Min number of cores that DAGAppMaster assumes as the cluster resource when initializing Map Tasks [All-In-One] TaskScheduler: All-In-One average memory usage = 11.3% (4181MB / 36864MB)
Newly launched PODs were in pending state, and cluster_autoscaler was not adding node:
kubectl -n hivemr3 get pods
NAME READY STATUS RESTARTS AGE
hivemr3-hiveserver2-p8zjs 1/1 Running 0 21m
CLUSTER_AUTOSCALER:
hivemr3-metastore-0 1/1 Running 0 21m
mr3master-7732-0 1/1 Running 0 17m
mr3worker-27df-1 1/1 Running 0 17m
mr3worker-27df-14 0/1 Pending 0 3m10s
mr3worker-27df-15 0/1 Pending 0 2m55s
mr3worker-27df-2 1/1 Running 0 17m
mr3worker-27df-3 1/1 Running 0 17m
mr3worker-27df-4 1/1 Running 0 17m
mr3worker-27df-5 1/1 Running 0 17m
mr3worker-27df-6 1/1 Running 0 17m
mr3worker-27df-7 1/1 Running 0 14m
mr3worker-27df-8 1/1 Running 0 14m
mr3worker-27df-9 1/1 Running 0 13mI0719 11:15:24.948128 1 scale_up.go:263] Pod hivemr3/mr3worker-27df-16 is unschedulable
But after mr3.am.min.cluster.resource.memory.mb to 81920, mr3.am.min.cluster.resource.cpu.cores to 40. We are seeing CLUSTER was able to scale to 110592MB.
I0719 11:15:24.948135 1 scale_up.go:263] Pod hivemr3/mr3worker-27df-15 is unschedulable
I0719 11:15:24.948177 1 scale_up.go:300] Upcoming 0 nodes
I0719 11:15:24.948212 1 scale_up.go:338] Skipping node group megatron-prod-eks-worker - max size reached
I0719 11:15:24.948220 1 scale_up.go:416] No expansion options
So how does the upper limit to SCALE_OUT is controlled and determine.
We are using m5.xlarge for MATER nodes and m5d.xlarge for WORKER nodes. Set LIMIT MAX=10, MIN=0 amd DESIRED=1 while starting up the cluster in AWS AutoSCaling- We are currently using MR3: mr3project/hive3:1.0, does upgrading to MR3 1.1 help?
- As we are new to MR3, from your DOC it wasn't very clear whats the difference between MR3 HIVE3 and HIVE4.
(Hive 3 with MR3 master and Hive 4 are built with access to Amazon S3.)
- Where can I find change-log from MR3 1.0 to 1.1 ?
- And we have checked your roadmap, there are some exciting new features mark to be release in MR3 1.2, and tentative timeline for that release?
Cheers,
Vikas Rana
One more update on my previous post:
After keeping cluster IDEAL for an hour/60 min. PODs were by itself started SCALING_OUT:
2020-07-20T17:13:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 1.8% (1792MB / 102400MB)
2020-07-20T17:13:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 103424MB)2020-07-20T18:12:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 110592MB)
2020-07-20T18:12:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 106496MB)
2020-07-20T18:12:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 98304MB)
2020-07-20T18:12:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 90112MB)
2020-07-20T18:12:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 81920MB)
2020-07-20T18:13:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 73728MB)
2020-07-20T18:13:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 65536MB)
2020-07-20T18:13:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 57344MB)
2020-07-20T18:13:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 48810MB)
2020-07-20T18:13:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 39594MB)
2020-07-20T18:13:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 30378MB)
2020-07-20T18:14:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 21162MB)
2020-07-20T18:14:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 11946MB)
2020-07-20T18:14:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 6826MB)
2020-07-20T18:14:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 5802MB)
2020-07-20T18:14:40,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 4778MB)
2020-07-20T18:14:50,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 3754MB)
2020-07-20T18:15:00,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 2730MB)
2020-07-20T18:15:10,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 1706MB)
2020-07-20T18:15:20,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 0.0% (0MB / 682MB)
2020-07-20T18:15:30,063 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
From our configs:
grep -A1 scale kubernetes/conf/mr3-site.xml
<name>mr3.auto.scale.out.threshold.percent</name>
<value>80</value>
--
<name>mr3.auto.scale.in.threshold.percent</name>
<value>50</value>
--
<name>mr3.auto.scale.out.grace.period.secs</name>
<value>60</value>
--
<name>mr3.auto.scale.in.delay.after.scale.out.secs</name>
<value>60</value>
--
<name>mr3.auto.scale.in.grace.period.secs</name>
<value>60</value>
--
<name>mr3.auto.scale.in.wait.dag.finished</name>
<value>true</value>
--
<name>mr3.auto.scale.out.num.increment.containers</name>
<value>1</value>
--
<name>mr3.auto.scale.in.num.decrement.hosts</name>
<value>1</value>
--
<name>mr3.auto.scale.in.min.hosts</name>
<value>1</value>Cheers,
On Sunday, July 19, 2020 at 3:08:28 PM UTC+5:30, kauts shukla wrote:
Vikas Rana
We have seen all WORKER nodes were removed.
Only master nodes were left:
Vikass-MacBook-Pro:helm vrana$ kubectl -nhivemr3 get pods
NAME READY STATUS RESTARTS AGE
hivemr3-hiveserver2-b2vqz 1/1 Running 0 123m
hivemr3-metastore-0 1/1 Running 0 124m
mr3master-4718-0 1/1 Running 0 123m
Vikass-MacBook-Pro:helm vrana$ kubectl -nhivemr3 get nodes
NAME STATUS ROLES AGE VERSION
ip-10-111-144-167.eu-central-1.compute.internal Ready <none> 134m v1.16.12-eks-904af05
ip-10-111-147-110.eu-central-1.compute.internal Ready <none> 133m v1.16.12-eks-904af05
Vikass-MacBook-Pro:helm vrana$ kubectl -nhivemr3 describe node ip-10-111-144-167.eu-central-1.compute.internal | grep role
roles=masters
Vikass-MacBook-Pro:helm vrana$ kubectl -nhivemr3 describe node ip-10-111-147-110.eu-central-1.compute.internal | grep role
roles=mastersIdeally we want to keep 1 node [m5d.xlarge, 16GB] and 4 worker POD on it, to keep resources any new incoming query.
Cheers,
On Sunday, July 19, 2020 at 3:08:28 PM UTC+5:30, kauts shukla wrote:
Sungwoo Park
Vikas Rana
Vikass-MacBook-Pro:hive-mr3 vrana$ git diff
diff --git a/kubernetes/conf/hive-site.xml b/kubernetes/conf/hive-site.xml
index fe0577d..ae2749c 100644
--- a/kubernetes/conf/hive-site.xml
+++ b/kubernetes/conf/hive-site.xml
@@ -1052,7 +1052,7 @@
<property>
<name>hive.mr3.map.task.memory.mb</name>
- <value>1024</value>
+ <value>4096</value>
</property>
<property>
@@ -1062,7 +1062,7 @@
<property>
<name>hive.mr3.reduce.task.memory.mb</name>
- <value>1024</value>
+ <value>4096</value>
</property>
<property>
@@ -1072,17 +1072,17 @@
<property>
<name>hive.mr3.all-in-one.containergroup.memory.mb</name>
- <value>4096</value>
+ <value>12288</value>
</property>
<property>
<name>hive.mr3.all-in-one.containergroup.vcores</name>
- <value>1</value>
+ <value>3</value>
</property>
<property>
<name>hive.mr3.map.containergroup.memory.mb</name>
- <value>3850</value>
+ <value>4096</value>
</property>
<property>
@@ -1092,7 +1092,7 @@
<property>
<name>hive.mr3.reduce.containergroup.memory.mb</name>
- <value>3850</value>
+ <value>4096</value>
</property>
<property>
diff --git a/kubernetes/conf/mr3-site.xml b/kubernetes/conf/mr3-site.xml
index a66d7ce..d827f4f 100644
--- a/kubernetes/conf/mr3-site.xml
+++ b/kubernetes/conf/mr3-site.xml
@@ -108,7 +108,7 @@
<property>
<name>mr3.container.idle.timeout.ms</name>
- <value>3600000</value>
+ <value>300000</value>
</property>
<property>
2020-07-21T19:42:56,340 INFO [DAG1-Map 1 Dispatcher] Task$: [Map 1]task_18797_0000_1_01_000000 sets diagnostic to Some(com.datamonad.mr3.api.common.MR3Exception: FATAL: Cannot recover from this error
at com.datamonad.mr3.worker.TaskRunner.signalFatalError(TaskRunner.scala:214)
at com.datamonad.mr3.tez.TaskContext.reportFailure(TaskContext.scala:136)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:322)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280)
at com.datamonad.mr3.tez.ProcessorWrapper.run(TezProcessor.scala:59)
at com.datamonad.mr3.worker.LogicalIOProcessorRuntimeTask$$anonfun$run$1.apply$mcV$sp(RuntimeTask.scala:295)
at com.datamonad.mr3.worker.LogicalIOProcessorRuntimeTask$$anonfun$run$1.apply(RuntimeTask.scala:263)
at com.datamonad.mr3.worker.LogicalIOProcessorRuntimeTask$$anonfun$run$1.apply(RuntimeTask.scala:263)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:357)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:311)
... 10 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Async Initialization failed. abortRequested=false
at org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:466)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:400)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:576)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:525)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:338)
... 11 more
Caused by: org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionError: Hash table loading exceeded memory limits for input: Map 2 numEntries: 25200000 estimatedMemoryUsage: 4253053664 effectiveThreshold: 3664143994 memoryMonitorInfo: { isLlap: true executorsPerNode: 3 maxExecutorsOverSubscribeMemory: 3 memoryOverSubscriptionFactor: 0.20000000298023224 memoryCheckInterval: 100000 noConditionalTaskSize: 1145044992 adjustedNoConditionalTaskSize: 1832071997 hashTableInflationFactor: 2.0 threshold: 3664143994 }
at org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:139)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:358)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:427)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.run(MapJoinOperator.java:225)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.run(MapJoinOperator.java:222)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:222)
at org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:120)
at org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache$1.call(LlapObjectCache.java:147)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
) true
--
You received this message because you are subscribed to the Google Groups "MR3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hive-mr3+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/154b25db-4c5c-4cc0-b415-8f93d92dac99o%40googlegroups.com.
Vikas Rana
kubectl get nodes --watch
NAME STATUS ROLES AGE VERSION
ip-10-111-144-193.eu-central-1.compute.internal Ready <none> 10m v1.16.12-eks-904af05
ip-10-111-145-247.eu-central-1.compute.internal Ready <none> 36m v1.16.12-eks-904af05
Vikass-MacBook-Pro:hive-mr3 vrana$ kubectl -nhivemr3 get pods --watch
NAME READY STATUS RESTARTS AGE
hivemr3-hiveserver2-8wwrj 1/1 Running 0 32m
hivemr3-metastore-0 1/1 Running 0 32m
mr3master-8797-0 1/1 Running 0 32m
mr3worker-de2e-1 1/1 Running 0 8m57s
2020-07-21T19:40:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-21T19:41:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-21T19:41:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-21T19:41:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = (0GB / 0GB)
2020-07-21T19:41:39,334 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:41:49,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:41:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:42:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:42:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:42:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:42:39,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:42:49,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:42:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 96.3% (11832MB / 12288MB)
2020-07-21T19:43:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 93.3% (11468MB / 12288MB)
2020-07-21T19:43:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 93.9% (11543MB / 12288MB)
2020-07-21T19:43:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 94.4% (11605MB / 12288MB)
2020-07-21T19:43:39,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 91.7% (11264MB / 12288MB)
2020-07-21T19:43:49,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 88.9% (10922MB / 12288MB)
2020-07-21T19:43:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 80.6% (9898MB / 12288MB)
2020-07-21T19:44:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 72.2% (8874MB / 12288MB)
2020-07-21T19:44:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 63.9% (7850MB / 12288MB)
2020-07-21T19:44:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 55.6% (6826MB / 12288MB)
2020-07-21T19:44:39,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 47.2% (5802MB / 12288MB)
2020-07-21T19:44:49,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 38.9% (4778MB / 12288MB)
2020-07-21T19:44:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 33.3% (4096MB / 12288MB)
2020-07-21T19:45:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 36.1% (4437MB / 12288MB)
2020-07-21T19:45:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 36.1% (4437MB / 12288MB)
2020-07-21T19:45:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 36.1% (4437MB / 12288MB)
2020-07-21T19:45:39,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 38.9% (4778MB / 12288MB)
2020-07-21T19:45:49,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:45:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:46:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:46:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:46:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:46:39,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:46:49,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:46:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:47:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:47:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:47:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:47:39,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:47:49,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:47:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:48:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:48:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:48:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:48:39,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:48:49,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:48:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:49:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:49:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 91.7% (11264MB / 12288MB)
2020-07-21T19:49:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 83.3% (10240MB / 12288MB)
2020-07-21T19:49:39,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 75.0% (9216MB / 12288MB)
2020-07-21T19:49:49,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 66.7% (8192MB / 12288MB)
2020-07-21T19:49:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 58.3% (7168MB / 12288MB)
2020-07-21T19:50:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 50.0% (6144MB / 12288MB)
2020-07-21T19:50:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 41.7% (5120MB / 12288MB)
2020-07-21T19:50:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 33.3% (4096MB / 12288MB)
2020-07-21T19:50:39,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 25.0% (3072MB / 12288MB)
2020-07-21T19:50:49,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 16.7% (2048MB / 12288MB)
2020-07-21T19:50:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 8.3% (1024MB / 12288MB)
2020-07-21T19:51:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:51:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:51:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:51:39,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:51:49,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:51:59,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:52:09,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:52:19,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 100.0% (12288MB / 12288MB)
2020-07-21T19:52:29,333 INFO [All-In-One] TaskScheduler: All-In-One average memory usage = 96.3% (11832MB / 12288MB)
For your reference I've attached the master log.
Thanks in advance,
Cheers,
VR
Vikas Rana
Caused by: org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionError: Hash table loading exceeded memory limits for input
<property>
<name>mr3.enable.auto.scaling</name>
- <value>false</value>
+ <value>true</value>
</property>
kubectl -nhivemr3 get pods
NAME READY STATUS RESTARTS AGE
hivemr3-hiveserver2-mwhkj 1/1 Running 0 11m
hivemr3-metastore-0 1/1 Running 0 11m
mr3master-3272-0 1/1 Running 0 10m
mr3worker-722f-1 1/1 Running 0 9m28s
mr3worker-722f-2 1/1 Running 0 9m23s
mr3worker-722f-3 1/1 Running 0 8m3s
mr3worker-722f-4 1/1 Running 0 7m23s
mr3worker-722f-5 0/1 Pending 0 93s
Sungwoo Park
Sungwoo Park
Cheers,
Vikas Rana
Caused by: java.io.IOException: Previous writer likely failed to write s3a://megatron-prod-data/workdir/hive/hive/_mr3_session_dir/09CBB311-78D7-45F7-9042-A44C89A1388B/hive-llap-common-3.1.2.jar. Failing because I am unlikely to write too.
at org.apache.hadoop.hive.ql.exec.mr3.DAGUtils.localizeResource(DAGUtils.java:1345) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.exec.mr3.DAGUtils.addTempResources(DAGUtils.java:1234) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.exec.mr3.DAGUtils.localizeTempFilesFromConf(DAGUtils.java:1142) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.setupHiveMr3Client(MR3SessionImpl.java:194) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.exec.mr3.session.MR3SessionImpl.start(MR3SessionImpl.java:131) ~[hive-exec-3.1.2.jar:3.1.2]
... 13 more
2020-07-25T16:51:48,515 ERROR [main] session.MR3Session: Failed to start MR3 Session
java.io.IOException: Previous writer likely failed to write s3a://megatron-prod-data/workdir/hive/hive/_mr3_session_dir/09CBB311-78D7-45F7-9042-A44C89A1388B/hive-llap-common-3.1.2.jar. Failing because I am unlikely to write too.
I've attached the hive-server log for your reference,
Cheers,
VR
On Sunday, July 19, 2020 at 3:08:28 PM UTC+5:30, kauts shukla wrote:
Vikas Rana
Is there any migration/changes in run-hive.sh script from 1.0 to 1.1 ?
On Sunday, July 19, 2020 at 3:08:28 PM UTC+5:30, kauts shukla wrote:
Sungwoo Park
Sungwoo Park
rana.v...@gmail.com
When we try to submit query from HUE [running in same k8s cluster as pod] to hiveserver via K8s service endpoint we are seeing following error:
2020-08-10T17:31:05,593 ERROR [HiveServer2-Handler-Pool: Thread-32] server.TThreadPoolServer: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) ~[hive-exec-3.1.2.jar:3.1.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_232] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_232] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232] Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:328) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-3.1.2.jar:3.1.2] ... 4 more 2020-08-10T17:31:14,475 ERROR [HiveServer2-Handler-Pool: Thread-32] server.TThreadPoolServer: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) ~[hive-exec-3.1.2.jar:3.1.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_232] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_232] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232] Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:328) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-3.1.2.jar:3.1.2] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-3.1.2.jar:3.1.2] ... 4 more
And looks like its due to this:
https://community.cloudera.com/t5/Support-Questions/Hive-server-2-thrift-error/td-p/106810
https://cloudera.ericlin.me/2017/09/no-data-or-no-sasl-data-in-the-stream-error-in-hiveserver2-log/
https://issues.apache.org/jira/browse/HIVE-12754
These errors are not FATAL, but any pointers to suppress them?
rana.v...@gmail.com
Our MR3 setup on EKS having warehouse and persistent storage [not in EFS] in S3.
So please share the steps to deploy Mr3 on fargate and open challenges/blockers if any.
Now AWS Fargate 1.4.0 onwards provide some task ephemeral storage, which is very important for running workload which require persistence layer.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-task-storage.html
Sungwoo Park
Sungwoo Park
Vikas Rana
https://www.datamonad.com/faqs/
https://www.datamonad.com/services/
Q. What restrictions does the MR3 distribution have?
A. On Kubernetes, Hive on MR3 can use up to 512 gigabytes for the aggregate memory of worker Pods. For example, the user can use 16 nodes each with 32 gigabytes of memory or 8 nodes each with 64 gigabytes of memory for worker Pods. On Hadoop, there is no limit on the aggregate memory of worker Containers.
Will the cluster won't do autoscale beyond 512 GB memory?
And is the memory limit applicable if we use in our ORG internally or dev/stage env.
--
You received this message because you are subscribed to the Google Groups "MR3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hive-mr3+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/2d6a91f2-dd18-403d-afbc-2941d9a84de3n%40googlegroups.com.
Sungwoo Park
Q. What restrictions does the MR3 distribution have?
A. On Kubernetes, Hive on MR3 can use up to 512 gigabytes for the aggregate memory of worker Pods. For example, the user can use 16 nodes each with 32 gigabytes of memory or 8 nodes each with 64 gigabytes of memory for worker Pods. On Hadoop, there is no limit on the aggregate memory of worker Containers.
Will the cluster won't do autoscale beyond 512 GB memory?
And is the memory limit applicable if we use in our ORG internally or dev/stage env.
rana.v...@gmail.com
We are seeing S3 connection timing out due to deadlock in thread pool, which may be related to https://issues.apache.org/jira/browse/HADOOP-13826
ERROR in worker PODs:
We tried increasing thread pool by tuning following property, following https://docs.cloudera.com/documentation/enterprise/5-11-x/topics/admin_hive_on_s3_tuning.html
But that din't helped, our fs.s3a.* property is as follows:
- fs.s3a.assumed.role.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
- fs.s3a.assumed.role.session.duration=30m
- fs.s3a.attempts.maximum=20
- fs.s3a.aws.credentials.provider=com.amazonaws.auth.InstanceProfileCredentialsProvider
- fs.s3a.block.size=32M
- fs.s3a.buffer.dir=${hadoop.tmp.dir}/s3a
- fs.s3a.committer.magic.enabled=false
- fs.s3a.committer.name=file
- fs.s3a.committer.staging.abort.pending.uploads=true
- fs.s3a.committer.staging.conflict-mode=fail
- fs.s3a.committer.staging.tmp.path=tmp/staging
- fs.s3a.committer.staging.unique-filenames=true
- fs.s3a.committer.threads=8
- fs.s3a.connection.establish.timeout=5000
- fs.s3a.connection.maximum=2000
- fs.s3a.connection.ssl.enabled=false
- fs.s3a.connection.timeout=200000
- fs.s3a.etag.checksum.enabled=false
- fs.s3a.fast.upload.active.blocks=4
- fs.s3a.fast.upload.buffer=disk
- fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
- fs.s3a.list.version=2
- fs.s3a.max.total.tasks=2000
- fs.s3a.metadatastore.authoritative=false
- fs.s3a.metadatastore.impl=org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
- fs.s3a.multiobjectdelete.enable=true
- fs.s3a.multipart.purge=false
- fs.s3a.multipart.purge.age=86400
- fs.s3a.multipart.size=100M
- fs.s3a.multipart.threshold=2147483647
- fs.s3a.paging.maximum=5000
- fs.s3a.path.style.access=false
- fs.s3a.readahead.range=64K
- fs.s3a.retry.interval=500ms
- fs.s3a.retry.limit=${fs.s3a.attempts.maximum}
- fs.s3a.retry.throttle.interval=1000ms
- fs.s3a.retry.throttle.limit=${fs.s3a.attempts.maximum}
- fs.s3a.s3guard.cli.prune.age=86400000
- fs.s3a.s3guard.ddb.background.sleep=25
- fs.s3a.s3guard.ddb.max.retries=9
- fs.s3a.s3guard.ddb.table.capacity.read=500
- fs.s3a.s3guard.ddb.table.capacity.write=100
- fs.s3a.s3guard.ddb.table.create=false
- fs.s3a.socket.recv.buffer=8192
- fs.s3a.socket.send.buffer=8192
- fs.s3a.threads.core=1024
- fs.s3a.threads.keepalivetime=60
- fs.s3a.threads.max=1000
Sungwoo Park
Vikas Rana
Unable to execute HTTP request: Timeout waiting for connection from pool
And after updating the following property, we don't see any exceptions in workers and the query eventually succeeded after running for 4-5hrs.
<property>
<name>fs.s3a.connection.maximum</name>
- <value>250</value>
+ <value>6000</value>
+ <description>Increase the value to increase the maximum number of simultaneous connections to S3. Cloudera recommends setting this value to 1500.</description>
</property>
<property>
<name>fs.s3a.threads.core</name>
- <value>250</value>
- <description>The value of fs.s3a.threads.max is 256.</description>
+ <value>2048</value>
+ <description>Increase the value to increase the number of core threads in the thread pool used to run any data uploads or copies.</description>
+</property>
+
+<property>
+ <name>fs.s3a.threads.max</name>
+ <value>2000</value>
+ <description>Increase the value to increase the maximum number of concurrent active partition uploads and copies, which each use a thread from the thread pool.</description>
+</property>
+
+<property>
+ <name>fs.s3a.max.total.tasks</name>
+ <value>4000</value>
+ <description>Increase the value to increase the number of partition uploads and copies allowed to the queue before rejecting additional uploads.</description>
+</property>
+
+<property>
+ <name>fs.s3a.blocking.executor.enabled</name>
+ <value>false</value>
+ <description>See HADOOP-13826. https://docs.cloudera.com/documentation/enterprise/5-11-x/topics/admin_hive_on_s3_tuning.html</description>
</property>
--
You received this message because you are subscribed to a topic in the Google Groups "MR3" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hive-mr3/LkGO2rWYrTc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hive-mr3+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/246cf759-2150-45cd-b5cd-4fef455dbe75n%40googlegroups.com.
Vikas Rana
So does it mean I can try enabling prometheus in MR3 1.1 ? or Is this plan to be released in MR3 1.2 only and any tentative ETA around this release.
Thanks in advance.
Vikas Rana
And if I use mr3-all-in-one image does it include RANGER and ATS apart from HIVE?
https://hub.docker.com/r/mr3project/hivemr3-all
Sungwoo Park
We manage to stabilize the MR3 cluster by tuning the following properties. Previously DAGAppMaster kept adding newer PODs till it hit a higher limit on auto scaling group and worker pods threw an exception on S3 timeout.Unable to execute HTTP request: Timeout waiting for connection from poolAnd DAGAppMaster try to rerun the failed tasks, but since DEADLOCK takes its own time, DAGAppMaster keeps adding pods. In a few instances jobs succeed but 60-70% of the time it fails.
And after updating the following property, we don't see any exceptions in workers and the query eventually succeeded after running for 4-5hrs.
Sungwoo Park
Sungwoo Park
rana.v...@gmail.com
Sungwoo Park
FYI, creating a single large ContainerWorker is better than creating many small ContainerWorkers in general, as long as not too many Tasks (> 20) run concurrently in a single ContainerWorker. For more information, please see 'Performance Tuning' in MR3docs (https://mr3docs.datamonad.com/docs/hadoop/advanced/performance-tuning/):
Vikas Rana
Node type: r5d.4xlarge: 8cpu, 64GB
all-in-one-container cpu: 7 core
all-in-one-container memory: 42GB
LLAP enabled
Maper specs: 6GB, 1 core
Reducer specs: 6GB, 1 core
hive.mr3.all-in-one.containergroup.memory.mb=42GB,
hive.mr3.container.max.java.heap.fraction=0.8f,
hive.mr3.llap.headroom.mb=0,
hive.llap.io.memory.size=10Gb
We want to check how worker PODs heap usage over its lifetime. We are getting CPU utilization of 90% when the cluster is running at 100% capacity. Based on the memory profile, we may want to switch to M5 or C5 which may provide more compute performance.
I wonder if your goal is to check the memory usage of individual Containerworker Pods to see if 6GB is enough. Currently only DAGAppMaster exports Prometheus metrics, and ContainerWorkers do not. So, with the current implementation, we cannot measure the memory usage of individual ContainerWorkers (unless we use JDK tools).
To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/46483598-dd7a-48f9-8cae-97d27ff66c32n%40googlegroups.com.
Sungwoo Park
We are using one single large ContainerWorker to run 7 concurrent mapper/reducers per pod. Our spec is as follows:
Node type: r5d.4xlarge: 8cpu, 64GB
all-in-one-container cpu: 7 core
all-in-one-container memory: 42GB
LLAP enabled
Maper specs: 6GB, 1 core
Reducer specs: 6GB, 1 core
hive.mr3.all-in-one.containergroup.memory.mb=42GB,
hive.mr3.container.max.java.heap.fraction=0.8f,
hive.mr3.llap.headroom.mb=0,
hive.llap.io.memory.size=10Gb
We want to check how worker PODs heap usage over its lifetime. We are getting CPU utilization of 90% when the cluster is running at 100% capacity. Based on the memory profile, we may want to switch to M5 or C5 which may provide more compute performance.
On Tue, Aug 18, 2020 at 5:33 PM Sungwoo Park <gla...@gmail.com> wrote:I wonder if your goal is to check the memory usage of individual Containerworker Pods to see if 6GB is enough. Currently only DAGAppMaster exports Prometheus metrics, and ContainerWorkers do not. So, with the current implementation, we cannot measure the memory usage of individual ContainerWorkers (unless we use JDK tools).Also want to check if in future you are planning to expose JVM metrics from ContainerWorkers as well.
Sungwoo Park
ATS image is not in DockerHub because there has been no request yet. Let me upload the ATS image sometime soon.
Sungwoo Park
Please release MR3-1.2-SNAPSHOT that would be great. We are using 6GB container and wanna check if we can reduce to smaller container size.
rana.v...@gmail.com
Below is the mr3-site.xml:
Also ContainerWorkers doesn't expose port 9890 [mr3.prometheus.worker.httpserver.port]. Please let know if I'm missing anything here.