After the latest version of the program is deployed, MR3 often exits abnormally

[Carol Chapman]

Hi

    At present, I have deployed the latest version of HIVE ON MR3 in the production environment  ( Release date: 2022-09-28). I found that this version of MR3 always exits abnormally shortly after running. I provided the last two running logs. Can you help me look at this problem? Thanks!

[Carol Chapman]

[Sungwoo Park]

Hi,

I have a few questions.

1. Does this problem occur after you upgrade MR3 1.5 to 1.6-SNAPSHOT? I would like to know if there was any change made to your installation that caused this problem.

2. Does HiveServer2 always fail with the same error? 

3. Is the number of concurrent users larger than 20? For example, Metastore client says 25 concurrent connections.

2022-10-10T19:03:54,596  INFO [Heartbeater-4] metastore.HiveMetaStoreClient: Opened a connection to metastore, current connections: 25

4. HiveServer2 stops after this message (when it has 40 sessions). Does 

2022-10-10T19:05:59,140  INFO [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@45f8415b] common.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4095ms
GC pool 'PS MarkSweep' had collection(s): count=2 time=4281ms
GC pool 'PS Scavenge' had collection(s): count=3 time=72ms
2022-10-10T19:05:59,140  INFO [HiveServer2-Handler-Pool: Thread-1045] service.CompositeService: Session closed, SessionHandle [979b5e81-79f1-493c-8f40-24d1fa98475c], current sessions:40

...

2022-10-10T19:05:59,162  INFO [HiveServer2-Handler-Pool: Thread-1045] server.HiveServer2: Shutting down HiveServer2

On Tue, Oct 11, 2022 at 2:02 AM Carol Chapman <carolcha...@gmail.com> wrote:

Hi

    At present, I have deployed the latest version of HIVE ON MR3 in the production environment  ( Release date: 2022-09-28). I found that this version of MR3 always exits abnormally shortly after running. I provided the last two running logs. Can you help me look at this problem? Thanks!

--
You received this message because you are subscribed to the Google Groups "MR3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hive-mr3+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hive-mr3/218fdb66-c7bb-4b40-98e2-a702fb855d40n%40googlegroups.com.

[Carol Chapman]

HI.

1. Does this problem occur after you upgrade MR3 1.5 to 1.6-SNAPSHOT? I would like to know if there was any change made to your installation that caused this problem.

  I didn't deliberately deploy mr31.6-snapshot, I just saw MR31.3 redeploy the latest MR31.5 after the September 28 update. Is this version using the 1.6-SNAPSHOT version of MR3?

2.Does HiveServer2 always fail with the same error? 

I'm not quite sure.... But the error messages appear to be of the same type. I can provide a log of the third run exception.

3.Is the number of concurrent users larger than 20? For example, Metastore client says 25 concurrent connections.

Yes. I see from the UI of hiveserver 2 that the current number of connections is 25

[Carol Chapman]

I was previously running online with the July release of MR31.5, which was stable in my environment. I did not change any configuration during the upgrade

[Carol Chapman]

Hi. Did you find anything?

[Sungwoo Park]

No, after sending my previous email. In case that you haven't found it:

1.

I didn't deliberately deploy mr31.6-snapshot, I just saw MR31.3 redeploy the latest MR31.5 after the September 28 update. Is this version using the 1.6-SNAPSHOT version of MR3?

--> No, it is MR3 1.5 rebuilt on Sep 28. It should be the same as the previous release of MR3 1.5.

2.

I haven't figured out why HS2 stops, but it might be that HS2 stops after a memory issue. From the log:

2022-10-10T19:05:59,140  INFO [org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@45f8415b] common.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4095ms
GC pool 'PS MarkSweep' had collection(s): count=2 time=4281ms
GC pool 'PS Scavenge' had collection(s): count=3 time=72ms

...

2022-10-10T19:05:59,162  INFO [HiveServer2-Handler-Pool: Thread-1045] server.HiveServer2: Shutting down HiveServer2

Could you check the amount of memory allocated to HS2, or the memory consumption of HS2 before it stops? That's all I can think of now.

[Carol Chapman]

OK, I will tracking the issue for a period of time