Jira (PUP-11373) Server with multithreaded enabled hangs after upgrade v6 -> v7

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11373 Server with multithreaded enabled hangs after upgrade v6 -> v7 Change By: Josh Cooper Summary: Server with multithreaded enabled hangs after upgrade v6 -> v7 Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11373

Re: Server with multithreaded enabled hangs after upgrade v6 -> v7 Pavel Sapezhka Thanks for filing this and including the threaddump. It looks like there's a deadlock due to multithreaded being enabled. We'll need to take a closer look.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11373

Server with multithreaded enabled hangs after upgrade v6 -> v7

Change By: Josh Cooper Epic Link: SERVER-2911

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11373

Re: Server with multithreaded enabled hangs after upgrade v6 -> v7 Thread qtp346358872-67 has locked <bd0f663> in https://github.com/puppetlabs/puppet/blob/7.12.1/lib/puppet/pops/loader/loader.rb#L153 and is waiting to lock <3f27e688> in https://github.com/puppetlabs/puppet/blob/7.12.1/lib/puppet/resource/type_collection.rb#L188 Thread qtp346358872-131 has locked <3f27e688> in https://github.com/puppetlabs/puppet/blob/7.12.1/lib/puppet/resource/type_collection.rb#L188 and is waiting to lock <bd0f663> in https://github.com/puppetlabs/puppet/blob/7.12.1/lib/puppet/node/environment.rb#L248

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11373

Re: Server with multithreaded enabled hangs after upgrade v6 -> v7

This is probably triggered due to https://github.com/puppetlabs/puppet/commit/ec319d778150eade0b6128c2ddd11cbf3926183b, because previously the loader didn't lock the environment (which was the cause of a different deadlock PUP-10959)

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11373

Re: Server with multithreaded enabled hangs after upgrade v6 -> v7

Could you give this a try: https://github.com/joshcooper/puppet/commit/107426595261935da8c6e96aa6d3cba6343dfc64?

Add Comment

[Pavel Sapezhka (Jira)]

Pavel Sapezhka commented on PUP-11373

Re: Server with multithreaded enabled hangs after upgrade v6 -> v7

Josh Cooper  I have been testing for 5 hours, have no issues, thank you!

Add Comment

[Pavel Sapezhka (Jira)]

Pavel Sapezhka commented on PUP-11373

Re: Server with multithreaded enabled hangs after upgrade v6 -> v7

The only issue I've found is sometimes puppetserver returns 500 error with a log like Dec 02 10:40:29 tor1-infra-puppet puppetserver[1874859]: ERROR [qtp580967019-3852] [puppetserver] Puppet Server Error: Evaluation Error: Error while evaluating a Method call, Internal Error: Attempt to redefine loader named 'wazuh private' (file: /etc/puppetlabs/code/environments/production/modules/wazuh/manifests/init.pp, line: 6, column: 11) on node tor1-prod-pathpuzzle-android-app-1.etrigan.net It's periodically happened with almost any node and module. Could it be related to this issue or it's another problem?

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11373

Re: Server with multithreaded enabled hangs after upgrade v6 -> v7

Thanks Pavel Sapezhka! The Attempt to redefine loader named 'xxx private' error is filed as PUP-11324 (as it happens in 6.x).

Add Comment

[Josh Cooper (Jira)]

Josh Cooper assigned an issue to Josh Cooper

Puppet / PUP-11373

Server with multithreaded enabled hangs after upgrade v6 -> v7

Change By: Josh Cooper Assignee: Josh Cooper

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11373 Server with multithreaded enabled hangs after upgrade v6 -> v7

Change By: Josh Cooper Sprint: Coremunity Kanban

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11373 Server with multithreaded enabled hangs after upgrade v6 -> v7

Change By: Josh Cooper Team: Coremunity

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11373 Server with multithreaded enabled hangs after upgrade v6 -> v7

Change By: Josh Cooper Sprint: Coremunity Kanban

Add Comment

This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11373 Server with multithreaded enabled hangs after upgrade v6 -> v7

Change By: Josh Cooper Team: Coremunity Phoenix

Add Comment

[Nirupama Mantha (Jira)]

Nirupama Mantha updated an issue

Puppet / PUP-11373 Server with multithreaded enabled hangs after upgrade v6 -> v7

Change By: Nirupama Mantha Sprint: Phoenix 2022-01-18

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11373 Server with multithreaded enabled hangs after upgrade v6 -> v7

Change By: Josh Cooper Fix Version/s: PUP 7.14.0

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11373 Server with multithreaded enabled hangs after upgrade v6 -> v7

Change By: Josh Cooper Release Notes: Bug Fix Release Notes Summary: Fixes a deadlock when running puppetserver in multithreaded mode between the environment and type collection.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11373

Re: Server with multithreaded enabled hangs after upgrade v6 -> v7 Passed in 2e131b4ff9

Add Comment

[Heston Hoffman (Jira)]

Heston Hoffman updated an issue

Puppet / PUP-11373

Server with multithreaded enabled hangs after upgrade v6 -> v7

Change By: Heston Hoffman Labels: docs-reviewed

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11373 Server with multithreaded enabled hangs after upgrade v6 -> v7

Change By: Josh Cooper Fix Version/s: 7.15.0

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11373

Re: Server with multithreaded enabled hangs after upgrade v6 -> v7 I removed the 7.15.0 fix version because the code was first fixed and released in 7.14.0

Add Comment