Java deadlock between KieRepositoryImpl and KieRepositoryScannerImpl

38 views

Skip to first unread message

Vlad L

unread,

Jul 20, 2021, 10:50:31 PM7/20/21

to Drools Development

Hi,

I am running into Java deadlocks running version 7.37 of kie server.

The deadlock happens between repository scanner timer threads (there are about 10 snapshot containers polling maven repo with an interval of 4 minutes).

The deadlock happens reliably after server restart. We are using an "unmanaged" kie server which polls a custom REST endpoint to get a list of containers to load (not Kie Controller).

I think the issue may involve common dependencies of the containers. The 2 stack traces are below.

Previously, there was another deadlock on startup whereby kie server tried to load the previously running containers from the xml file that it maintains, while the timer thread for a snapshot container that had already been loaded kicked in. This led to server lock-up, which I overcame by implementing a StartupStrategy which does not load any containers. However, there is now this new deadlock which does not kill the server but prevents snapshot containers from getting synchronised (there are another 8 threads waiting).

Please let me know what an appropriate approach could be here. Should I raise this issue on Jira? I wouldn't mind having a go at contributing to fixing this but am mindful that it's not a straightforward problem.

Thanks,
Vlad

Name: Timer-3

State: BLOCKED on org.kie.scanner.KieRepositoryScannerImpl@442a979 owned by: Timer-4

Total blocked: 20 Total waited: 28

Stack trace:

org.kie.scanner.KieRepositoryScannerImpl.getArtifactVersion(KieRepositoryScannerImpl.java:121)

org.drools.compiler.kie.builder.impl.KieRepositoryImpl$KieModuleRepo.load(KieRepositoryImpl.java:442)

- locked org.drools.compiler.kie.builder.impl.KieRepositoryImpl$KieModuleRepo@6fa07cb0

org.drools.compiler.kie.builder.impl.KieRepositoryImpl$KieModuleRepo.load(KieRepositoryImpl.java:426)

- locked org.drools.compiler.kie.builder.impl.KieRepositoryImpl$KieModuleRepo@6fa07cb0

org.drools.compiler.kie.builder.impl.KieRepositoryImpl.getKieModule(KieRepositoryImpl.java:128)

org.drools.compiler.kie.builder.impl.KieRepositoryImpl.getKieModule(KieRepositoryImpl.java:115)

org.drools.compiler.kie.builder.impl.KieContainerImpl.update(KieContainerImpl.java:236)

org.drools.compiler.kie.builder.impl.KieContainerImpl.updateToVersion(KieContainerImpl.java:195)

org.drools.compiler.kie.builder.impl.KieContainerImpl.updateDependencyToVersion(KieContainerImpl.java:217)

org.kie.scanner.KieRepositoryScannerImpl.updateKieModule(KieRepositoryScannerImpl.java:203)

org.kie.scanner.KieRepositoryScannerImpl.internalUpdate(KieRepositoryScannerImpl.java:186)

org.kie.scanner.KieRepositoryScannerImpl.internalUpdate(KieRepositoryScannerImpl.java:55)

org.drools.compiler.kie.builder.impl.AbstractKieScanner.scanNow(AbstractKieScanner.java:147)

- locked org.kie.scanner.KieRepositoryScannerImpl@2d69f996

org.drools.compiler.kie.builder.impl.AbstractKieScanner$ScanTask.run(AbstractKieScanner.java:127)

- locked org.kie.scanner.KieRepositoryScannerImpl@2d69f996

java.util.TimerThread.mainLoop(Timer.java:555)

java.util.TimerThread.run(Timer.java:505)

State: BLOCKED on org.drools.compiler.kie.builder.impl.KieRepositoryImpl$KieModuleRepo@6fa07cb0 owned by: Timer-3

Total blocked: 14 Total waited: 26

Stack trace:

org.drools.compiler.kie.builder.impl.KieRepositoryImpl$KieModuleRepo.load(KieRepositoryImpl.java:426)

org.drools.compiler.kie.builder.impl.KieRepositoryImpl.getKieModule(KieRepositoryImpl.java:128)

org.drools.compiler.kie.builder.impl.KieRepositoryImpl.getKieModule(KieRepositoryImpl.java:115)

org.kie.scanner.KieRepositoryScannerImpl.addDependencies(KieRepositoryScannerImpl.java:151)

org.kie.scanner.KieRepositoryScannerImpl.buildArtifact(KieRepositoryScannerImpl.java:144)

org.kie.scanner.KieRepositoryScannerImpl.loadArtifact(KieRepositoryScannerImpl.java:117)

org.kie.scanner.KieRepositoryScannerImpl.loadArtifact(KieRepositoryScannerImpl.java:112)

- locked org.kie.scanner.KieRepositoryScannerImpl@442a979

org.drools.compiler.kie.builder.impl.KieRepositoryImpl.loadKieModuleFromMavenRepo(KieRepositoryImpl.java:192)

org.drools.compiler.kie.builder.impl.KieRepositoryImpl.getKieModule(KieRepositoryImpl.java:138)