Number of Compiler Daemon Workers exceeds org.gradle.workers.max setting

[Kyle Moore]

Hi gradle devs.

After a previous discussion with Daz here (and speaking with Ethan at a meetup) I've decided to fork Gradle 2.11 and try adding a new plugin for the Gosu language.

The plugin is working locally with forked compilation, but I noticed something peculiar in my very large, multimodule project.

gradle.properties contains the following:

org.gradle.parallel=true
org.gradle.workers.max=2

When I only compile Java, the max # of worker JVMs is respected.

When I enable compilation with my plugin, something interesting happens:

1. The first subproject executes :compileJava and creates a worker. No problem.
2. The second subproject executes :compileGosu, whose fork options are not compatible with the first worker. A second worker is created. No problem.
3. The third subproject executes :compileGosu while the second is still busy. The original worker is idle but incompatible. A third worker is now created. Unexpected!

After debugging CompilerDaemonManager, CompilerClientsManager, and CompilerDaemonStarter I can see why this happens*. But I am confused as to how the workers.max property works correctly when only compiling Java, i.e. all JavaCompile tasks are using the same forkOptions and an existing, idle worker is always used for the next unit of work.

Can someone give me a hint where StartParameter#getMaxWorkerCount is checked before building new WorkerProcesses? Obviously it works in usual situations but I am confused how my number of workers is exceeding the max setting.

Thanks in advance,

Kyle Moore

* CompilerDaemonManager#getDaemon will indiscriminately create a new client if no idle client is available:

CompilerDaemonClient client = clientsManager.reserveIdleClient(forkOptions);
if (client == null) {
    client = clientsManager.reserveNewClient(workingDir, forkOptions); //appears to always create a new client
}

[Mark Vieira]

Can someone give me a hint where StartParameter#getMaxWorkerCount is checked before building new WorkerProcesses?

Quite simply, it isn't. The maxWorkersCount is only used a) for limiting max parallel threads in conjunction with --parallel (or org.gradle.parallel=true) and b) with native compilation.

--
You received this message because you are subscribed to the Google Groups "gradle-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gradle-dev+...@googlegroups.com.
To post to this group, send email to gradl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gradle-dev/d3400d71-eccc-47c1-8814-e25e101fb09c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mark Vieira

Principal Engineer

Gradle, Inc.

ma...@gradle.com

[Kyle Moore]

Thanks Mark for the quick reply. I now understand that maxWorkersCount is unrelated to the number of spawned WorkerProcesses - and that's good since it means my plugin is working correctly.

As a simple follow-up, is there any way to reap or shutdown an idle worker? I'm pretty sure the answer is "no" but thought I'd ask. As a result, I need to go back to the drawing board for a bit and doublecheck that all my forked compiler processes have compatible jvmArgs, memory requirements, classpaths and shared packages.

Thanks again for your time!

- Kyle Moore