High CPU when using redeploy feature

[Micah Smith]

Is anyone else getting crazy high CPU when using redeploy? The java process is using constantly >= 80% cpu when using redeploy, and virtually no CPU when not using redeploy feature.

Command im using:

java -jar build/libs/web-fat.jar --conf src/main/conf/config.json --redeploy="src/main/java/**/*.java" --on-redeploy="./gradlew shadowJar -x test"

Macbook Pro - OSX 10.11.2

Vertx - 3.3.0-SNAPSHOT    (I needed features in the latest redis-client, although i believe 3.2.x did it as well)

Single Vertical

[Julien Viet]

Hi,

can you record a Java Flight Recorder session and provide it ?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/8158812d-47fa-48f4-a410-cb764ff6b8d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Clement Escoffier]

Hi,

Here you are running gradle on every change, so yes it may take some CPU. You can configure the redeploy-scan-period and redeploy-grace-period to 1) avoid polling too much on the file system - this is useful if you have a large project, 2) avoid launching gradle too much (grace is the minimum time between 2 executions of the on-redeploy command).

Clement

[Micah Smith]

@Julien

I attached the JFR here. I've never done one before but used this command:

java -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:StartFlightRecording=duration=2m,filename=recording.jfr -jar build/libs/web-fat.jar --conf src/main/conf/config.json --redeploy="src/main/java/**/*.java" --on-redeploy="./gradlew shadowJar -x test"

@Clement

Its running ~80% all the time, not just when gradle is rebuilding. It stays at ~80% using an intellij idea launcher as well run with this:

run com.postsurge.web.MainVerticle --conf src/main/conf/config.json --redeploy=build/classes/main/**/*.class --launcher-class=io.vertx.core.Launcher -Dvertx.disableFileCaching=true

The redeploy-scan-period definitely brought it down, but still very high.

[Micah Smith]

Ok, I've discovered this is because I have a ton of files (node_modules) in my webroot...After I remove them cpu is back down. Can someone explain whats going on and how to remedy this? The redeploy flag is set to 'src/main/java/**/*.java

So why does it matter if i have a lot of files in src/main/webapp? My gradle setup is setup like the example here (with shadowJar and run tasks)
https://github.com/vert-x3/vertx-examples/tree/master/gradle-redeploy

On Monday, June 6, 2016 at 11:35:24 AM UTC-5, Micah Smith wrote:

[Micah Smith]

Update:

It actually runs with high cpu even when the webapp (and node_modules) are outside src, at the root of the project

On Monday, June 6, 2016 at 11:35:24 AM UTC-5, Micah Smith wrote:

[Julien Viet]

I just looked at the JFR and indeed it shows that the launcher Watcher spend lot of time there 

sorry I missed the JFR file… (somebody else could have look though…)

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/a1b56798-b692-4852-a232-94a9443161b6%40googlegroups.com.

[Clement Escoffier]

Hi,

 Can you tell me how many files you have in the watched directory.

I’ve tried with 30000 files and detect changes in less than a 500 ms.

We can optimize a bit by replacing the hash maps by another type, or use string as map key. But before that I will need to reproduce the issue as I don’t have it on mac and linux (hard for me to test on windows as it would be a virtual machine).

So if you can provide a reproducer with almost the same number of files you have and a similar structure it would be great.

Clement

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/3F4D7307-957B-4C77-9CD0-C4ADF60341F4%40julienviet.com.

[clement escoffier]

Quick update, I think I've found the issue. The watcher track all change of the current directory. It's not linked to the patterns. It should be changed.

Clement

[Clement Escoffier]

Hi,

I’ve made the changes in https://github.com/cescoffier/vert.x/tree/issues/improve-watcher-change-detection-perf. 

Clement