Vert.x reloading error

[Maciej Żerkowski]

Hello,

I found some interesting behavior - I have fat-jar (~7 modules) deployed to sandpit environment with very low traffic - even a week without even single login. Situations repeats regulary - after about a week or so getting the below exception.
My questions are:

1. Module com.xxxxx~xxxxx-gxxxxx-mod~1.0.0-SNAPSHOT has changed, reloading it.   - What have changed?? The code is in fat-jar, so I did not change anything. Why it tries to reload it?

2. Why it throws an error?

Almost all mu modules are written in Groovy, if it matters.

from the documentation I found this link: http://vertx.io/mods_manual.html#resident  But I do not write a language implementation...

Can you please explain what is going on here? I do not want to restart the server each week:)

2015-03-01 01:00:03,621 [vert.x-worker-thread-6] INFO  org.vertx.java.platform.impl.Redeployer - Module com.xxxx.vertx~mod-auth-mgr~1.0.1-SNAPSHOT has changed, reloading it.
2015-03-01 01:00:03,636 [vert.x-worker-thread-6] INFO  org.vertx.java.platform.impl.Redeployer - Module com.xxxx~xxx-persistor-mod~1.0.7-SNAPSHOT has changed, reloading it.
2015-03-01 01:00:03,701 [vert.x-worker-thread-6] ERROR org.vertx.java.platform.impl.DefaultPlatformManager - Failed to run task
org.vertx.java.platform.PlatformManagerException: Module directory /tmp/vertx-33cb0cd7-3c4b-483a-baa1-a8d59b25fc69/mods/com.xxxx~ramxxxxt-web-mod~1.0.0 contains no mod.json nor module.link file
    at org.vertx.java.platform.impl.DefaultPlatformManager.loadModuleInfo(DefaultPlatformManager.java:1224) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager.deployModuleFromFileSystem(DefaultPlatformManager.java:1242) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager.access$1300(DefaultPlatformManager.java:57) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager$13.run(DefaultPlatformManager.java:471) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager$14.run(DefaultPlatformManager.java:487) ~[vertx-platform-2.1.2.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_09-icedtea]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_09-icedtea]
    at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09-icedtea]
2015-03-01 01:00:03,709 [vert.x-worker-thread-0] ERROR org.vertx.java.platform.impl.DefaultPlatformManager - Failed to run task
org.vertx.java.platform.PlatformManagerException: Module directory /tmp/vertx-33cb0cd7-3c4b-483a-baa1-a8d59b25fc69/mods/com.xxxxx~xxxxx-web-mod~1.0.0 contains no mod.json nor module.link file
    at org.vertx.java.platform.impl.DefaultPlatformManager.loadModuleInfo(DefaultPlatformManager.java:1224) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager.deployModuleFromFileSystem(DefaultPlatformManager.java:1242) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager.access$1300(DefaultPlatformManager.java:57) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager$13.run(DefaultPlatformManager.java:471) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager$14.run(DefaultPlatformManager.java:487) ~[vertx-platform-2.1.2.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_09-icedtea]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_09-icedtea]
    at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09-icedtea]
2015-03-01 01:00:03,876 [vert.x-worker-thread-13] INFO  org.vertx.java.platform.impl.Redeployer - Module com.xxxxx~xxxxx-gxxxxx-mod~1.0.0-SNAPSHOT has changed, reloading it.
2015-03-01 01:00:03,876 [vert.x-worker-thread-13] ERROR org.vertx.java.platform.impl.Redeployer - Failed to run task
java.lang.NullPointerException: null
    at org.vertx.java.platform.impl.DefaultPlatformManager.reloadModules(DefaultPlatformManager.java:369) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.Redeployer.checkForChanges(Redeployer.java:229) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.Redeployer.access$000(Redeployer.java:28) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.Redeployer$1.run(Redeployer.java:49) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.Redeployer$3.run(Redeployer.java:240) ~[vertx-platform-2.1.2.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_09-icedtea]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_09-icedtea]
    at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09-icedtea]

[Tim Fox]

In Vert.x 2.x when you execute a fatjar it unzips it to a tmp directory
and runs it from there as if the modules were on disk.

You can see from your stacktrace that this directory is mentioned and
your module files can't be found:

/tmp/vertx-33cb0cd7-3c4b-483a-baa1-a8d59b25fc69/mods/com.xxxxx~xxxxx-web-mod~1.0.0

contains no mod.json nor module.link file

So most probably you have some process (e.g. cron job) that deletes your
tmp directory periodically.

> *2015-03-01 01:00:03,876 [vert.x-worker-thread-13] ERROR

> org.vertx.java.platform.impl.Redeployer - Failed to run

> taskjava.lang.NullPointerException: null*

[Maciej Żerkowski]

Oh, I see - it makes sens - I will check that...

But don't you think that using /tmp for that purpose is a bit dangerous?
Is there a way to configure that location? I would like to change it into something like /var/opt and mount private file system there...

Thanks!

[Tim Fox]

On 11/03/15 18:49, Maciej Żerkowski wrote:
> Oh, I see - it makes sens - I will check that...
>
> But don't you think that using /tmp for that purpose is a bit dangerous?
> Is there a way to configure that location? I would like to change it into
> something like /var/opt and mount private file system there...

It just uses the standard OS tmp dir. So you should be able to change
that in the normal way for your OS.

But.... I'd say any process that deletes tmp dir while the server is
running is doing something wrong. tmp dir is used by many running
applications and if you delete stuff while they're running you're asking
for trouble imho :)

[Maciej Żerkowski]

W dniu piątek, 13 marca 2015 13:52:00 UTC+1 użytkownik Tim Fox napisał:

On 11/03/15 18:49, Maciej Żerkowski wrote:
> Oh, I see - it makes sens - I will check that...
>
> But don't you think that using /tmp for that purpose is a bit dangerous?
> Is there a way to configure that location? I would like to change it into
> something like /var/opt and mount private file system there...

It just uses the standard OS tmp dir. So you should be able to change
that in the normal way for your OS.

But.... I'd say any process that deletes tmp dir while the server is
running is doing something wrong. tmp dir is used by many running
applications and if you delete stuff while they're running you're asking
for trouble imho :)

But servers have their own temp dirs like for example JBoss...
Anyway, it seems that there was something that was cleaning this folder - thanks a lot!
But I still have issues with it and have to restart servers. From the logs I can see that 'something has changed and module will be reloaded' - my question would be:

1. What and how was changed - it's inside the fat-jar (or in tmp folder)?
2. It cannot find mod.json - it is in fat-jat but I cannot see it in tmp... shouldn't it be also put in tmp folder?

2015-03-15 03:42:03,441 [vert.x-worker-thread-9] INFO  org.vertx.java.platform.impl.Redeployer - Module com.xxx.vertx~mod-auth-mgr~1.0.2 has changed, reloading it.
2015-03-15 03:42:03,441 [vert.x-worker-thread-9] INFO  org.vertx.java.platform.impl.Redeployer - Module com.xxxx~xxxx-persistor-mod~1.0.8 has changed, reloading it.       

Why something 'was changed' - it's inside of jar.....???????

2015-03-15 03:42:03,444 [vert.x-worker-thread-19] ERROR org.vertx.java.platform.impl.DefaultPlatformManager - Failed to run task
org.vertx.java.platform.PlatformManagerException: Module directory /tmp/vertx-b4237280-94f9-4657-944e-7d186393f2c4/mods/com.xxx~xxxx-web-mod~1.0.0 contains no mod.json nor module.link file

There is mod.json in this module but it seems that it is missing from tmp folder
Same for others modules:

    at org.vertx.java.platform.impl.DefaultPlatformManager.loadModuleInfo(DefaultPlatformManager.java:1224) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager.deployModuleFromFileSystem(DefaultPlatformManager.java:1242) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager.access$1300(DefaultPlatformManager.java:57) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager$13.run(DefaultPlatformManager.java:471) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager$14.run(DefaultPlatformManager.java:487) ~[vertx-platform-2.1.2.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_09-icedtea]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_09-icedtea]
    at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09-icedtea]

2015-03-15 03:42:03,444 [vert.x-worker-thread-1] ERROR org.vertx.java.platform.impl.DefaultPlatformManager - Failed to run task
org.vertx.java.platform.PlatformManagerException: Module directory /tmp/vertx-b4237280-94f9-4657-944e-7d186393f2c4/mods/com.xxx~xxxx-web-mod~1.0.0 contains no mod.json nor module.link file

    at org.vertx.java.platform.impl.DefaultPlatformManager.loadModuleInfo(DefaultPlatformManager.java:1224) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager.deployModuleFromFileSystem(DefaultPlatformManager.java:1242) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager.access$1300(DefaultPlatformManager.java:57) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager$13.run(DefaultPlatformManager.java:471) ~[vertx-platform-2.1.2.jar:na]
    at org.vertx.java.platform.impl.DefaultPlatformManager$14.run(DefaultPlatformManager.java:487) ~[vertx-platform-2.1.2.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_09-icedtea]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_09-icedtea]
    at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09-icedtea]

2015-03-15 03:42:03,468 [vert.x-worker-thread-19] ERROR org.vertx.java.platform.impl.DefaultPlatformManager - Failed to run task
org.vertx.java.platform.PlatformManagerException: Module directory /tmp/vertx-b4237280-94f9-4657-944e-7d186393f2c4/mods/com.xxx~xxx-web-mod~1.0.0 contains no mod.json nor module.link file

[Tim Fox]

On 16/03/15 07:44, Maciej Żerkowski wrote:
>
> W dniu piątek, 13 marca 2015 13:52:00 UTC+1 użytkownik Tim Fox napisał:
>> On 11/03/15 18:49, Maciej Żerkowski wrote:
>>> Oh, I see - it makes sens - I will check that...
>>>
>>> But don't you think that using /tmp for that purpose is a bit dangerous?
>>> Is there a way to configure that location? I would like to change it
>> into
>>> something like /var/opt and mount private file system there...
>> It just uses the standard OS tmp dir. So you should be able to change
>> that in the normal way for your OS.
>>
>> But.... I'd say any process that deletes tmp dir while the server is
>> running is doing something wrong. tmp dir is used by many running
>> applications and if you delete stuff while they're running you're asking
>> for trouble imho :)
>>
> But servers have their own temp dirs like for example JBoss...
> Anyway, it seems that there was something that was cleaning this folder -
> thanks a lot!
> But I still have issues with it and have to restart servers. From the logs
> I can see that 'something has changed and module will be reloaded' - my
> question would be:
>
> 1. What and how was changed - it's inside the fat-jar (or in tmp folder)?

When you run a fatjar it is unzipped into your tmp directory and run
from there.

As mentioned before, you probably have a cron job that is deleting the
the contents of your tmp directory.

> 2. It cannot find mod.json - it is in fat-jat but I cannot see it in tmp...

You cannot see it because your cron job has deleted it :)

> shouldn't it be also put in tmp folder?
>
> 2015-03-15 03:42:03,441 [vert.x-worker-thread-9] INFO
> org.vertx.java.platform.impl.Redeployer - Module

> com.xxx.vertx~mod-auth-mgr~1.0.2 *has changed, reloading it.*

> 2015-03-15 03:42:03,441 [vert.x-worker-thread-9] INFO
> org.vertx.java.platform.impl.Redeployer - Module

> com.xxxx~xxxx-persistor-mod~1.0.8* has changed, reloading it.*

>
> Why something 'was changed' - it's inside of jar.....???????
>
> 2015-03-15 03:42:03,444 [vert.x-worker-thread-19] ERROR
> org.vertx.java.platform.impl.DefaultPlatformManager - Failed to run task
> org.vertx.java.platform.PlatformManagerException: Module directory
> /tmp/vertx-b4237280-94f9-4657-944e-7d186393f2c4/mods/com.xxx~xxxx-web-mod~1.0.0

> *contains no mod.json nor module.link file*

> *contains no mod.json nor module.link file*

[Maciej Żerkowski]

I checked it - there is no cron, folders are in place, even from January.

This one also: /tmp/vertx-b4237280-94f9-4657-944e-7d186393f2c4 there:

lib
META-INF
mods
org

[Tim Fox]

If there is no mod.json file there, then _something_ has deleted it
(assuming it's in the original fatjar)

[Maciej Żerkowski]

And you are absolutely right, Tim.

D'oh. I checked that there are very old folders so assumed that there is no cron (in the mean time asked to check if there is some and if it is the case to stop it - just received resolved ticket but without any info what was done so they must have just ignored me....) but after getting into each folder it appears that there is no files. So looks like someone setup cron that deletes only files without folders. OMG.

I need to chase infrastructure guys again....

Thanks again!

[John Moscato]

Wow, I just ran into the same problem with files getting deleted from the /tmp directory on Red Hat Linux.

In my case, looking at the /tmp directory was misleading at first. I have some very old files in there, including some within subdirectories.

However, I found a cron.daily job that runs tmpwatch to delete unused/unaccessed files every 10 days - but it has exclude paths and exclude patterns which leave some files and directories in /tmp alone!! Very sneaky!

# cat /etc/cron.daily/tmpwatch

#! /bin/sh

flags=-umc

/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \

        -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \

        -X '/tmp/hsperfdata_*' 10d /tmp

/usr/sbin/tmpwatch "$flags" 30d /var/tmp

for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do

    if [ -d "$d" ]; then

        /usr/sbin/tmpwatch "$flags" -f 30d "$d"

    fi

done

Since this cron is setup by default as part of the OS install (Red Hat Enterprise Linux 6.6, in my case), and I have Vert.x applications running on several Linux servers, I decided on setting the JVM tmpdir system property on startup of my apps, something like:

java -Djava.io.tmpdir=/path/to/tmpdir .....

I'm not too concerned about disk space since Vert.x apps seem to clean up their temp directories when shut down cleanly.

Use of the /tmp dir does seem risky though, especially if Linux deletes files periodically by default.

At the least, perhaps some notes could be added to the Vert.x docs to make others aware of this situation, if they want long-running Vert.x apps.

Thanks!

[Tim Fox]

Nice, detective work!

I don't think Vert.x 3 uses the tmp directory in the way Vert.x 2 does (i.e. unzipping fatjars there to run) so should be sufficient to add this to the Vert.x 2 docs I guess... I will add an issue to do this.

Thanks

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Moscato]

Sounds great - thanks very much Tim!

...