Illegal access: this web application instance has been stopped already.

[mechri...@gmail.com]

I've gotten the below error a few times in the past few weeks. When it happens, the Java client simply returns `null` to whatever API call I was making. I then have to restart Kill Bill to get it working again. Any ideas?

```
killbill_1 | INFO: Server startup in 32933 ms
killbill_1 | Nov 14, 2017 8:02:42 AM org.apache.catalina.loader.WebappClassLoaderBase loadClass
killbill_1 | INFO: Illegal access: this web application instance has been stopped already. Could not load java.util.concurrent.ScheduledFuture. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
killbill_1 | java.lang.IllegalStateException
killbill_1 | at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1743)
killbill_1 | at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1701)
killbill_1 | at com.zaxxer.hikari.pool.PoolBagEntry.cancelMaxLifeTermination(PoolBagEntry.java:68)
killbill_1 | at com.zaxxer.hikari.pool.HikariPool.closeConnection(HikariPool.java:94)
killbill_1 | at com.zaxxer.hikari.pool.HikariPool$HouseKeeper.run(HikariPool.java:220)
killbill_1 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
killbill_1 | at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
killbill_1 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
killbill_1 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
killbill_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
killbill_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
killbill_1 | at java.lang.Thread.run(Thread.java:748)
killbill_1 |
killbill_1 | Nov 14, 2017 8:02:42 AM org.apache.catalina.loader.WebappClassLoaderBase loadClass
killbill_1 | INFO: Illegal access: this web application instance has been stopped already. Could not load com.zaxxer.hikari.pool.HikariPool$1. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
killbill_1 | java.lang.IllegalStateException
killbill_1 | at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1743)
killbill_1 | at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1701)
killbill_1 | at com.zaxxer.hikari.pool.HikariPool.closeConnection(HikariPool.java:101)
killbill_1 | at com.zaxxer.hikari.pool.HikariPool$HouseKeeper.run(HikariPool.java:220)
killbill_1 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
killbill_1 | at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
killbill_1 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
killbill_1 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
killbill_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
killbill_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
killbill_1 | at java.lang.Thread.run(Thread.java:748)
```

[Pierre-Alexandre Meyer]

Hi Chris,

I don't think we have enough information to understand what is happening. It looks like Kill Bill is shutting down on its own? Can you find more logs on the container about why it is happening (e.g. OOM killer)?

--
You received this message because you are subscribed to the Google Groups "Kill Bill users mailing-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to killbilling-users+unsubscribe@googlegroups.com.
To post to this group, send email to killbilling-users@googlegroups.com.
Visit this group at https://groups.google.com/group/killbilling-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/killbilling-users/d353dbf8-2803-457c-b1af-c56bbf8337f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Pierre

[Chris Smith]

Hi Pierre,

I don't think anything has been OOM killed. But I may be wrong on that. I tried searching the logs, but didn't see anything. The server has 2 GB of RAM, and 4 GB of swap space. The RAM is (practically) full, with 2 GB of swap used. According to netdata, I've always had plenty of swap.

Like I said, it's very intermittent (might happen every other week or so). Do you have any other suggestions on how I could track down the cause?

Thanks!

[Pierre-Alexandre Meyer]

Is Kill Bill the only process on that server? I'm concerned about the 2GB of swap used - ideally, you would have more breathing room.

Anything in the container logs (dmesg, /var/log/messages, etc.)? What about on the host (e.g. docker daemon logs)?

--

Pierre

[Chris Smith]

I've got several different servers on that server, Kill Bill being one of them.

I did see on the host, something from syslog, but am unsure if it is related:

root@gt2 /var/log # gunzip < syslog.2.gz | grep kill
Nov 20 13:32:01 gt2 systemd[29310]: Received SIGRTMIN+24 from PID 19847 (kill).
Nov 20 13:32:29 gt2 systemd[19875]: Received SIGRTMIN+24 from PID 20041 (kill).
Nov 20 21:24:31 gt2 systemd[13975]: Received SIGRTMIN+24 from PID 15236 (kill).
Nov 20 21:24:58 gt2 systemd[15243]: Received SIGRTMIN+24 from PID 19243 (kill).
Nov 20 21:24:59 gt2 systemd[19247]: Received SIGRTMIN+24 from PID 19302 (kill).
Nov 20 21:25:01 gt2 systemd[19307]: Received SIGRTMIN+24 from PID 19387 (kill).

I don't see anything wrong with Docker. But it's showing logs today, but the errors happened yesterday, odd.

chris13524@gt2 ~ % sudo journalctl -u docker.service --since "2017-11-20"            
-- Logs begin at Wed 2017-11-22 16:15:38 UTC, end at Wed 2017-11-22 16:15:52 UTC. --
-- No entries --

I've since redeployed the Kill Bill container, so I don't have any logs from it unfortunately. (I suppose I could've just restarted it, oh well) Next time it happens, I'll do more inspections on the container end.