FR Lucee instance appears to have stopped working after Tomcat restart

417 views
Skip to first unread message

Julian Halliwell

unread,
Mar 21, 2016, 7:45:26 AM3/21/16
to FusionReactor
(Apologies if this is ends up a duplicate - my original post hasn't showed up in Google Groups, so am re-sending).

We've been running FR 6 to monitor our Lucee 4.5 installation on Win2012 without problems for a few months.

A couple of days ago we made some host changes to the Tomcat server.xml and restarted the Lucee/Tomcat service, something we have done a number of times without issue.

However this time the FR instance monitoring Lucee appears to have stopped working. Lucee is running fine, as is the FR Instance Manager on port 8087. The Instance Manager shows a green "Running" button for the Lucee instance, however trying to connect to the monitoring UI on port 8088 results in a browser "connection refused" message.

The daily instance status emails FR normally generates have not been sent and there appears to have been no log activity in the instance's log directory since the restart.

I've checked file permissions and they are unchanged: Lucee is running under the same dedicated account as before which has the same access to the FR directory.

The only indication of a problem I can find is in the lucee-stdout log for the date of the restart which contains the entry below. The osgi bundle10 folder mentioned is indeed missing from the directory (there are others there though).

Thanks
Julian.

ERROR: Error reloading cached bundle, removing it: E:\FusionReactor\instance\lucee\osgi\env\runtimefw\bundle10 (java.io.FileNotFoundException: E:\FusionReactor\instance\lucee\osgi\env\runtimefw\bundle10\bundle.location (The system cannot find the file specified))
INFO
Fusionreactor: --------------------------------------------------------------------------------------
INFO
Fusionreactor: FusionReactor - Copyright (C) Intergral GmbH. All Rights Reserved
INFO
Fusionreactor: Revision:  6.0.1 fusionreactor.6.53531.branches/FR-6.0.x-MAINT
INFO
Fusionreactor: Date    :  Fri, 20 Nov 2015 11:14:00 GMT
INFO
Fusionreactor: Java    :  1.8.0_73 [25.73-b02] Oracle Corporation
INFO
Fusionreactor: --------------------------------------------------------------------------------------
[DEBUG] Ignoring Servlet Service [javax.servlet.Servlet], alias is missing or empty
[INFO] Http service whiteboard started
ERROR
: Bundle fusionreactor-odl-bundle [8] Error starting com.intergral.fusionreactor.resources/osgi/env/bundle/fusionreactor-odl-bundle-2.0.0.jar (org.osgi.framework.BundleException: Unresolved constraint in bundle fusionreactor-odl-bundle [8]: Unable to resolve 8.0: missing requirement [8.0] osgi.wiring.package; (&(osgi.wiring.package=org.hyperic.sigar)(version>=1.6.0)(!(version>=2.0.0))))


mike_wilkinson

unread,
Mar 21, 2016, 11:49:27 AM3/21/16
to FusionReactor
Hey Julian,

Sorry to hear you're having issues with FusionReactor.

This issue has come up before... it seems that the FusionReactor process is running, however, it was unable to 'unpack' properly. This is usually caused by permissions being changed and causing FR to blow up, and/or updating FR. There is a potential simple fix; simply delete the E:\FusionReactor\instance\lucee\osgi directory, and then restart the server. the osgi directory is auto-generated on the start of FR, and theoretically, deleting it and restarting should bring FR out of it's corrupted state.

Please let me know the outcome. There may be another underlying issue if the above fix does not solve the problem for you.

Kind Regards,
Mike Wilkinson
Fusion Support Team

Julian Halliwell

unread,
Mar 21, 2016, 5:55:19 PM3/21/16
to fusion...@googlegroups.com
Hi Mike

Yes, that has worked, thank you.

But we hadn't made any changes to permissions or (knowingly) updates
to FR, so still not sure of the cause. At least I'll know what to do
next time.

Cheers
Julian.

Julian Halliwell

unread,
Apr 3, 2016, 4:56:13 AM4/3/16
to fusion...@googlegroups.com
This has happened again when I added a new host to Tomcat and
restarted the Lucee instance to pick it up: the same OSGI bundle10
folder is missing and FR stops working for that instance.

Deleting the OSGI folder and restarting Lucee fixed it again, but
frankly I'd rather not have to go through that each time I make a host
change. Having to restart Tomcat once to pick up the change is
inconvenient enough on a production server.

Permissions are not being changed, nor is there any update to FR that
I'm aware of.

Any idea how this can be prevented?

On 21 March 2016 at 21:54, Julian Halliwell <julianh...@gmail.com> wrote:
> Yes, that has worked, thank you.
>
> But we hadn't made any changes to permissions or (knowingly) updates
> to FR, so still not sure of the cause. At least I'll know what to do
> next time.
>

eleni_grosdouli

unread,
Apr 4, 2016, 10:13:17 AM4/4/16
to FusionReactor
Hi Julian,

Sorry to hear that you still have problems with FusionReactor.

We are currently investigating your issue,but a possible solution might be to upgrade your FR from the 6.0.1 to 6.1.1.

Additionally, can you please tell me which version of OS do you use, 32-bit or 64-bit?

Please do not hesitate to contact with us again.

Best Regards,
Eleni Grosdouli
FusionReactor Support Team

mike_wilkinson

unread,
Apr 4, 2016, 10:40:44 AM4/4/16
to FusionReactor
Hi Julian,

I have been trying to reproduce the issue you are experiencing but it is proving to be quite difficult to reproduce;

These are the steps I have taken:
  1. Set up a Windows 2012 64-Bit R2 Virtual Machine;
  2. installed Lucee 4.5 via the Windows installer. ( lucee-4.5.1.000-pl0-windows-installer.exe );
  3. installed FusionReactor 6.0.1 to the VM, and then used FusionReactor Administration Manager to install the java args to the Lucee server;
  4. verified that FR correctly boots up, alongside the Lucee server;
  5. proceeded to configure the tomcat/server.xml file and added an additional host, then saved the file;
  6. restarted the Lucee service, and checked that FR and Lucee both booted up as expected;
Throughout the entire process, I have been observing the logs and ( lucee-stderr.txt, lucee-stdout.txt, instance-manager.log ). 

As an additional test, I re-tried the above steps another two times. The 2nd iteration, I installed FR manually to the Lucee service, instead of using FRAM. The 3rd run-through, I made some different configurations to the server.xml, instead of just configuring a host. In both of these cases, everything still works expectantly. 

I am running out of other options to try; However, what I haven't yet tried is what Eleni above has mentioned regarding the type of OS, whether it is 32-bit or 64-bit. 

Might I add, is it possible that a virus scanner is potentially blocking the un-packing of the OSGi bundles? Instead of checking this directly, It might just be easier to add the FusionReactor directory to your virus scanners exclude list, or equivalent.

If there is any additional information you can provide me that might aid in my investigation, please don't hesitate to put it forward. If i have missed any key steps in my attempt at re-producing the issue, please let me know.

Kind Regards,
Mike Wilkinson
FusionReactor Support Team

Julian Halliwell

unread,
Apr 4, 2016, 4:06:28 PM4/4/16
to fusion...@googlegroups.com
Hi Mike

Thanks for your efforts to try and replicate the issue. To answer
Eleni's question, our OS is 64-bit Windows 2012 R2, so the same as
your test VM. The only minor difference is that we are using the
latest stable release of Lucee which is 4.5.2.018, but I don't think
this is relevant.

As I mentioned in my original report, this has only recently begun
happening. Previously we were able to add hosts to Tomcat without
affecting FR.

The only recent change to the environment that I can think of is that
the JRE (we use the Oracle JDK version) used by both Lucee and FR has
been updated to the latest baseline security patch level which is
currently 1.8 update 77. Could this be a factor?

Cheers
Julian.

mike_wilkinson

unread,
Apr 5, 2016, 5:11:17 AM4/5/16
to FusionReactor
Hi Julian,

Here are the steps that I have tried based on the new information:
  1. Uninstalled Lucee 4.5.1 and cleared up directories, then cleanly installed Lucee 4.5.2.018. It installed fine and I then verified it boots up correctly.
  2. Opened the Luceew.exe and configured the Java Virtual Machine to point to a freshly-downloaded JDK -> JRE 1.8 Update 77 -> server/jvm.dll. 
  3. I installed FR 6.0.1 to that Lucee server, and restarted it, and confirmed that it works as expected.
  4. I repeated the steps that I specified in my previous post, and I also tried restarting FR/Lucee many times to see if it as an odd case; such as a timing issue or a race condition, but I still have not witnessed any sort of problems.
We are very sorry that you are still experiencing this issue. As I am sure you can tell, We are really trying to knuckle down and get to the bottom of this for you. I can only assume that there is another exterior factor that could potentially be causing this issue, that we are still unaware of. We will continue to attempt any further configurations that may cause the problem.

A work colleague has mentioned to me that there has been an issue somewhere around v6.0.1 to do with org.hyperic.Sigar. This may be linked to the 'missing requirement' you are seeing in logs:


ERROR: Bundle fusionreactor-odl-bundle [8] Error starting com.intergral.fusionreactor.resources/osgi/env/bundle/fusionreactor-odl-bundle-2.0.0.jar (org.osgi.framework.BundleException:Unresolved constraint in bundle fusionreactor-odl-bundle [8]: Unable to resolve 8.0: missing requirement [8.0] osgi.wiring.package; (&(osgi.wiring.package=org.hyperic.sigar)(version>=1.6.0)(!(version>=2.0.0))))

We recommend that you update to the latest version you can - maybe 6.0.5, or preferably 6.1.1 -  and hopefully, this will alleviate the issue.

Kind Regards,
Mike Wilkinson
FusionReactor Support Team

Julian Halliwell

unread,
Apr 9, 2016, 5:43:06 AM4/9/16
to fusion...@googlegroups.com
Hi Mike

I've just upgraded FR to 6.1.1. We'll see if the problem recurs when
we next add a host.

I'm still wondering if the Java upgrade is a factor since there seems
to be a correlation (both cases have followed JRE upgrades)

When upgrading the JRE I've followed the instructions for Windows
here: http://www.fusion-reactor.com/support/kb/frs-358/

They don't mention the need to restart the Lucee FR instance as well
as the FRAM service, but perhaps that is necessary. Lucee itself had
already been upgraded to the latest JRE and restarted prior to
upgrading FR's JRE, but perhaps that is the wrong way round? In other
words, maybe FR's JRE should be upgraded first, then Lucee's as that
would mean the Lucee FR instance would get a restart with the same JRE
version as the FRAM.

Cheers
Julian.

Julian Halliwell

unread,
Apr 23, 2016, 5:30:24 AM4/23/16
to fusion...@googlegroups.com
This has just happened again with the latest FR release.

It does appear to be the JRE upgrade that is causing this. I made a
host change in server.xml a week or so ago and the restart did *not*
cause any issues with FR.

However, today I have just upgraded our JDK8 to Oracle's new security
baseline of U91 and once again the FR Lucee instance would not respond
following the Lucee/Tomcat restart.

Here are the steps that led to the issue:

1) Stopped the FRAM windows service and copied the new JRE as per your
instructions: http://www.fusion-reactor.com/support/kb/frs-358/
2) Restarted the FRAM service
3) Upgraded the JRE used by Lucee/Tomcat (by changing the JAVA_HOME
environment variable)
4) Restarted Lucee/Tomcat service.

At this point Lucee itself is running fine with the new JRE but the FR
Lucee instance doesn't work. As previously described it seems to be
running in the FR Instance Manager, but trying to connect to it
results in a "connection refused" browser error. In the lucee-stdout
log the following appears:

===================================================================
ERROR: Error reloading cached bundle, removing it:
E:\FusionReactor\instance\lucee\osgi\env\runtimefw\bundle6
(java.io.FileNotFoundException:
D:\FusionReactor\instance\lucee\osgi\env\runtimefw\bundle6\bundle.location
(The system cannot find the file specified))
INFO Fusionreactor:
--------------------------------------------------------------------------------------
INFO Fusionreactor: FusionReactor - Copyright (C) Intergral GmbH. All
Rights Reserved
INFO Fusionreactor: Revision: 6.1.1
fusionreactor.8.56594.branches/FR-6.1.x-MAINT
INFO Fusionreactor: Date : Mon, 4 Apr 2016 14:13:00 GMT
INFO Fusionreactor: Java : 1.8.0_91 [25.91-b14] Oracle Corporation (64 bit)
INFO Fusionreactor: OS : Windows Server 2012 R2 [6.3] amd64
INFO Fusionreactor:
--------------------------------------------------------------------------------------
[DEBUG] Ignoring Servlet Service [javax.servlet.Servlet], alias is
missing or empty
[INFO] Http service whiteboard started
ERROR: Bundle fusionreactor-odl-bundle [9] Error starting
com.intergral.fusionreactor.resources/osgi/env/bundle/fusionreactor-odl-bundle-3.0.0.jar
(org.osgi.framework.BundleException: Unresolved constraint in bundle
fusionreactor-odl-bundle [9]: Unable to resolve 9.0: missing
requirement [9.0] osgi.wiring.package;
(&(osgi.wiring.package=org.hyperic.sigar)(version>=1.6.0)(!(version>=2.0.0))))
lucee-server-root:E:\lucee\lib\lucee-server
===================================================================

After restarting Lucee/Tomcat a *second* time, the FR instance is working again.

We consider keeping our JRE updated to the latest security level
important so it would be good if FR didn't hinder this process as it
seems to be doing at the moment.

Thanks
Julian.

mike_wilkinson

unread,
Apr 25, 2016, 9:14:53 AM4/25/16
to FusionReactor
Hey Julian,

Thanks very much for the additional information!

I'll get on it right away and I will get back to you shortly on my findings.

Apologies that this issue continues to persist. Hopefully now we can get this irritating bug squished for good!


Kind Regards,
Mike Wilkinson
FusionReactor Support Team

eleni_grosdouli

unread,
Apr 26, 2016, 6:35:07 AM4/26/16
to FusionReactor
Hi Julian,

We have tried to re-produce the issue several times on different machines and Lucee seems to work fine. However, we update JRE used by Lucee through the Luceew.exe GUI tool.

When you update the JRE in Lucee, did you first stop the Lucee service and afterwards update the JRE or first you update the JRE and afterwards restart the Lucee service.

In order to take a better look of the issue, could you please send us a screenshot of the Java tab inside the Luceew.exe. The screenshot is required in order to see if you use the default Java library path or another Java path.

  • Luceew.exe can be located in the following directory: {Lucee Directory}\tomcat\bin\Luceew.exe

Moreover, it would be useful if you could provide us with more information about the steps you followed changing the JAVA_HOME environment variable.

Finally, could you please send us all the log files from Lucee.
The log files can be located in the following directory: {FusionReactor Directory}\instance\lucee4\log\{time stamp}

It is advisable to send the log files and all the required information to the FusionReactor support email: support@fusion-reactor.com

Thanks,
Eleni


On Saturday, April 23, 2016 at 11:30:24 AM UTC+2, Julian Halliwell wrote:

Julian Halliwell

unread,
Apr 26, 2016, 7:53:49 AM4/26/16
to fusion...@googlegroups.com
Hi Eleni

> When you update the JRE in Lucee, did you first stop the Lucee service and
> afterwards update the JRE or first you update the JRE and afterwards restart
> the Lucee service.

I changed the JRE and then restarted the Lucee/Tomcat. A detail I
forgot to mention is that I am not using the JRE location within the
Lucee install folder. We have a dedicated directory where we install
each new version of the JDK, without overwriting previous versions. We
switch between versions by changing the %JAVA_HOME% environment
variable. We uninstall older versions once they're no longer in use.

> In order to take a better look of the issue, could you please send us a
> screenshot of the Java tab inside the Luceew.exe. The screenshot is required
> in order to see if you use the default Java library path or another Java
> path.

As I say we use %JAVA_HOME% to avoid hard-coding the physical path, so
the value in the JVM field is:

%JAVA_HOME%\jre\bin\server\jvm.dll

> Moreover, it would be useful if you could provide us with more information
> about the steps you followed changing the JAVA_HOME environment variable.

1) Go to System Properties > Advanced > Environment Variables
2) Under System variables change the value of JAVA_HOME to point to
the latest installed JDK root folder

> Finally, could you please send us all the log files from Lucee.
> The log files can be located in the following directory: {FusionReactor
> Directory}\instance\lucee4\log\{time stamp}

OK, will do.

Cheers
Julian.
Reply all
Reply to author
Forward
0 new messages