Another issue with Bitronix

[Ronald Gouldner]

I moved my KC 1508.3 application to a new server.

All I did was upgrade one of our testing systems to the 1508.3 version and point it to my 1508.3 database which I had previously been testing.

 

However now when I start that application I am getting this bitronix error that there is an OverlappingFileLockException.

 

Shared filelock table ? 

 

Does anyone know how to clear this ?

 

My old instance isn’t running anymore so  don’t understand this error.

This also makes me wonder can more than one copy of KC run against the same database for load balancing ?  

 

Ron

 

===========

 

2015-08-29 17:28:00,087 [localhost-startStop-1] u:/d: INFO  org.kuali.rice.ksb.m

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:612)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)

        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)

        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:745)

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionManagerBitronix' defined in class path resource [org/kuali/coeus/sys/impl/DataSourceSpringBeans.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static bitronix.tm.BitronixTransactionManager bitronix.tm.TransactionManagerServices.getTransactionManager()] threw exception; nested exception is bitronix.tm.utils.InitializationException: initialization failed, cannot safely start the transaction manager

        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:603)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1057)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:953)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:487)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)

        ... 60 more

Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static bitronix.tm.BitronixTransactionManager bitronix.tm.TransactionManagerServices.getTransactionManager()] threw exception; nested exception is bitronix.tm.utils.InitializationException: initialization failed, cannot safely start the transaction manager

        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:181)

        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:592)

        ... 69 more

Caused by: bitronix.tm.utils.InitializationException: initialization failed, cannot safely start the transaction manager

        at bitronix.tm.BitronixTransactionManager.<init>(BitronixTransactionManager.java:102)

        at bitronix.tm.TransactionManagerServices.getTransactionManager(TransactionManagerServices.java:75)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:160)

        ... 70 more

Caused by: java.nio.channels.OverlappingFileLockException

        at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)

        at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)

        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1075)

        at bitronix.tm.journal.TransactionLogAppender.<init>(TransactionLogAppender.java:71)

        at bitronix.tm.journal.DiskJournal.open(DiskJournal.java:157)

        at bitronix.tm.BitronixTransactionManager.<init>(BitronixTransactionManager.java:85)

        ... 76 more

[Ronald Gouldner]

I figured out what is causing this issue but I have not yet found a solution.

 

We have two instances of KC on this machine.   One is the test drive out of the box version and the other is our locally modified version.

 

It seems they can’t both run at the same time.   The Bitronix Transaction Manager creates a lock file so you can’t have the Transaction Manager running twice on once machine.   I am hoping there is some way to configure the location of this file so I can alter the location for one of my instances.  I would think it should be ok to have two transaction managers when they are for completely different instances pointing to different database schemas.

 

http://bitronix-transaction-manager.10986.n7.nabble.com/Bitronix-Transaction-Manager-cannot-open-disk-journal-td1385.html

 

Ron

To unsubscribe from this group and stop receiving emails from it, send an email to kc.tech+u...@kuali.org.

[Ken Geis]

Do any of your bitronix configurations specify these with absolute paths? I would expect that to fail in this manner.

• bitronix.tm.journal.disk.logPart1Filename
• bitronix.tm.journal.disk.logPart2Filename

Ken

To unsubscribe from this group and stop receiving emails from it, send an email to kc.technical.co...@kuali.org.

[Ronald Gouldner]

Don’t see those set anywhere.   Maybe that is the problem, perhaps they are defaulting to something.

 

KC

[Java8-ronald@infss14-08 kc (uh-modifications)]$ git grep bitronix.tm

coeus-impl/src/main/resources/org/kuali/coeus/sys/impl/DataSourceSpringBeans.xml:    <bean id="btmConfig" factory-method="getConfiguration" class="bitronix.tm.TransactionManagerServices" lazy-init="true">

coeus-impl/src/main/resources/org/kuali/coeus/sys/impl/DataSourceSpringBeans.xml:    <bean id="transactionManagerBitronix" class="bitronix.tm.TransactionManagerServices"

coeus-impl/src/main/resources/org/kuali/coeus/sys/impl/DataSourceSpringBeans.xml:  <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

coeus-impl/src/main/resources/org/kuali/coeus/sys/impl/DataSourceSpringBeans.xml:  <bean id="dataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

coeus-impl/src/main/resources/org/kuali/coeus/sys/impl/DataSourceSpringBeans.xml:  <bean id="riceDataSourceBitronixXa" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

coeus-impl/src/main/resources/org/kuali/coeus/sys/impl/DataSourceSpringBeans.xml:  <bean id="dataSourceBitronixXa" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

coeus-webapp/src/main/resources/log4j.properties:log4j.logger.bitronix.tm.twopc.Preparer=ERROR

 

RICE

[Java8-ronald@infss14-08 rice (uh-rice-2.5.3)]$ git grep bitronix.tm

rice-framework/krad-it/src/test/resources/KRADTestHarnessSpringBeans.xml:  <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

rice-middleware/core/impl/src/main/resources/org/kuali/rice/core/RiceDataSourceSpringBeans.xml:  <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

rice-middleware/core/impl/src/main/resources/org/kuali/rice/core/RiceDataSourceSpringBeans.xml:  <bean id="riceDataSourceBitronixXa" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

rice-middleware/core/impl/src/main/resources/org/kuali/rice/core/RiceDataSourceStandaloneClientSpringBeans.xml.template:  <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

rice-middleware/core/impl/src/main/resources/org/kuali/rice/core/RiceJTASpringBeans.xml:  <bean id="btmConfig" factory-method="getConfiguration" class="bitronix.tm.TransactionManagerServices" lazy-init="true">

rice-middleware/core/impl/src/main/resources/org/kuali/rice/core/RiceJTASpringBeans.xml:  <bean id="transactionManagerBitronix" class="bitronix.tm.TransactionManagerServices"

rice-middleware/core/impl/src/main/resources/org/kuali/rice/core/config/bitronix-default-config.properties:bitronix.tm.allowMultipleLrc=true

rice-middleware/core/impl/src/main/resources/org/kuali/rice/core/config/bitronix-default-config.properties:bitronix.tm.timer.defaultTransactionTimeout=3600

rice-middleware/impl/src/main/resources/META-INF/common-config-defaults.xml:    <param name="datasource.pool.class.non.xa" override="false">bitronix.tm.resource.jdbc.lrc.LrcXADataSource</param>

rice-middleware/impl/src/main/resources/bitronix-default-config.properties:bitronix.tm.allowMultipleLrc=true

rice-middleware/impl/src/main/resources/bitronix-default-config.properties:bitronix.tm.timer.defaultTransactionTimeout=3600

rice-middleware/it/core/src/test/resources/CORETestHarnessSpringBeans.xml:    <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

rice-middleware/it/internal-tools/src/main/resources/TestHarnessSpringBeans.xml:  <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

rice-middleware/it/kew/src/test/resources/org/kuali/rice/kew/config/TestKEWSpringBeans.xml:    <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

rice-middleware/it/kim/src/test/resources/KIMTestHarnessSpringBeans.xml:    <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

rice-middleware/it/krms/src/test/resources/KRMSTestHarnessSpringBeans.xml:    <!--  <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

rice-middleware/it/ksb/src/test/resources/KSBTestHarnessSpring.xml:    <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

rice-middleware/it/ksb/src/test/resources/org/kuali/rice/ksb/testclient1/resources/TestClient1SpringBeans.xml:  <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

rice-middleware/it/ksb/src/test/resources/org/kuali/rice/ksb/testclient2/resources/TestClient2SpringBeans.xml:    <bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

src/site/docbook/Intro_To_Rice/TRG_Global.xml:          class="bitronix.tm.TransactionManagerServices" lazy-init="true"/>

src/site/docbook/Intro_To_Rice/TRG_Global.xml:&lt;bean id="transactionManagerBitronix" class="bitronix.tm.TransactionManagerServices"

src/site/docbook/Intro_To_Rice/TRG_Global.xml:  <programlisting>&lt;bean id="riceDataSourceBitronixXa" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

src/site/docbook/Intro_To_Rice/TRG_Global.xml:  <programlisting>&lt;bean id="riceDataSourceBitronix" class="bitronix.tm.resource.jdbc.PoolingDataSource" init-method="init"

src/site/docbook/Intro_To_Rice/TRG_Global.xml:    &lt;property name="className" value="bitronix.tm.resource.jdbc.lrc.LrcXADataSource" />

[Ronald Gouldner]

Looking at my test instance the two files are btm1.tlog and brm2.tlog and they get dropped in whatever folder you were in when you started tomcat.  So I think it is a safe bet that you are correct.

 

Doesn’t look like rice allows this to be configured and if I add a bitronix properties file to my project I lose the rice settings which is OK as long as I always remember to keep my file in sync with any changes made in rice.  Having a bad memory I plan NOT to try that J

 

So I will have to modify rice to allows this as a configurable parameter.  But that can wait for Monday.

 

Thanks Ken!   I will report back if modifying rice works.

[Ken Geis]

Are you running these as services, or calling the catalina script from the command line? If as services, I would look into starting the two Tomcats with different working directories. I’ve found that on Centos, Tomcat runs with the instance user’s home directory as the working directory, and init scripts have problems if different instances run under the same account.

Ken

[Ronald Gouldner]

We run two war files in one Tomcat instance.   We start tomcat from a specific folder because KC generates S2S form files where ever you start tomcat from.  I find the btm1.tlog and btm2.tlog in this folder so I know it is just dropping the file without a path.    From samples I have seen online the files can use full path so I plan to make this a configuration.

 

I am just not sure yet how to make a properties file use a configuration value.  I have seem examples using env variables but that will not work since there will only be one environment in a Tomcat instance.  I think KC has some other properties for bitronix in the code somewhere so I play to track them down today and see if I can implement this improvement.

 

Ron

[Largent, Jeff]

Hi Ron,

We ran into a similar issue with the transaction logs, since for us at least Bitronix was defaulting to looking for them in the Tomcat bin directory, and not all of our test instances had their bin directories set up the same way. We were able to configure the tlog locations to be in environment/app-specific directories by setting them directly on the btmConfig bean in DataSourceSpringBeans.xml like so:

<bean id="btmConfig" factory-method="getConfiguration" class="bitronix.tm.TransactionManagerServices" lazy-init="true">

    <property name="serverId" value="#{T(org.apache.commons.lang3.RandomStringUtils).randomAlphanumeric(20)}_TM_KC"/>
    <property name="logPart1Filename" value=“/example/dir/logs/${app.code}/${environment}/btm1.tlog"/>
    <property name="logPart2Filename" value=“/example/dir/logs/${app.code}/${environment}/btm2.tlog"/>
    <property name="defaultTransactionTimeout" value="3600”/>

</bean>

If you set the properties this way, you don’t have to worry about having a KC-specific bitronix-default-config.properties file or accidentally overriding Bitronix settings from Rice— the btmConfig bean should pull in the any default properties from that properties file, and you can override anything else you’d like to customize on the bean itself.

Hope that helps, and sorry for being a little late to the party— I actually tried to suggest a similar approach in earlier threads, but had a configuration issue in KIS and my emails never made it across :)

-Jeff Largent

[Ken Geis]

Nice fix, Jeff!

Ron, I've had enough problems with applications changing JVM-wide settings (e.g. Coeus sets the time zone to UTC) that I stopped running multiple applications in a servlet container when I can avoid it.


Ken

[Ronald Gouldner]

Excellent, fortunately I had another issue to address today which prevented me from doing what I was planning.   This is a better solution which I will try now !

 

Ron

[Dixit, Vivek]

Hello,

I experienced an issue over the last couple of days that is related to Bitronix as well, so continuing upon this thread.

2 days ago, when I started local tomcat, catalina.out  log stopped displaying any log statements after following set.. and tomcat did not start successfully.

...

2015-12-28 20:36:15,601 [localhost-startStop-1] D: U: []  WARN  org.kuali.rice.core.impl.config.property.JAXBConfigImpl ::   + Skipping non-existent location [/opt/sa_forms/java/dev/kra/kc-config.xml]

2015-12-28 20:36:15,601 [localhost-startStop-1] D: U: []  WARN  org.kuali.rice.core.impl.config.property.JAXBConfigImpl ::   + Skipping non-existent location []

2015-12-28 20:36:15,601 [localhost-startStop-1] D: U: []  INFO  org.kuali.rice.core.impl.config.property.JAXBConfigImpl :: - Parsed  config: [classpath:META-INF/kc-config.xml]

2015-12-28 20:36:15,604 [localhost-startStop-1] D: U: []  INFO  org.kuali.rice.core.impl.config.property.JAXBConfigImpl :: ----------------Rice Configuration Loaded-----------------

2015-12-28 20:36:15,604 [localhost-startStop-1] D: U: []  INFO  org.kuali.rice.core.impl.config.property.JAXBConfigImpl :: Loaded 553 properties

Next day, when I restarted tomcat again, it went further and showed following.. and started successfully.

2015-12-30 09:49:59,815 [localhost-startStop-1] D: U: []  INFO  org.kuali.rice.core.impl.config.property.JAXBConfigImpl :: Loaded 553 properties

2015-12-30 09:50:00,883 [localhost-startStop-1] D: U: []  WARN  bitronix.tm.journal.DiskJournal :: active log file is unclean, did you call BitronixTransactionManager.shutdown() at the end of the last run?

I just deleted all btm1.tlog and btm2.tlog files wherever I found them all over the file system and the error was gone. However since it looks more like a workaround than a solution, here is what I found on the net which looks like a proper solution to me.

http://bitronix-transaction-manager.10986.n7.nabble.com/questions-about-the-btm-tlog-files-td1533.html

at the very end..

"I was able to resolve this issue by calling TransactionManagerServices.getTransactionManager().shutdown() in the contextDestroyed(ServletContextEvent sce) method of my implementation of ServletContextListener. "

If this is the right solution, can we ask KualiCo to get this change in the monthly release?

Thanks!

Vivek