Re: H2 FILE_LOCK=FILE with Red Hat enterprise Linux hangs the system

[Noel Grandin]

Can you post the thread dump?
And what version of H2 you are using.

On 2012-10-26 09:54, Tejas Shah wrote:

I am using H2 with default settings on Red Hat Enterprise Linux and it causes my tocat server to hang. I created a thread dump and it looks like H2 keep waiting for gaining lock over the database file. No Exceptions anywhere. No deadlocks detected in thread dump. I have tried using sun/Oracle JDK as well as Open JDK. Please note the same application works fine under Windows 7, ubuntu 12.04 and Cent OS 5.x

Solution to this was change it to use: FILE_LOCK=SOCKET 

Now, can anybody tell me if I can use this or not? I have a single application accessing the database.

--
You received this message because you are subscribed to the Google Groups "H2 Database" group.
To view this discussion on the web visit https://groups.google.com/d/msg/h2-database/-/GCG1LYWYLvcJ.
To post to this group, send email to h2-da...@googlegroups.com.
To unsubscribe from this group, send email to h2-database...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/h2-database?hl=en.

[Tejas Shah]

I'm using H2 1.3.166 with Hibernate and Spring on...

java version "1.6.0_25"

Java(TM) SE Runtime Environment (build 1.6.0_25-b06)

Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)

Here is the thread dump:

Server compiler detected.

JVM version is 20.0-b11

Deadlock Detection:

No deadlocks found.

Thread 18107: (state = BLOCKED)

 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)

 - org.apache.log4j.helpers.FileWatchdog.run() @bci=11, line=103 (Interpreted frame)

Thread 18106: (state = BLOCKED)

 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)

 - sun.misc.GC$Daemon.run() @bci=51, line=100 (Interpreted frame)

Thread 18101: (state = BLOCKED)

Thread 18100: (state = BLOCKED)

 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)

 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)

 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted frame)

 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)

Thread 18099: (state = BLOCKED)

 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)

 - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)

 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Interpreted frame)

 Thread 18087: (state = BLOCKED)

 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)

 - org.h2.store.FileLock.sleep(int) @bci=2, line=428 (Interpreted frame)

 - org.h2.store.FileLock.lockFile() @bci=118, line=334 (Interpreted frame)

 - org.h2.store.FileLock.lock(int) @bci=49, line=128 (Interpreted frame)

 - org.h2.engine.Database.open(int, int) @bci=403, line=542 (Interpreted frame)

 - org.h2.engine.Database.openDatabase(int, int, boolean) @bci=3, line=222 (Interpreted frame)

 - org.h2.engine.Database.<init>(org.h2.engine.ConnectionInfo, java.lang.String) @bci=511, line=217 (Interpreted frame)

 - org.h2.engine.Engine.openSession(org.h2.engine.ConnectionInfo, boolean, java.lang.String) @bci=91, line=56 (Interpreted frame)

 - org.h2.engine.Engine.openSession(org.h2.engine.ConnectionInfo) @bci=39, line=159 (Interpreted frame)

 - org.h2.engine.Engine.createSessionAndValidate(org.h2.engine.ConnectionInfo) @bci=51, line=138 (Interpreted frame)

 - org.h2.engine.Engine.createSession(org.h2.engine.ConnectionInfo) @bci=4, line=121 (Interpreted frame)

 - org.h2.engine.Engine.createSession(org.h2.engine.ConnectionInfo) @bci=2, line=28 (Interpreted frame)

 - org.h2.engine.SessionRemote.connectEmbeddedOrServer(boolean) @bci=97, line=305 (Interpreted frame)

 - org.h2.jdbc.JdbcConnection.<init>(org.h2.engine.ConnectionInfo, boolean) @bci=48, line=110 (Interpreted frame)

 - org.h2.jdbc.JdbcConnection.<init>(java.lang.String, java.util.Properties) @bci=11, line=94 (Interpreted frame)

 - org.h2.Driver.connect(java.lang.String, java.util.Properties) @bci=59, line=72 (Interpreted frame)

 - org.apache.commons.dbcp.DriverConnectionFactory.createConnection() @bci=12, line=38 (Interpreted frame)

 - org.apache.commons.dbcp.PoolableConnectionFactory.makeObject() @bci=4, line=582 (Interpreted frame)

 - org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(org.apache.commons.dbcp.PoolableConnectionFactory) @bci=3, line=1556 (Interpreted frame)

 - org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(org.apache.commons.dbcp.ConnectionFactory, org.apache.commons.pool.KeyedObjectPoolFactory, org.apache.commons.dbcp.AbandonedConfig) @bci=49, line=1545 (Interpreted frame)

 - org.apache.commons.dbcp.BasicDataSource.createDataSource() @bci=71, line=1388 (Interpreted frame)

 - org.apache.commons.dbcp.BasicDataSource.getConnection() @bci=1, line=1044 (Interpreted frame)

 - org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection() @bci=36, line=46 (Interpreted frame)

 - org.hibernate.cfg.SettingsFactory.buildSettings(java.util.Properties) @bci=70, line=111 (Interpreted frame)

 - org.hibernate.cfg.Configuration.buildSettings(java.util.Properties) @bci=5, line=2101 (Interpreted frame)

 - org.hibernate.cfg.Configuration.buildSessionFactory() @bci=68, line=1325 (Interpreted frame)

 - org.hibernate.cfg.AnnotationConfiguration.buildSessionFactory() @bci=372, line=867 (Interpreted frame)

 - org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory() @bci=50, line=669 (Interpreted frame)

 - org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(javax.persistence.spi.PersistenceUnitInfo, java.util.Map) @bci=23, line=132 (Interpreted frame)

 - org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory() @bci=229, line=242 (Interpreted frame)

 - org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet() @bci=220, line=310 (Interpreted frame)

 - org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(java.lang.String, java.lang.Object, org.springframework.beans.factory.support.RootBeanDefinition) @bci=107, line=1504 (Interpreted frame)

 - org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(java.lang.String, java.lang.Object, org.springframework.beans.factory.support.RootBeanDefinition) @bci=61, line=1442 (Interpreted frame)

 - org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(java.lang.String, org.springframework.beans.factory.support.RootBeanDefinition, java.lang.Object[]) @bci=230, line=522 (Interpreted frame)

 - org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(java.lang.String, org.springframework.beans.factory.support.RootBeanDefinition, java.lang.Object[]) @bci=122, line=459 (Interpreted frame)

 - org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject() @bci=16, line=294 (Interpreted frame)

 - org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(java.lang.String, org.springframework.beans.factory.ObjectFactory) @bci=126, line=225 (Interpreted frame)

 - org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(java.lang.String, java.lang.Class, java.lang.Object[], boolean) @bci=307, line=291 (Interpreted frame)

 - org.springframework.beans.factory.support.AbstractBeanFactory.getBean(java.lang.String) @bci=5, line=193 (Interpreted frame)

 - org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons() @bci=136, line=567 (Interpreted frame)

 - org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(org.springframework.beans.factory.config.ConfigurableListableBeanFactory) @bci=62, line=913 (Interpreted frame)

 - org.springframework.context.support.AbstractApplicationContext.refresh() @bci=54, line=464 (Interpreted frame)

 - org.springframework.web.context.ContextLoader.createWebApplicationContext(javax.servlet.ServletContext, org.springframework.context.ApplicationContext) @bci=245, line=276 (Interpreted frame)

 - org.springframework.web.context.ContextLoader.initWebApplicationContext(javax.servlet.ServletContext) @bci=69, line=197 (Interpreted frame)

 - org.springframework.web.context.ContextLoaderListener.contextInitialized(javax.servlet.ServletContextEvent) @bci=28, line=47 (Interpreted frame)

 - org.apache.catalina.core.StandardContext.listenerStart() @bci=455, line=3972 (Interpreted frame)

 - org.apache.catalina.core.StandardContext.start() @bci=1253, line=4467 (Interpreted frame)

 - org.apache.catalina.core.ContainerBase.addChildInternal(org.apache.catalina.Container) @bci=154, line=791 (Interpreted frame)

 - org.apache.catalina.core.ContainerBase.addChild(org.apache.catalina.Container) @bci=26, line=771 (Interpreted frame)

 - org.apache.catalina.core.StandardHost.addChild(org.apache.catalina.Container) @bci=25, line=526 (Interpreted frame)

 - org.apache.catalina.startup.HostConfig.deployWAR(java.lang.String, java.io.File, java.lang.String) @bci=617, line=905 (Interpreted frame)

 - org.apache.catalina.startup.HostConfig.deployWARs(java.io.File, java.lang.String[]) @bci=225, line=740 (Interpreted frame)

 - org.apache.catalina.startup.HostConfig.deployApps() @bci=25, line=500 (Interpreted frame)

 - org.apache.catalina.startup.HostConfig.start() @bci=150, line=1277 (Interpreted frame)

 - org.apache.catalina.startup.HostConfig.lifecycleEvent(org.apache.catalina.LifecycleEvent) @bci=132, line=321 (Interpreted frame)

 - org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(java.lang.String, java.lang.Object) @bci=32, line=119 (Interpreted frame)

 - org.apache.catalina.core.ContainerBase.start() @bci=309, line=1053 (Interpreted frame)

 - org.apache.catalina.core.StandardHost.start() @bci=319, line=722 (Interpreted frame)

 - org.apache.catalina.core.ContainerBase.start() @bci=265, line=1045 (Interpreted frame)

 - org.apache.catalina.core.StandardEngine.start() @bci=217, line=443 (Interpreted frame)

 - org.apache.catalina.core.StandardService.start() @bci=136, line=516 (Interpreted frame)

 - org.apache.catalina.core.StandardServer.start() @bci=85, line=710 (Interpreted frame)

 - org.apache.catalina.startup.Catalina.start() @bci=51, line=593 (Interpreted frame)

 - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)

 - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=39 (Interpreted frame)

 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Interpreted frame)

 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Interpreted frame)

 - org.apache.catalina.startup.Bootstrap.start() @bci=32, line=289 (Interpreted frame)

 - org.apache.catalina.startup.Bootstrap.main(java.lang.String[]) @bci=137, line=414 (Interpreted frame)

--
---------------------
Tejas M Shah
Group Lead - Tech

ArguSoft India Ltd
Tel: +91 79 23287029  Xtn: 109
cell: +91 9974550108

[Thomas Mueller]

Hi,

That's strange. I never saw a case so far where the file locking mechanism was hanging (either it works within a second or, or it fails). Are you sure it actually hangs, or is it possibly just slow (taking a second or so)?

What file system do you use? Maybe the last modified time doesn't match the system time. I saw this in some cases (when using NFS for example).

I don't recommend using FILE_LOCK=SOCKET because it is rather dangerous as it relies on network connectivity and correctly configured network settings. I would rather suggest to use FILE_LOCK=FS. Could you try that?

Regards,

Thomas

[Noel Grandin]

I'm guessing, but based on my previous experience with Tomcat, you're probably trying to write to a directory which Tomcat believes you don't have permission to access.
Tomcat installs it's own SecurityContext which restricts each WAR to have access to only certain directories.
You need to do something in the configuration to make it work.

Thomas, our problem is that FilePathDisk#createFile() cannot tell the difference between
"access denied because of permission problem"
and
"access denied because of locking problem"

Perhaps when we start using the new JDK7 file API's, we'll be able to tell the difference.

[Tejas Shah]

umm...I do not think Tomcat security can be the issue as H2 is able to create the 3 files. db, lock and trace. After that it hangs.

@Thomas, yes it hangs I waited for 15 minutes at least and even 45 minutes in some cases. I did not get any idea initially about what was going on, so I kept trying to see something in logs, but no luck. The deployment starts for sometime, until Spring framework initiates hibernate and Hibernate initiates it's data-source. That's it, it hangs and keep waiting there. Tomcat shows 503 for all the applications deployed(Do not think, other applications can be causing problems, as I tried deploying it individually also). 

[Thomas Mueller]

Hi,

In my previous mail I suggest to use FILE_LOCK=FS. Could you try that? What is the result?

Regards,

Thomas

--
You received this message because you are subscribed to the Google Groups "H2 Database" group.

To view this discussion on the web visit https://groups.google.com/d/msg/h2-database/-/aLNMkKCsa4wJ.