Exception communicating with Hive metastore server

1251 views
Skip to first unread message

Ron Sher

unread,
Jan 15, 2013, 8:11:32 AM1/15/13
to hue-user
Hello,

I'm encountering the following error after rebuilding the DB with syncDB:
Exception communicating with Hive Metastore Server at localhost:8003

the Hue error.log shows the following:

[14/Jan/2013 18:34:26 +0000] supervisor   ERROR    Exception in supervisor main loop
Traceback (most recent call last):
  File "/usr/share/hue/desktop/core/src/desktop/supervisor.py", line 379, in main
    wait_loop(sups, options)
  File "/usr/share/hue/desktop/core/src/desktop/supervisor.py", line 389, in wait_loop
    time.sleep(1)
  File "/usr/share/hue/desktop/core/src/desktop/supervisor.py", line 218, in sig_handler
    raise SystemExit("Signal %d received. Exiting" % signum)
SystemExit: Signal 15 received. Exiting


Any suggestions?

Thanks,
Ron

Romain Rigaux

unread,
Jan 15, 2013, 1:05:57 PM1/15/13
to Ron Sher, hue-user

Ron Sher

unread,
Jan 16, 2013, 1:59:45 AM1/16/13
to Romain Rigaux, hue-user
Hi,

Thanks for trying to help out.
chmod didn't seem to help.
Here's what I get in the /var/log/hue/runcpserver.log:

[15/Jan/2013 22:57:16 +0000] thrift_util  INFO     Thrift saw exception: timed out
[15/Jan/2013 22:57:16 +0000] middleware   INFO     Processing exception: timed out: Traceback (most recent call last):
  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/handlers/base.py", line 100, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/share/hue/apps/beeswax/src/beeswax/views.py", line 98, in index
    tables = db_utils.meta_client().get_tables("default", ".*")
  File "/usr/share/hue/desktop/core/src/desktop/lib/thrift_util.py", line 285, in wrapper
    ret = res(*args, **kwargs)
  File "/usr/share/hue/desktop/core/src/desktop/lib/thrift_util.py", line 331, in wrapper
    ret = res(*args, **kwargs)
  File "/usr/share/hue/apps/beeswax/gen-py/hive_metastore/ThriftHiveMetastore.py", line 989, in get_tables
    return self.recv_get_tables()
  File "/usr/share/hue/apps/beeswax/gen-py/hive_metastore/ThriftHiveMetastore.py", line 1001, in recv_get_tables
    (fname, mtype, rseqid) = self._iprot.readMessageBegin()
  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Thrift-0.1-py2.6-linux-x86_64.egg/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin
    sz = self.readI32()
  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Thrift-0.1-py2.6-linux-x86_64.egg/thrift/protocol/TBinaryProtocol.py", line 203, in readI32
    buff = self.trans.readAll(4)
  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Thrift-0.1-py2.6-linux-x86_64.egg/thrift/transport/TTransport.py", line 58, in readAll
    chunk = self.read(sz-have)
  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Thrift-0.1-py2.6-linux-x86_64.egg/thrift/transport/TTransport.py", line 155, in read
    self.__rbuf = StringIO(self.__trans.read(max(sz, self.DEFAULT_BUFFER)))
  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Thrift-0.1-py2.6-linux-x86_64.egg/thrift/transport/TSocket.py", line 92, in read
    buff = self.handle.recv(sz)
  File "/usr/share/hue/build/env/lib/python2.6/site-packages/eventlet-0.9.14-py2.6.egg/eventlet/greenio.py", line 238, in recv
    timeout_exc=socket.timeout("timed out"))
  File "/usr/share/hue/build/env/lib/python2.6/site-packages/eventlet-0.9.14-py2.6.egg/eventlet/hubs/__init__.py", line 121, in trampoline
    return hub.switch()
  File "/usr/share/hue/build/env/lib/python2.6/site-packages/eventlet-0.9.14-py2.6.egg/eventlet/hubs/hub.py", line 177, in switch
    return self.greenlet.switch()
timeout: timed out

[15/Jan/2013 22:57:16 +0000] access       INFO     192.168.10.178 ron - "POST /profile HTTP/1.0"
[15/Jan/2013 22:57:16 +0000] access       INFO     192.168.10.178 ron - "GET /debug/check_config_ajax HTTP/1.0"

Romain Rigaux

unread,
Jan 16, 2013, 2:11:36 AM1/16/13
to Ron Sher, hue-user
The logs show that the client is time outing probably because the server (Beeswax) could not start or died.

Do you have the beeswax.logs?

What does this give you?
ls -l /var/lib/hive
ls -l /var/lib/hive/metastore

Romain

Ron Sher

unread,
Jan 16, 2013, 2:31:04 AM1/16/13
to Romain Rigaux, hue-user
[root@hadoop-local ron]# ls -l /var/lib/hive
total 4
drwxrwxrwx 3 hive hive 4096 Dec  2 04:00 metastore
[root@hadoop-local ron]# ls -l /var/lib/hive/metastore
total 4
drwxrwxrwx 5 root root 4096 Jan 16 09:07 metastore_db

beeswax.out shows:
[root@hadoop-local ron]# less /var/log/hue/beeswax_server.out
        at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:377)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:364)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:403)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:309)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:273)
        at com.cloudera.beeswax.Server.serveMeta(Server.java:349)
        at com.cloudera.beeswax.Server$1.run(Server.java:200)
        at java.lang.Thread.run(Thread.java:662)
NestedThrowablesStackTrace:
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:521)
        at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:290)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:593)
        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:300)
        at org.datanucleus.ObjectManagerFactoryImpl.initialiseStoreManager(ObjectManagerFactoryImpl.java:161)
        at org.datanucleus.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:583)
        at org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:286)
        at org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
        at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:248)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:277)
        at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:210)
        at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:185)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
        at org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:62)
        at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:377)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:364)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:403)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:309)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:273)
        at com.cloudera.beeswax.Server.serveMeta(Server.java:349)
        at com.cloudera.beeswax.Server$1.run(Server.java:200)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.util.NoSuchElementException: Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1191)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)


Romain Rigaux

unread,
Jan 16, 2013, 3:15:24 AM1/16/13
to Ron Sher, hue-user
Getting there!

You are not using ClouderaManager right?

In this case the Hive metastore needs to belong to the user starting Beeswax and Hue (the 'ron' user).

First you happen to have a process already connected to the metastore (default DerbyDB supports only one user concurrently: https://ccp.cloudera.com/display/CDH4DOC/Hive+Installation#HiveInstallation-ConfiguringtheMetastoreDatabase) so you can 'ps -ef | grep beeswax', 'ps -ef | grep hive' and kill the process (or stop it cleanly if it is Hive Server 1 or 2).

Then make your metastore readable:
sudo chmod -R 1777 /var/lib/hive/metastore

And restart beeswax. If any other problems, the beeswax logs should detail the cause.

Romain

Ron Sher

unread,
Jan 16, 2013, 4:02:59 AM1/16/13
to Romain Rigaux, hue-user
Did that, but didn't seem to help :-(

it now fails with:
Another instance of Derby may have already booted the database /var/lib/hive/metastore/metastore_db

Thanks again for your time,
Ron

Romain Rigaux

unread,
Jan 16, 2013, 5:55:13 PM1/16/13
to Ron Sher, hue-user
You can see which process is using it and you can kill it:
lsof /var/lib/hive/metastore/metastore_db/dbex.lck
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
java    11348 romain  261uw  REG    8,5        4 3672416 /var/lib/hive/metastore/metastore_db/dbex.lck

You can then delete the lock files and restart Beeswax:
sudo rm /var/lib/hive/metastore/metastore_db/db*.lck

Romain

Ron Sher

unread,
Jan 17, 2013, 6:35:46 AM1/17/13
to Romain Rigaux, hue-user
Working, finally, thanks a lot!!!!
On Thu, Jan 17, 2013 at 12:55 AM, Romain Rigaux <rom...@cloudera.com> wrote:
sudo rm /var/lib/hive/metastore/metastore_db/db*.lck


Reply all
Reply to author
Forward
0 new messages