Exception communicating with Hive metastore server

[Ron Sher]

Hello,

I'm encountering the following error after rebuilding the DB with syncDB:

Exception communicating with Hive Metastore Server at localhost:8003

the Hue error.log shows the following:

[14/Jan/2013 18:34:26 +0000] supervisor   ERROR    Exception in supervisor main loop

Traceback (most recent call last):

  File "/usr/share/hue/desktop/core/src/desktop/supervisor.py", line 379, in main

    wait_loop(sups, options)

  File "/usr/share/hue/desktop/core/src/desktop/supervisor.py", line 389, in wait_loop

    time.sleep(1)

  File "/usr/share/hue/desktop/core/src/desktop/supervisor.py", line 218, in sig_handler

    raise SystemExit("Signal %d received. Exiting" % signum)

SystemExit: Signal 15 received. Exiting

Any suggestions?

Thanks,

Ron

[Romain Rigaux]

This is probably the same problem as https://groups.google.com/a/cloudera.org/d/msg/hue-user/7kr9l9Zddeg/jfaQvFaN8YsJ

Romain

[Ron Sher]

Hi,

Thanks for trying to help out.

chmod didn't seem to help.

Here's what I get in the /var/log/hue/runcpserver.log:

[15/Jan/2013 22:57:16 +0000] thrift_util  INFO     Thrift saw exception: timed out

[15/Jan/2013 22:57:16 +0000] middleware   INFO     Processing exception: timed out: Traceback (most recent call last):

  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/handlers/base.py", line 100, in get_response

    response = callback(request, *callback_args, **callback_kwargs)

  File "/usr/share/hue/apps/beeswax/src/beeswax/views.py", line 98, in index

    tables = db_utils.meta_client().get_tables("default", ".*")

  File "/usr/share/hue/desktop/core/src/desktop/lib/thrift_util.py", line 285, in wrapper

    ret = res(*args, **kwargs)

  File "/usr/share/hue/desktop/core/src/desktop/lib/thrift_util.py", line 331, in wrapper

    ret = res(*args, **kwargs)

  File "/usr/share/hue/apps/beeswax/gen-py/hive_metastore/ThriftHiveMetastore.py", line 989, in get_tables

    return self.recv_get_tables()

  File "/usr/share/hue/apps/beeswax/gen-py/hive_metastore/ThriftHiveMetastore.py", line 1001, in recv_get_tables

    (fname, mtype, rseqid) = self._iprot.readMessageBegin()

  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Thrift-0.1-py2.6-linux-x86_64.egg/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin

    sz = self.readI32()

  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Thrift-0.1-py2.6-linux-x86_64.egg/thrift/protocol/TBinaryProtocol.py", line 203, in readI32

    buff = self.trans.readAll(4)

  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Thrift-0.1-py2.6-linux-x86_64.egg/thrift/transport/TTransport.py", line 58, in readAll

    chunk = self.read(sz-have)

  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Thrift-0.1-py2.6-linux-x86_64.egg/thrift/transport/TTransport.py", line 155, in read

    self.__rbuf = StringIO(self.__trans.read(max(sz, self.DEFAULT_BUFFER)))

  File "/usr/share/hue/build/env/lib/python2.6/site-packages/Thrift-0.1-py2.6-linux-x86_64.egg/thrift/transport/TSocket.py", line 92, in read

    buff = self.handle.recv(sz)

  File "/usr/share/hue/build/env/lib/python2.6/site-packages/eventlet-0.9.14-py2.6.egg/eventlet/greenio.py", line 238, in recv

    timeout_exc=socket.timeout("timed out"))

  File "/usr/share/hue/build/env/lib/python2.6/site-packages/eventlet-0.9.14-py2.6.egg/eventlet/hubs/__init__.py", line 121, in trampoline

    return hub.switch()

  File "/usr/share/hue/build/env/lib/python2.6/site-packages/eventlet-0.9.14-py2.6.egg/eventlet/hubs/hub.py", line 177, in switch

    return self.greenlet.switch()

timeout: timed out

[15/Jan/2013 22:57:16 +0000] access       INFO     192.168.10.178 ron - "POST /profile HTTP/1.0"

[15/Jan/2013 22:57:16 +0000] access       INFO     192.168.10.178 ron - "GET /debug/check_config_ajax HTTP/1.0"

[Romain Rigaux]

The logs show that the client is time outing probably because the server (Beeswax) could not start or died.

Do you have the beeswax.logs?

What does this give you?
ls -l /var/lib/hive
ls -l /var/lib/hive/metastore

Romain

[Ron Sher]

[root@hadoop-local ron]# ls -l /var/lib/hive

total 4

drwxrwxrwx 3 hive hive 4096 Dec  2 04:00 metastore

[root@hadoop-local ron]# ls -l /var/lib/hive/metastore

total 4

drwxrwxrwx 5 root root 4096 Jan 16 09:07 metastore_db

beeswax.out shows:

[root@hadoop-local ron]# less /var/log/hue/beeswax_server.out

        at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:377)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:364)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:403)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:309)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:273)

        at com.cloudera.beeswax.Server.serveMeta(Server.java:349)

        at com.cloudera.beeswax.Server$1.run(Server.java:200)

        at java.lang.Thread.run(Thread.java:662)

NestedThrowablesStackTrace:

org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.

        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)

        at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:521)

        at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:290)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:593)

        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:300)

        at org.datanucleus.ObjectManagerFactoryImpl.initialiseStoreManager(ObjectManagerFactoryImpl.java:161)

        at org.datanucleus.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:583)

        at org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:286)

        at org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)

        at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)

        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)

        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)

        at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:248)

        at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:277)

        at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:210)

        at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:185)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:62)

        at org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:377)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:364)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:403)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:309)

        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:273)

        at com.cloudera.beeswax.Server.serveMeta(Server.java:349)

        at com.cloudera.beeswax.Server$1.run(Server.java:200)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.util.NoSuchElementException: Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.

        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1191)

        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

[Romain Rigaux]

Getting there!

You are not using ClouderaManager right?

In this case the Hive metastore needs to belong to the user starting Beeswax and Hue (the 'ron' user).

First you happen to have a process already connected to the metastore (default DerbyDB supports only one user concurrently: https://ccp.cloudera.com/display/CDH4DOC/Hive+Installation#HiveInstallation-ConfiguringtheMetastoreDatabase) so you can 'ps -ef | grep beeswax', 'ps -ef | grep hive' and kill the process (or stop it cleanly if it is Hive Server 1 or 2).

Then make your metastore readable:
sudo chmod -R 1777 /var/lib/hive/metastore

And restart beeswax. If any other problems, the beeswax logs should detail the cause.

Romain

[Ron Sher]

Did that, but didn't seem to help :-(

it now fails with:

Another instance of Derby may have already booted the database /var/lib/hive/metastore/metastore_db

Thanks again for your time,

Ron

[Romain Rigaux]

You can see which process is using it and you can kill it:
lsof /var/lib/hive/metastore/metastore_db/dbex.lck
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
java    11348 romain  261uw  REG    8,5        4 3672416 /var/lib/hive/metastore/metastore_db/dbex.lck

You can then delete the lock files and restart Beeswax:
sudo rm /var/lib/hive/metastore/metastore_db/db*.lck

Romain

[Ron Sher]

Working, finally, thanks a lot!!!!

On Thu, Jan 17, 2013 at 12:55 AM, Romain Rigaux <rom...@cloudera.com> wrote:

sudo rm /var/lib/hive/metastore/metastore_db/db*.lck