Hypertable error!!

[ciskshit...@gmail.com]

Hello There,

How are you ? I am trying to setup Hypertable on hadoop multi node.(I have 2 systems in my cluser named "master" and "slave")
When I run "cap start" command its gives the following output on terminal

root@CISM479:~# cap start
  * executing `start'
 ** transaction: start
  * executing `start_servers'
  * executing `start_hyperspace'
  * executing "/opt/hypertable/current/bin/start-hyperspace.sh       --config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg"
    servers: ["master"]
    [master] executing command
 ** [out :: master] Started Hyperspace
    command finished in 5296ms
  * executing `start_master'
  * executing "/opt/hypertable/current/bin/start-dfsbroker.sh hadoop       --config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg &&\\\n   /opt/hypertable/current/bin/start-master.sh --config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg &&\\\n   /opt/hypertable/current/bin/start-monitoring.sh"
    servers: ["master"]
    [master] executing command
 ** [out :: master] DFS broker: available file descriptors: 1024
 ** [out :: master] Started DFS Broker (hadoop)
 ** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come up...
 ** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come up...
 ** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come up...
 ** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come up...
 ** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come up...
 ** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come up...
 ** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come up...
 ** [out :: master] Waiting for Hypertable.Master (localhost:38050) to come up...
 ** [out :: master] ERROR: Hypertable.Master did not come up
    command finished in 133321ms
failed: "sh -c '/opt/hypertable/current/bin/start-dfsbroker.sh hadoop       --config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg &&\\\n   /opt/hypertable/current/bin/start-master.sh --config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg &&\\\n   /opt/hypertable/current/bin/start-monitoring.sh'" on master


Can you please let me know the possible causes of the issue ? Below is the content from log files

Log file  :DfsBroker.hadoop.log

Num CPUs=2
HdfsBroker.Port=38030
HdfsBroker.Reactors=2
HdfsBroker.Workers=20
HdfsBroker.Hadoop.ConfDir=/usr/lib/hadoop-0.20/conf
Adding hadoop configuration file /usr/lib/hadoop-0.20/conf/hdfs-site.xml
Adding hadoop configuration file /usr/lib/hadoop-0.20/conf/core-site.xml
HdfsBroker.dfs.client.read.shortcircuit=false
HdfsBroker.dfs.replication=2
HdfsBroker.Server.fs.default.name=hdfs://master:54310
5 Mar, 2014 3:10:08 PM org.hypertable.AsyncComm.IOHandler DeliverEvent
INFO: [/192.168.2.225:56729 ; Wed Mar 05 15:10:08 IST 2014] Connection Established
5 Mar, 2014 3:10:10 PM org.hypertable.DfsBroker.hadoop.ConnectionHandler handle
INFO: [/192.168.2.225:56729 ; Wed Mar 05 15:10:10 IST 2014] Disconnect - COMM broken connection : Closing all open handles from /192.168.2.225:56729
Closed 0 input streams and 0 output streams for client connection /192.168.2.225:56729
5 Mar, 2014 3:10:11 PM org.hypertable.AsyncComm.IOHandler DeliverEvent
INFO: [/192.168.2.225:57752 ; Wed Mar 05 15:10:11 IST 2014] Connection Established
5 Mar, 2014 3:10:11 PM org.hypertable.DfsBroker.hadoop.ConnectionHandler handle
INFO: [/192.168.2.225:57752 ; Wed Mar 05 15:10:11 IST 2014] Disconnect - COMM broken connection : Closing all open handles from /192.168.2.225:57752
Closed 0 input streams and 0 output streams for client connection /192.168.2.225:57752
5 Mar, 2014 3:10:13 PM org.hypertable.AsyncComm.IOHandler DeliverEvent
INFO: [/192.168.2.225:57119 ; Wed Mar 05 15:10:13 IST 2014] Connection Established
5 Mar, 2014 3:10:13 PM org.hypertable.AsyncComm.IOHandlerData handle_message_body
WARNING: Received response for non-pending event (id=0,version=1,total_len=38)
5 Mar, 2014 3:10:13 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Exists
INFO: Testing for existence of file '/hypertable/servers/master/log/mml
5 Mar, 2014 3:10:13 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Exists
INFO: Testing for existence of file '/hypertable/servers/master/log/mml
5 Mar, 2014 3:10:13 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Mkdirs
INFO: Making directory '/hypertable/servers/master/log/mml'
5 Mar, 2014 3:10:13 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Mkdirs
SEVERE: I/O exception while making directory '/hypertable/servers/master/log/mml' - org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /hypertable/servers/master/log/mml. Name node is in safe mode.
Resources are low on NN. Safe mode must be turned off manually.
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:2140)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2114)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs(NameNode.java:892)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1439)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1435)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1433)

5 Mar, 2014 3:10:13 PM org.hypertable.DfsBroker.hadoop.ConnectionHandler handle
INFO: [/192.168.2.225:57119 ; Wed Mar 05 15:10:13 IST 2014] Disconnect - COMM broken connection : Closing all open handles from /192.168.2.225:57119
Closed 0 input streams and 0 output streams for client connection /192.168.2.225:57119


Log file  : Hyperspace.log


1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Common/Config.cc:601) Initializing Hyperspace.Master (Hypertable 0.9.7.16 (v0.9.7.16-0-g817a161))...
CPU cores count=2
CephBroker.MonAddr=10.0.1.245:6789
DfsBroker.Local.Root=fs/local
DfsBroker.Port=38030
HdfsBroker.Hadoop.ConfDir=/usr/lib/hadoop-0.20/conf
Hyperspace.GracePeriod=200000
Hyperspace.KeepAlive.Interval=30000
Hyperspace.Lease.Interval=1000000
Hyperspace.Replica.Dir=hyperspace
Hyperspace.Replica.Host=[master]
Hyperspace.Replica.Port=38040
Hyperspace.Replica.Reactors=2
Hypertable.Master.Port=38050
Hypertable.RangeServer.LowActivityPeriod=["* 2-4 * * *"]
Hypertable.RangeServer.Port=38060
Hypertable.Verbose=true
Qfs.MetaServer.Name=master
Qfs.MetaServer.Port=20000
ThriftBroker.Port=38080
config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg
dir=hyperspace
keepalive=30000
lease-interval=1000000
pidfile=/opt/hypertable/current/run/Hyperspace.pid
port=38040
reactors=2
verbose=true
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:157) BerkeleyDB base directory = '/opt/hypertable/0.9.7.16/hyperspace'
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:111) localhost=CISM479 localip=192.168.2.225
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:125) Removing statedb
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:333) Replication master init done
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff13c32b700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff13bb2a700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff13b329700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff13ab28700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff13a327700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff139b26700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff139325700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff138b24700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff138323700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff137b22700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff137321700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff136b20700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff13631f700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff135b1e700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff13531d700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff134b1c700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff13431b700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff133b1a700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff133319700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff132b18700
1394012400 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:500) Created DB handles for thread: 7ff142473740
1394012403 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:269) Create session for 192.168.2.225:59353
1394012403 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:281) created session 1
1394012403 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/request/RequestHandlerRenewSession.cc:74) Session handle 1 created
1394012403 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=192.168.2.225:57446
1394012403 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:343) Initialized session 1 (serverup)
1394012403 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/request/RequestHandlerDestroySession.cc:42) Destroying session 1
1394012403 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:317) destroyed session 1(serverup)
1394012403 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:468) Expiring session 1 name=serverup
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:269) Create session for 192.168.2.225:40061
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:281) created session 2
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/request/RequestHandlerRenewSession.cc:74) Session handle 2 created
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=192.168.2.225:50565
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:343) Initialized session 2 (Hypertable.Master)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284) exists(session_id=2, name=/hypertable)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting exists(session_id=2, name=/hypertable)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284) exists(session_id=2, name=/hypertable/master)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting exists(session_id=2, name=/hypertable/master)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1935) open(session_id=2, session_name = Hypertable.Master, fname=/hypertable/master, flags=0x7, event_mask=0x0)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2064) handle 1 created ('/hypertable/master', session=2(Hypertable.Master), flags=0x7, mask=0x0)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:703) exitting open(session_id=2, session_name = Hypertable.Master, fname=/hypertable/master, flags=0x7, event_mask=0x0)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1237) lock(session=2(Hypertable.Master), handle=1, mode=0x2, try_lock=1)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1356) lock txn={BDbTxn m_handle_namespace_db=0x16a25a0, m_handle_state_db=0x16a26c0, m_db_txn=0} commited  handle=1 node=/hypertable/master mode=2 status=1 lock_generation=3 notification_count=0
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2389) attrexists(session=2(Hypertable.Master), handle=1, attr=next_server_id)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284) exists(session_id=2, name=/hypertable/servers)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting exists(session_id=2, name=/hypertable/servers)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284) exists(session_id=2, name=/hypertable/tables)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting exists(session_id=2, name=/hypertable/tables)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1935) open(session_id=2, session_name = Hypertable.Master, fname=/hypertable/root, flags=0xb, event_mask=0x0)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2064) handle 2 created ('/hypertable/root', session=2(Hypertable.Master), flags=0xb, mask=0x0)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:703) exitting open(session_id=2, session_name = Hypertable.Master, fname=/hypertable/root, flags=0xb, event_mask=0x0)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:755) close(session=2(Hypertable.Master), handle=2)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284) exists(session_id=2, name=/hypertable/namemap/names)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting exists(session_id=2, name=/hypertable/namemap/names)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2284) exists(session_id=2, name=/hypertable/namemap/ids)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2290) exitting exists(session_id=2, name=/hypertable/namemap/ids)
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/request/RequestHandlerDestroySession.cc:42) Destroying session 2
1394012413 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:317) destroyed session 2(Hypertable.Master)
1394012414 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:468) Expiring session 2 name=Hypertable.Master
1394012414 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:487) Destroying handle 1
1394012414 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1541) Persisting lock released notifications
1394012414 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1552) Finished persisting lock released notifications


Log file  : DfsBroker.local.log
FILE WAS EMPTY

Log file  : Hypertable.Master.log

1394012413 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Common/Config.cc:601) Initializing Hypertable.Master (Hypertable 0.9.7.16 (v0.9.7.16-0-g817a161))...
CPU cores count=2
CephBroker.MonAddr=10.0.1.245:6789
DfsBroker.Local.Root=fs/local
DfsBroker.Port=38030
HdfsBroker.Hadoop.ConfDir=/usr/lib/hadoop-0.20/conf
Hyperspace.GracePeriod=200000
Hyperspace.KeepAlive.Interval=30000
Hyperspace.Lease.Interval=1000000
Hyperspace.Replica.Dir=hyperspace
Hyperspace.Replica.Host=[master]
Hyperspace.Replica.Port=38040
Hypertable.Master.Port=38050
Hypertable.Master.Reactors=2
Hypertable.RangeServer.LowActivityPeriod=["* 2-4 * * *"]
Hypertable.RangeServer.Port=38060
Hypertable.Verbose=true
Qfs.MetaServer.Name=master
Qfs.MetaServer.Port=20000
ThriftBroker.Port=38080
config=/opt/hypertable/0.9.7.16/conf/hypertable.cfg
dfs-port=38030
grace-period=200000
hs-host=[master]
hs-port=38040
keepalive=30000
lease-interval=1000000
pidfile=/opt/hypertable/current/run/Hypertable.Master.pid
port=38050
reactors=2
verbose=true
1394012413 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/main.cc:434) Obtained lock on '/hypertable/master'
1394012413 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:75) rangeservers monitoring stats dir /opt/hypertable/0.9.7.16/run/monitoring exists
1394012413 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:75) rangeservers monitoring stats dir /opt/hypertable/0.9.7.16/run/monitoring/tables exists
1394012413 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:75) rangeservers monitoring stats dir /opt/hypertable/0.9.7.16/run/monitoring/rangeservers exists
1394012413 ERROR Hypertable.Master : main (/root/src/hypertable/src/cc/Hypertable/Master/main.cc:384): Hypertable::Exception: Error mkdirs DFS directory /hypertable/servers/master/log/mml - DFS BROKER i/o error
    at virtual void Hypertable::DfsBroker::Client::mkdirs(const String&) (/root/src/hypertable/src/cc/DfsBroker/Lib/Client.cc:517)
    at virtual void Hypertable::DfsBroker::Client::mkdirs(const String&) (/root/src/hypertable/src/cc/DfsBroker/Lib/Client.cc:514): org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /hypertable/servers/master/lo

Thank you.

[Doug Judd]

HDFS is in "safe mode" as indicated by this message:

Resources are low on NN. Safe mode must be turned off manually.

You need to resolve this first before running Hypertable on top of HDFS.  This sometimes means that all of the DataNodes did not start up properly.

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-de...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-dev.
For more options, visit https://groups.google.com/groups/opt_out.

--

Doug Judd

CEO, Hypertable Inc.