RangeServer crashes

Kenny F.

unread,

Jul 31, 2012, 10:03:20 AM7/31/12

to hyperta...@googlegroups.com

RangeServer crashes often: in 2-4 hours

actions after crash:
>hypertable/0.9.6.0/bin/ht stop-servers
Killing ThriftBroker.pid 17175
/opt/hypertable/0.9.6.0/bin/ht-env.sh: line 67: kill: (17175) - No such process
Shutdown master complete
Sending shutdown command
Unable to establish connection to range server
...
sometimes: Waiting for range server to shutdown...
Waiting for range server to shutdown...
Waiting for range server to shutdown...
Waiting for range server to shutdown...
Waiting for range server to shutdown...
Waiting for range server to shutdown...

when I try to restart severs:
>/hypertable/0.9.6.0/bin/ht start all-servers local
...
Started Hypertable.RangeServer
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
ERROR: ThriftBroker did not come up

Master Logs:
1343746823 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343746823 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343746824 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:218) Dropping OperationCollectGarbage because another one is outstanding
1343746824 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-2372 state=INITIAL
1343746824 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343746824 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
sh: dot: not found
1343746824 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
1343746824 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-2372
1343746824 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343746824 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343746825 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343746825 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343746826 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343746826 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343746827 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343746827 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected

ThriftBroker Logs:
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343743796 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=111.1111.111.111:38111; Problem connecting to Root RangeServer,
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343743796 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
....
1343747503 ERROR ThriftBroker : TThreadedServer client died: write() send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 34830>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write() send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 34835>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write() send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 34712>Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 34720>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write() send(): Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write() send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 34750>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write() send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 34653>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write() send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 34788>Broken pipe
1343747503 ERROR ThriftBroker : TThreadedServer client died: write() send(): Broken pipe
1343747503 ERROR ThriftBroker : TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 34808>Broken pipe
...

Doug Judd

unread,

Jul 31, 2012, 10:29:30 AM7/31/12

to hyperta...@googlegroups.com

Is there anything in the Hypertable.RangeServer.log file for 111.1111.111.111 that would indicate why it disconnected? If you post all of your log files we can take a look.

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/bN3Xud3yvcoJ.
To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

--

Doug Judd

CEO, Hypertable Inc.

Kenny F.

unread,

Jul 31, 2012, 11:41:13 AM7/31/12

to hyperta...@googlegroups.com, do...@hypertable.com

Master Logs:
...
1343675397 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
ERROR: /opt/hypertable/0.9.6.0/run/monitoring/rangeservers/rs1_stats_v0.rrd: found extra data on update argument: 7.51:1.40
1343675397 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returne
1343675397 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-1410
1343675427 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-1412 state=INITIAL
1343675427 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-1413
1343675427 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-1413
sh: dot: not found
...
...... many times (typical, everything is working)
...
1343675457 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
1343675488 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-1414
1343675488 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:237) socket read(27, len=38) failure : Connection reset by peer
1343675488 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:194) Event: type=DISCONNECT from=111.111.111.11:52020
1343675488 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:218) Dropping OperationGatherStatistics because another one is outstanding
1343675518 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-1418 state=INITIAL
1343675518 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675518 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-1419
1343675518 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1343675518 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-1419
1343675518 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
sh: dot: not found
1343675518 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
1343675518 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-1418
1343675548 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-1420 state=INITIAL
1343675548 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675548 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675548 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-1421
1343675548 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1343675548 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-1421
sh: dot: not found
1343675548 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
1343675548 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-1420
1343675578 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-1422 state=INITIAL
1343675578 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675578 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-1423
1343675578 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1343675578 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-1423
1343675578 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
sh: dot: not found
1343675608 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
1343675608 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-1424
1343675638 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-1426 state=INITIAL
1343675638 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675638 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675638 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:38) Entering CollectGarbage-1427
1343675638 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675638 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675638 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-1428
1343675638 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1343675638 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-1428
sh: dot: not found
1343675638 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
1343675638 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-1426
1343675639 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675639 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675640 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675640 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675641 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675641 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675642 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675642 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675643 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675643 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675644 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675644 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675645 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675645 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
..........

ThriftBroker Logs, same time:
...
1343675488 ERROR ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:237) socket read(29, len=38) failure : Connection reset by peer
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756) FileUtils::writev(29, len=150) failed : Broken pipe
1343675488 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT from=111.111.111.111:38060; Problem connecting to Root RangeServer, will retry in 3000 milliseconds...
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem flushing send queue - COMM broken connection
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
...
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
...
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675488 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
...
1343675489 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675489 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675489 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675489 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675489 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675489 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675489 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675489 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1343675489 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1343675489 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected

RangeServer Logs (several seconds before):
...
1343675427 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3085) Entering get_statistics()
1343675427 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3330) Exiting get_statistics()
1343675428 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:84) Maintenance stats scans=(2581 483 607807 0.015195) updates=(284 2114 10693135 0.267328 283)
1343675428 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/root' with latest revision older than 1312045326
1343675428 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/metadata' with latest revision older than 131204
1343675428 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/system' with latest revision older than 13120524
1343675428 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/user' with latest revision older than 1343669380
1343675428 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=3058.45, RSS=2388.57, tracked=2352.42, computed=2352.40 limit=3244.80
1343675428 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=38.85% BlockIndex=0.06% BloomFilter=0.06% CellCache=59.01% ShadowCache=0.00% QueryCache=2.03%
1343675428 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3822) Memory Usage: 2466686111 bytes
1343675429 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 13.600000, 14.679829
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TimerHandler.cc:89) Scheduling urgent maintenance for 0 millis in the future
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:84) Maintenance stats scans=(2581 483 607807 0.015195) updates=(284 2114 10693135 0.267328 283)
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/root' with latest revision older than 1312045326
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/metadata' with latest revision older than 131204
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/system' with latest revision older than 13120524
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/user' with latest revision older than 1343669380
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=3061.45, RSS=2391.28, tracked=2355.87, computed=2355.95 limit=3244.80
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=38.79% BlockIndex=0.06% BloomFilter=0.06% CellCache=59.01% ShadowCache=0.00% QueryCache=2.03%
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenancePrioritizer.cc:170) Adding maintenance for range 2/7[music.. 2] because disk_total 276137113 exceeds split threshold
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3822) Memory Usage: 2470306820 bytes
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:110) Range reference for '/hypertable/servers/rs1/log/2/7/4-liqWJRj8yF25PQ-1343675437' is required
1343675437 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:498) Starting Major Compaction of 2/7[music.. 2](default)
1343675439 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 15.800000, 14.680427
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
[[EOF]]

- Doug

To post to this group, send email to hypertable-dev@googlegroups.com.
To unsubscribe from this group, send email to hypertable-dev+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Doug Judd

unread,

Jul 31, 2012, 1:07:49 PM7/31/12

to hyperta...@googlegroups.com

This message at the end of the RangeServer is the problem:

terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc

Would you be willing to run a debug version of Hypertable to help us chase down the error? I can work with you on this off-list.

- Doug

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/kqSPtH63IM0J.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 1, 2012, 3:45:37 AM8/1/12

to hyperta...@googlegroups.com, do...@hypertable.com

How to enable debug mode?

Kenny F.

unread,

Aug 1, 2012, 9:35:19 AM8/1/12

to hyperta...@googlegroups.com, do...@hypertable.com

by the way sometimes RangeServer dies with other message:

RangeServer Logs:
1343824903 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3085) Entering get_statistics()
1343824903 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3330) Exiting get_statistics()
1343824906 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 13.300000, 14.876933
1343824909 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:84) Maintenance stats scans=(2830 944 2251315 0.056283) updates=(324 1998 10817679 0.270442 323)
1343824909 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/root' with latest revision older than 1312045326
1343824909 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/metadata' with latest revision older than 131204
1343824909 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/system' with latest revision older than 13120524
1343824909 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:321) Purging log fragments from '/hypertable/servers/rs1/log/user' with latest revision older than 1343818250
1343824909 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=3068.23, RSS=2388.04, tracked=2353.47, computed=2353.79 limit=3244.80
1343824909 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=43.80% BlockIndex=0.13% BloomFilter=0.14% CellCache=53.90% ShadowCache=0.00% QueryCache=2.03%
1343824909 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3822) Memory Usage: 2467796384 bytes
terminate called recursively
[[EOF]]

Doug Judd

unread,

Aug 1, 2012, 1:05:13 PM8/1/12

to hyperta...@googlegroups.com

Hi Kenny,

You'll need to install a special debug version of Hypertable. I'll go ahead and build one and post the link. Are you running 64-bit?

- Doug

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/jFNfxzJwztEJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Doug Judd

unread,

Aug 2, 2012, 8:30:02 PM8/2/12

to hyperta...@googlegroups.com

Hi Kenny,

We've fixed a couple of bugs recently that may be the source of this problem. Can you trying it out by installing the following pre-release of 0.9.6.1 and re-running your test?

http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.d051768-linux-x86_64.tar.bz2

But before you do that, please verify that you have no virtual memory limitations in place:

ulimit -v

(or 'ulimit memorysize' in csh)

- Doug

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/jFNfxzJwztEJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F. (2)

unread,

Aug 4, 2012, 3:13:34 PM8/4/12

to hyperta...@googlegroups.com, do...@hypertable.com

Thank you for help!

I work on Debian, 32 bit.

P.S. ulimit -v is unlimited

Doug Judd wrote:
> Hi Kenny,
>
>
> We've fixed a couple of bugs recently that may be the source of this problem. Can you trying it out by installing the following pre-release of 0.9.6.1 and re-running your test?
>
>
>
> http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.d051768-linux-x86_64.tar.bz2
>
>
>

> --
>
> Doug Judd
> CEO, Hypertable Inc.

Doug Judd

unread,

Aug 4, 2012, 3:16:53 PM8/4/12

to Kenny F. (2), hyperta...@googlegroups.com

No problem. Thank you for helping us to isolate this problem. It's a very big help to the project. I will build you a 32-bit build on Monday and point you to it.

- Doug

Doug Judd

unread,

Aug 5, 2012, 10:12:10 AM8/5/12

to Kenny F. (2), hyperta...@googlegroups.com

Here are 32-bit versions of the 0.9.6.1 pre-release:

http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.a0551dd-linux-i386.deb

http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.a0551dd-linux-i386.tar.bz2

It's a normal release build. Give this package a try and if the problem still persists, please let us know. One other question, do you know if the error occurs on the same RangeServer every time?

- Doug

On Sat, Aug 4, 2012 at 12:13 PM, Kenny F. (2) <zink...@gmail.com> wrote:

Message has been deleted

Kenny F.

unread,

Aug 6, 2012, 6:51:32 AM8/6/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

Thank for the release!
I installed it.

Have a crash :(

>the error occurs on the same RangeServer every time?

sorry, dunno

RangeServer Logs:
...
1344251972 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/

Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2317990351 bytes
1344251981 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 11.000000, 16.383754
1344251984 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) Entering get_statistics()
1344251984 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) Exiting get_statistics()
1344251992 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(2811 1011 2972785 0.074320) updates=(317 1915 12652963 0.316324 312)
1344251992 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/root' with latest revision older than 1312045326
1344251992 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/metadata' with latest revision older than 131204
1344251992 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/system' with latest revision older than 13120524
1344251992 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/user' with latest revision older than 1343832886
1344251992 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=3065.34, RSS=2373.27, tracked=2218.39, computed=2218.56 limit=3244.80
1344251992 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=46.67% BlockIndex=0.19% BloomFilter=0.18% CellCache=50.81% ShadowCache=0.00% QueryCache=2.15%
1344251992 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2326151392 bytes
1344251993 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 11.100000, 16.375433
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively

terminate called after throwing an instance of 'std::bad_alloc'

[[EOF]]

ThriftBroker Logs just after crash:
...
1344252038 ERROR ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:237) socket read(29, len=38) failure : Connection reset by peer
1344252038 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756) FileUtils::writev(29, len=252) failed : Broken pipe
1344252039 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem flushing send queue - COMM broken connection
1344252039 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT from=178.162.111.111:38060; Problem connecting to Root RangeServer, will retry in 3000 milliseconds...
1344252039 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756) FileUtils::writev(29, len=252) failed : Bad file descriptor
1344252039 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem flushing send queue - COMM broken connection
1344252039 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344252039 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344252039 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344252039 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
...
(when I try to get data):
...
1344252654 ERROR ThriftBroker : TThreadedServer: Caught TException: pthread_create failed
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
src/central_freelist.cc:322] tcmalloc: allocation failed 16384
1344252654 ERROR ThriftBroker : dump_error_history (/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.h:144): Hypertable::Exception: Problem creating scanner for start row 'some_data' on METADATA[..??] - COMM not connected
        at int Hypertable::RangeLocator::find(const Hypertable::TableIdentifier*, const char*, Hypertable::RangeLocationInfo*, Hypertable::Timer&, bool) (/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.cc:352)
        at void Hypertable::RangeServerClsrc/central_freelist.cc:322] tcmalloc: allocation failed 16384
1344252654 ERROR ThriftBroker : dump_error_history (/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.h:144): Hypertable::Exception: Problem creating scanner for start row 'some_data' on METADATA[..??] - COMM not connected
        at int Hypertable::RangeLocator::find(const Hypertable::TableIdentifier*, const char*, Hypertable::RangeLocationInfo*, Hypertable::Timer&, bool) (/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.cc:352)
        at void Hypertable::RangeServesrc/central_freelist.cc:322] tcmalloc: allocation failed 16384
src/central_freelist.cc:322] tcmalloc: allocation failed 16384
1344252654 ERROR ThriftBroker : get_cells (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:909): virtual void Hypertable::ThriftBroker::ServerHandler::get_cells(Hypertable::ThriftBroker::ThriftCells&, Hypertable::ThriftGen::Namespace, const Hypertable::String&, const Hypertable::ThriftGen::ScanSpec&)namespace=38465 table=SomeTable scan_spec={ScanSpec: cells=[
{CellInterval: start_row='some_data' start_column='data' start_inclusive=1 end_row='some_data' src/central_freelist.cc:322] tcmalloc: allocation failed 8192
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
...
(in little time):
...
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344252654 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
src/central_freelist.cc:322] tcmalloc: allocation failed 16384
src/central_freelist.cc:322] tcmalloc: allocation failed 8192

terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
[[EOF]]

Master Logs (before and ater crash):
sh: dot: not found
1344251984 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
ERROR: /opt/hypertable/0.9.6.0/run/monitoring/rangeservers/rs1_stats_v0.rrd: found extra data on update argument: 7.50:1.44
1344251984 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returne
1344251984 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-450
1344252014 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-452 state=INITIAL
1344252014 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-453
1344252014 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-453
sh: dot: not found
1344252014 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") failed - No such file or directory
1344252038 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:237) socket read(27, len=38) failure : Connection reset by peer
1344252038 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-452
1344252038 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:195) Event: type=DISCONNECT from=178.162.111.111:38938
1344252044 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-455 state=INITIAL
1344252044 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344252044 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-456
1344252044 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1344252044 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-456
1344252044 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
sh: dot: not found
1344252044 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
1344252044 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-455
1344252074 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-457 state=INITIAL
1344252074 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344252074 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344252074 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-458
1344252074 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1344252074 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-458

sh: dot: not found
...

HyperSapce Logs (just after crach):
...
1344252038 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 3
1344252038 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session 3(Hypertable.RangeServer)
1344252039 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 3 name=Hypertable.RangeServer
1344252039 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 9
1344252039 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 10
1344252039 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1523) Persisting lock released notifications
1344252039 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1534) Finished persisting lock released notifications
1344252039 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 11
1344252657 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 4
1344252657 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session 4(ThriftBroker)
1344252658 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 4 name=ThriftBroker
1344252658 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 14
1344252658 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 15
[[EOF]]

DfsBroker Logs (before and just after crash):
...
1344251305 INFO localBroker : (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.cc:137) open( /hypertable/tables/2/7/default/FDyEnEg1nT03Mrcj/cs18 ) = 367 (local=59)
1344251305 INFO localBroker : (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close( /hypertable/tables/2/7/default/FDyEnEg1nT03Mrcj/cs17 , 41 )
1344251305 INFO localBroker : (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close( /hypertable/tables/2/7/default/FDyEnEg1nT03Mrcj/cs17 , 49 )
1344251519 INFO localBroker : (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close( /hypertable/servers/rs1/log/user/264 , 39 )
1344251519 INFO localBroker : (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.cc:200) create( /hypertable/servers/rs1/log/user/265 ) = 368 (local=39)
1344252038 INFO localBroker : (/root/src/hypertable/src/cc/DfsBroker/Lib/OpenFileMap.h:88) Removing handle 4 from open file map because of lost owning client connection
1344252038 INFO localBroker : (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close( /hypertable/servers/rs1/log/rsml/52 , 26 )
1344252038 INFO localBroker : (/root/src/hypertable/src/cc/DfsBroker/Lib/OpenFileMap.h:88) Removing handle 6 from open file map because of lost owning client connection
1344252038 INFO localBroker : (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close( /hypertable/servers/rs1/log/root/1 , 27 )
...

Doug Judd

unread,

Aug 6, 2012, 6:57:53 PM8/6/12

to hyperta...@googlegroups.com, Kenny F. (2)

Hi Kenny,

Bummer. If you're not running too many RangeServers, here's what you can do to help us pin this down. Install the following debug build of Hypertable:

http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.370edcf-linux-i386-debug.deb

http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.370edcf-linux-i386-debug.tar.bz2

Then start Hypertable. Once the debug version of Hypertable is up and running, do the following for each RangeServer in your system:

--------------

1. Log into the RangeServer machine

2. Make sure 'screen' is installed on the machine (apt-get install screen)

3. Run screen

4. Make sure gdb is installed (apt-get install gdb)

5. Determine the process ID of the RangeServer (second field of output below):

# ps auxww | grep RangeServer | grep -v grep | grep -v cronolog

root 13472 0.1 0.8 835144 15956 ? Sl 21:34 0:01 /opt/hypertable/0.9.6.0.370edcf/bin/Hypertable.RangeServer --pidfile /opt/hypertable/0.9.6.0.370edcf/run/Hypertable.RangeServer.pid --verbose

5. Start gdb on the RangeServer executable:

# /opt/hypertable/current/bin/ht gdb /opt/hypertable/current/bin/Hypertable.RangeServer

6. Set a breakpoint on the constructor for the std::bad_alloc class:

(gdb) break 'std::bad_alloc::bad_alloc()'

Breakpoint 1 at 0x86137f0: file /usr/include/c++/4.3/new, line 61. (2 locations)

7. Attach to the RangeServer and immediately continue:

(gdb) attach 13472

...

(gdb) cont

Continuing.

8. Detach from your screen session with the following keystrokes:

ctrl-a ctrl-d

--------------

Once you've done the above for every RangeServer in your system, then run your test. This should cause one of the gdb sessions to hit the std::bad_alloc breakpoint. You should be able to figure out the IP address of the RangeServer that failed by looking for errors in the Hypertable.Master.log file. Once you figure out what RangeServer threw the exception, do the following:

1. Log into the RangeServer machine of the RangeServer that failed

2. Attach to the previously created screen session with:

$ screen -r

3. This will bring you back to the gdb session and the (gdb) prompt. Then run the following commands and post the output:

(gdb) where

...

(gdb) thread apply all where

...

If you're ok with leaving your system in this state for a bit, then at this point, just detach from the screen session again with:

ctrl-a ctrl-d

Thank you. Any help you can give us here will be much appreciated.

- Doug

--

You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/lyg8hS4e0nIJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 7, 2012, 4:53:46 AM8/7/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

Hi Doug!
Thanks for detail instruction!

>If you're not running too many RangeServers

I just start with "start all-servers local" with default configuration.

As I understand,

# ps auxww | grep RangeServer | grep -v grep | grep -v cronolog

show all RangeServers.So I have one RangeServer.

I'll tell the results )

To post to this group, send email to hypertable-dev@googlegroups.com.
To unsubscribe from this group, send email to hypertable-dev+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Message has been deleted

Kenny F.

unread,

Aug 7, 2012, 8:34:18 AM8/7/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

here they are:

$ screen -r

[New Thread 0x940f8b70 (LWP 15093)]
[Thread 0x940f8b70 (LWP 15093) exited]

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xae141b70 (LWP 14426)]
0xffffe424 in __kernel_vsyscall ()

(gdb) where

#0 0xffffe424 in __kernel_vsyscall ()
#1 0xb7bb2781 in raise () from /lib/i686/cmov/libc.so.6
#2 0xb7bb5bb2 in abort () from /lib/i686/cmov/libc.so.6
#3 0xb7dbe959 in __gnu_cxx::__verbose_terminate_handler() () from /opt/hypertable/0.9.6.0.370edcf/lib/libstdc++.so.6
#4 0xb7dbc865 in ?? () from /opt/hypertable/0.9.6.0.370edcf/lib/libstdc++.so.6
#5 0xb7dbc8a2 in std::terminate() () from /opt/hypertable/0.9.6.0.370edcf/lib/libstdc++.so.6
#6 0xb7dbc9da in __cxa_throw () from /opt/hypertable/0.9.6.0.370edcf/lib/libstdc++.so.6
#7 0xb7ec1eb4 in cpp_alloc (size=1000004) at src/tcmalloc.cc:1381
#8 tc_newarray (size=1000004) at src/tcmalloc.cc:1560
#9 0x08697165 in Hypertable::DynamicBuffer::grow (this=0xae13fff4, new_size=1000004, nocopy=false) at /root/src/hypertable/src/cc/Common/DynamicBuffer.h:120
#10 0x08697279 in Hypertable::DynamicBuffer::reserve (this=0xae13fff4, len=1000004, nocopy=false) at /root/src/hypertable/src/cc/Common/DynamicBuffer.h:72
#11 0x087697b3 in Hypertable::FillScanBlock (scanner=..., dbuf=..., buffer_size=1000000) at /root/src/hypertable/src/cc/Hypertable/RangeServer/FillScanBlock.cc:104
#12 0x0865b721 in Hypertable::RangeServer::create_scanner (this=0x8d2cb00, cb=0xae1412a8, table=0xae14129c, range_spec=0xae141290, scan_spec=0xae141200, cache_key=0xae141278)
at /root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1371
#13 0x087cb312 in Hypertable::RequestHandlerCreateScanner::run (this=0x1c0e6740) at /root/src/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerCreateScanner.cc:59
#14 0x08626d1c in Hypertable::ApplicationQueue::Worker::operator() (this=0x8ceadf0) at /root/src/hypertable/src/cc/AsyncComm/ApplicationQueue.h:172
#15 0x08626d78 in boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run (this=0x8cead20) at /usr/local/include/boost/thread/detail/thread.hpp:61
#16 0xb7fa8e68 in thread_proxy () from /opt/hypertable/0.9.6.0.370edcf/lib/libboost_thread.so.1.44.0
#17 0xb7eee955 in start_thread () from /lib/i686/cmov/libpthread.so.0
#18 0xb7c545ee in clone () from /lib/i686/cmov/libc.so.6

(gdb) thread apply all where

... (Attached in the file all 72 threads)

RangeServer Logs:
1344344059 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=51.97% BlockIndex=0.22% BloomFilter=0.24% CellCache=45.
1344344059 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2307694309 bytes
1344344060 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) Entering get_statistics()
1344344060 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) Exiting get_statistics()
1344344066 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 18.100000, 12.989667

terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called after throwing an instance of 'std::bad_alloc'
terminate called recursively
terminate called recursively

[[EOF]]

Also fall down ThriftBroker, after RangeServer crash, Logs:
...
1344344671 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:385): Received error: is_create=1 - Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:38060 - HYPERTABLE request timeout
1344344671 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:385): Received error: is_create=1 - Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:38060 - HYPERTABLE request timeout
1344344671 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:402): Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:38060 - HYPERTABLE request timeout
1344344671 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:402): Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:38060 - HYPERTABLE request timeout
1344344671 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:385): Received error: is_create=1 - Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:38060 - HYPERTABLE request timeout
1344344671 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:402): Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:38060 - HYPERTABLE request timeout
1344344671 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:385): Received error: is_create=1 - Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:38060 - HYPERTABLE request timeout
1344344671 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:402): Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:38060 - HYPERTABLE request timeout
1344344671 ERROR ThriftBroker : get_cells (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:909): virtual void Hypertable::ThriftBroker::ServerHandler::get_cells(Hypertable::ThriftBroker::ThriftCells&, Hypertable::ThriftGen::Namespace, const Hypertable::String&, const Hypertable::ThriftGen::ScanSpec&)namespace=37702 table=SomeTable scan_spec={ScanSpec:
some_spec}: Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout
at bool Hypertable::TableScanner::next(Hypertable::Cell&) (/root/src/hypertable/src/cc/Hypertable/Lib/TableScanner.cc:82)
....
a lot of same errors...
...
src/central_freelist.cc:322] tcmalloc: allocation failed 24576
1344345148 ERROR ThriftBroker : TThreadedServer exception: St9bad_alloc: std::bad_alloc
1344345149 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:385): Received error: is_create=1 - Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:38060 - HYPERTABLE request timeout
1344345149 ERROR ThriftBroker : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:402): Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.11138060: - HYPERTABLE request timeout
1344345149 ERROR ThriftBroker : get_cells (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:909): virtual void Hypertable::ThriftBroker::ServerHandler::get_cells(Hypertable::ThriftBroker::ThriftCellss&, Hypertable::ThriftGen::Namespace, const Hypertable::String&, const Hypertable::ThriftGen::ScanSpec&)namespace=39195 table=Searches scan_spec={some_spec}: Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:38060 - HYPERTABLE request timeout
at bool Hypertable::TableScanner::next(Hypertable::Cell&) (/root/src/hypertable/src/cc/Hypertable/Lib/TableScanner.cc:82)
...
several same errors...
...

src/central_freelist.cc:322] tcmalloc: allocation failed 8192

1344345149 FATAL ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/IntervalScannerAsync.cc:198) failed expectation: !has_outstanding_requests()
[[EOF]]

Master Logs:
...
sh: dot: not found
1344343490 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.370edcf/run/monitoring
ERROR: /opt/hypertable/0.9.6.0.370edcf/run/monitoring/rangeservers/rs1_stats_v0.rrd: found extra data on update argument: 8.11:1.47
1344343490 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returne
1344343490 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-624
1344343520 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-626 state=INITIAL
1344343520 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:38) Entering CollectGarbage-627
1344343520 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-628
1344343520 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-628
sh: dot: not found
1344343520 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.370edcf/run/monitoring
1344343520 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:47) Leaving CollectGarbage-627
ERROR: /opt/hypertable/0.9.6.0.370edcf/run/monitoring/rangeservers/rs1_stats_v0.rrd: found extra data on update argument: 7.18:1.24
1344343520 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returne
1344343520 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-626
1344343550 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-629 state=INITIAL
1344343550 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-630
1344343550 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasicDistributeTableRanges.cc:33) Distributing table ranges
1344343550 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-630
sh: dot: not found
1344343550 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.370edcf/run/monitoring
ERROR: /opt/hypertable/0.9.6.0.370edcf/run/monitoring/rangeservers/rs1_stats_v0.rrd: found extra data on update argument: 6.81:1.29
1344343550 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returne
1344343550 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-629
1344343580 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-631 state=INITIAL
1344343580 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-632
1344343580 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:119) Found 1 new/unbalanced servers, total ranges =37, total rangeservers=1
1344343580 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-632

sh: dot: not found
...

same logs...
...
sh: dot: not found
1344344090 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.370edcf/run/monitoring
1344344120 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-666
1344344120 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344150 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-669 state=INITIAL
1344344150 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:38) Entering CollectGarbage-670
1344344150 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-671
1344344150 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1344344150 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-671
sh: dot: not found
1344344150 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.370edcf/run/monitoring
1344344180 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344180 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-669
1344344210 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-673 state=INITIAL
1344344210 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-674
1344344210 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1344344210 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-674
sh: dot: not found
1344344210 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.370edcf/run/monitoring
1344344240 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344270 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344300 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344330 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344360 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344390 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344420 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344450 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344480 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344510 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344540 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344570 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344600 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344630 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344660 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344690 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344720 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344750 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-673
1344344750 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationCollectGarbage because another one is outstanding
1344344750 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-692 state=INITIAL
sh: dot: not found
1344344750 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.370edcf/run/monitoring
1344344750 ERROR Hypertable.Master : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:385): Received error: is_create=1 - Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:53530 - HYPERTABLE request timeout
1344344750 ERROR Hypertable.Master : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:402): Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:53530 - HYPERTABLE request timeout
1344344750 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/GcWorker.cc:50) Error: caught exception while gc'ing: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:53530 - HYPERTABLE request timeout
1344344750 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:47) Leaving CollectGarbage-670
1344344780 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344780 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-692
1344344810 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-695 state=INITIAL
1344344810 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:38) Entering CollectGarbage-696
1344344810 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-697
1344344810 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1344344810 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-697
sh: dot: not found
1344344810 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.370edcf/run/monitoring
1344344840 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-695
1344344840 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-698 state=INITIAL
1344344840 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-699
1344344840 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1344344840 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-699
sh: dot: not found
1344344840 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.370edcf/run/monitoring
1344344870 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344900 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344930 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344960 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344344990 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345020 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345050 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345080 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345110 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345140 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345154 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:195) Event: type=DISCONNECT from=178.162.111.111:49944
1344345200 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345230 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345260 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345290 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345320 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345350 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345380 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:219) Dropping OperationGatherStatistics because another one is outstanding
1344345410 ERROR Hypertable.Master : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:385): Received error: is_create=1 - Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:53530 - HYPERTABLE request timeout
1344345410 ERROR Hypertable.Master : handle_error (/root/src/hypertable/src/cc/Hypertable/Lib/TableScannerAsync.cc:402): Hypertable::Exception: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:53530 - HYPERTABLE request timeout
1344345410 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/GcWorker.cc:50) Error: caught exception while gc'ing: Event: type=ERROR "HYPERTABLE request timeout" from=178.162.111.111:53530 - HYPERTABLE request timeout
1344345410 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:47) Leaving CollectGarbage-696
1344345410 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-698
1344345410 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:38) Entering CollectGarbage-719
1344345410 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-718 state=INITIAL
1344345410 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-720
1344345410 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1344345410 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-720

sh: dot: not found
...

HyperSpace Logs:
...
1344345059 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 3 name=Hypertable.RangeServer
1344345059 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 9
1344345059 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 10
1344345059 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1523) Persisting lock released notifications
1344345059 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1534) Finished persisting lock released notifications
1344345059 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 11
1344345154 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 4
1344345154 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session 4(ThriftBroker)
1344345155 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 4 name=ThriftBroker
1344345155 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 14
1344345155 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 15
[[EOF]]

debug_errs_h_0.9.6.0.txt

Doug Judd

unread,

Aug 7, 2012, 2:57:52 PM8/7/12

to hyperta...@googlegroups.com, Kenny F. (2)

Great! We're getting close. One thing I'd like to know is what the virtual memory size and resident set size of the RangeServer process was immediately before the failure. If you haven't blown away your database yet, you should be able to get the monitoring system up and running and inspect the stats page for rs1. To do that, do the following:

# install ruby and rubygems

apt-get install ruby rubygems

# make sure you have ruby >= 1.8.7

$ ruby -version

ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

# install gems required for monitoring system:

gem install capistrano sinatra rack thin json titleize

# start monitoring system

/opt/hypertable/current/bin/start-monitoring.sh

# pull up the monitoring system in your web brower:

http://178.162.111.111:38090

You should see a table with one row in it. The first column is the "Server" column and should have a link "rs1" in it. Click on that link and it should pull up a page of graphs for that server. Set the "Start Time" and "End Time" to cover the period leading up to and including the time of the failure. The click the "show" button. Now scroll down to the graphs "Virtual Memory Size" and "Resident Memory Size" and screen capture those two graphs and post the screen capture to this list so that we can take a look.

If you have trouble getting the monitoring graphs up, let us know and we can help you get it up and running.

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/LlYYrNU0mVoJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 8, 2012, 4:26:53 AM8/8/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

Done.
I also installed 'thin' for monitoring.
I still have no 2 graphs between "Load average" and "Outstanding Scanner Count" at monitoring, but I have "Virtual Memory Size".

Message has been deleted

Kenny F.

unread,

Aug 8, 2012, 5:26:20 AM8/8/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

I have a little problem.
I have no data-points(lines) at graphs for RangeServer ((
I have a legend: 178-162-111-111.local
and title: RRD Graphs for rs1 (178-162-111-111.local)
and empty graphs...

At the "Tables", I have graphs with data.
But have no there a "Virtual Memory"

Doug Judd

unread,

Aug 8, 2012, 9:37:00 AM8/8/12

to hyperta...@googlegroups.com, Kenny F. (2)

Do you have 'rrdtool' installed and was 'rrdtool' installed when you ran your test? Or did you just install it to get the monitoring system working? If it wasn't installed when you ran your test, then it wouldn't have captured any statistics, so you'll need to run the test again with rrdtool installed.

If rrdtool was installed when you ran the test, then maybe it's a time range issue. Did you set the time range to the period immediately prior to the crash? Try setting the start date to August 2nd and the end date to August 8th and remember to hit the "show" button. That should definitely include the period of time when you ran the test and will give you an idea of the exact time period that represents the end of the test.

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/ACEvwLiYJKMJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 8, 2012, 9:47:10 AM8/8/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

When i use
# apt-get install rrdtool
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.

RRDtool 1.4.7

It is not a time-range issue, I checked ).

Doug Judd

unread,

Aug 8, 2012, 9:50:27 AM8/8/12

to hyperta...@googlegroups.com, Kenny F. (2)

Ok, can you take a look at the Master log file to see if there are any error messages about rrdtool?

grep -i rrdtool /opt/hypertable/current/log/Hypertable.Master.log

- Doug

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/M9jAxZ608CQJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 8, 2012, 9:59:37 AM8/8/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

Yes, I have warnings:

1344440844 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returned status 256)
1344440874 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returned status 256)
1344440904 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returned status 256)
1344440934 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returned status 256)
1344440964 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returned status 256)
1344440994 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returned status 256)
1344441024 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returned status 256)
1344441054 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returned status 256)
1344441084 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returned status 256)

Kenny F.

unread,

Aug 8, 2012, 10:17:12 AM8/8/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

warning:

`rrdtool`; make sure it's properly installed and in your $PATH

How do I need to configure rrdtool?

Doug Judd

unread,

Aug 8, 2012, 11:48:23 AM8/8/12

to hyperta...@googlegroups.com, Kenny F. (2)

First make sure it is in your path. Just type 'rrdtool'. If it is not, add the directory containing rrdtool to your PATH environment variable in .bashrc (or equivalent startup script for other shells). If it is in your path, then take a deeper look at the Hypertable.Master.log file and see if it is printing anymore information about what might be causing rrdtool to fail (e.g. dynamic link issue).

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/3ihe-kEL0N0J.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 9, 2012, 3:31:38 AM8/9/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

Rrdtool is in my path.

I posted earlier Master Logs, there are some errors :

...
sh: dot: not found

1344420922 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.jpg") failed - No such file or directory
ERROR: /opt/hypertable/0.9.6.0.370edcf/run/monitoring/rangeservers/rs1_stats_v0.rrd: found extra data on update argument: 9.04:1.43
1344420922 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returned status 256)

Kenny F.

unread,

Aug 9, 2012, 4:01:37 AM8/9/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

1344428210 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-180
1344428210 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-180
sh: dot: not found
1344428210 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.370edcf/run/monitoring/mop.jpg") failed - No such file or directory
ERROR: /opt/hypertable/0.9.6.0.370edcf/run/monitoring/rangeservers/rs1_stats_v0.rrd: found extra data on update argument: 8.36:1.43
1344428210 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor: failed to invoke `rrdtool`; make sure it's properly installed and in your $PATH (command returned status 256)
1344428210 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-179
1344428240 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-181 state=INITIAL
1344428240 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:38) Entering CollectGarbage-182
1344428240 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-183
1344428240 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-183
sh: dot: not found

Kenny F.

unread,

Aug 9, 2012, 7:17:37 AM8/9/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

I really don't understand why rrdtool don't grab results.

at MonitoringServer.log there are a lot of GET:
222.222.222.222 - - [08/Aug/2012 10:10:51] "GET /graph/RangeServer/rs1/disk_used_pct/1344410220/1344413820 HTTP/1.1" 200 13417 0.0664
rrdtool graph - --imgformat PNG --slope-mode --interlaced --end 1344413820 --start 1344410220 --width 900 --height 260 --title 'Virtual Memory Size' --color BACK#ffffff00 --color CANVAS#ffffff00 --color SHA..

when I retype in console # rrdtool graph - --imgformat PNG ...
it gave me no errors.
At browser, at "Monitoring" page I see graphs (axes, legend, grid), but they are empty, with no data.
But this is on "RangeServer" page. On "Table" page I have normal graphs.

Christoph Rupp

unread,

Aug 9, 2012, 7:19:02 AM8/9/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

Hi Kenny,

the problem is caused by rrdtool; it does not store data in the rrd files because of this (from the Hypertable.Master.log):

ERROR: /opt/hypertable/0.9.6.0.370edcf/run/monitoring/rangeservers/rs1_stats_v0.rrd: found extra data on update argument: 8.36:1.43

I did not follow the whole discussion, but there seems to be an incompatibility between your rrdtool version and ours.

This is my version:
RRDtool 1.4.7 Copyright 1997-2012 by Tobias Oetiker <to...@oetiker.ch>
Compiled Mar 29 2012 19:18:32

If you have a different build then maybe you can download rrdtool and compile an up-to-date version?

bye
Christoph

2012/8/9 Kenny F. <kfu...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/E9LWMkJZkMIJ.

Kenny F.

unread,

Aug 9, 2012, 7:30:05 AM8/9/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com, ch...@hypertable.com

I have the same:

Compiled Jul 6 2012 12:30:14
...

Kenny F.

unread,

Aug 9, 2012, 8:37:12 AM8/9/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

Ok, issue with rrdtool is solved! ))

I just delete /opt/hypertable/current/run/monitoring/rangeserversrs1_stats_v0.rrd
May be it was corrupted.
So, I'll test and post graphs as soon as possible.

Kenny F.

Christoph Rupp

unread,

Aug 9, 2012, 8:39:09 AM8/9/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

great :)

2012/8/9 Kenny F. <kfu...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/7iEzIkhQrjsJ.

Kenny F.

unread,

Aug 9, 2012, 10:42:16 AM8/9/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

I add "Virtual Memory Size" and "Resident Memory Size" graphs to attached file.
Crash is on the right-end of data, sure.

Kenny F.

hyp_g_3.jpg

Doug Judd

unread,

Aug 9, 2012, 1:58:09 PM8/9/12

to hyperta...@googlegroups.com, Kenny F. (2)

Ok, it doesn't look to me like you've run out of memory. There is one last thing that we can try. The last version I sent you was linked against tcmalloc which is less agressive about detecting memory errors than glibc malloc. I'll go ahead and build you a version that is linked against glib. If you could run your test with this new version and post the logs (at least the last hundred lines of the Hypertable.RangeServer.log), that might give us more clues about what's going on.

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/EGDFaREGYS4J.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Doug Judd

unread,

Aug 9, 2012, 5:30:09 PM8/9/12

to hyperta...@googlegroups.com, Kenny F. (2)

Ok, here are debug packages linked against glibc:

http://www.hypertable.com/debug/hypertable-0.9.6.0.95b0abc-linux-i386-debug.tar.bz2

http://www.hypertable.com/debug/hypertable-0.9.6.0.95b0abc-linux-i386-debug.deb

Please give it another try and report back and let us know if the RangeServer crashed and if so, whether or not it generated any more useful error messages. Thanks for your help!

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/EGDFaREGYS4J.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 10, 2012, 4:40:24 AM8/10/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

First results:

# /opt/hypertable/0.9.6.0.95b0abc/bin/ht start all-servers local

DFS broker: available file descriptors: 65536
Started DFS Broker (local)
Started Hyperspace
Hypertable.Master appears to be running (12416):
root 12416 12415 0 Aug09 ? 00:00:04 /opt/hypertable/0.9.6.0.370edcf/bin/Hypertable.Master --pidfile /opt/hypertable/0.9.6.0.370edcf/run/Hypertable.Master.pid --verbose
/proc/sys/vm/swappiness = 60
Started Hypertable.RangeServer
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
Waiting for ThriftBroker to come up...
ERROR: ThriftBroker did not come up
ThriftBroker appears to be running (3323):
root 3323 3321 0 10:29 pts/2 00:00:00 /opt/hypertable/0.9.6.0.95b0abc/bin/ThriftBroker --pidfile /opt/hypertable/0.9.6.0.95b0abc/run/ThriftBroker.pid --verbose

# /opt/hypertable/0.9.6.0.95b0abc/bin/ht stop-servers

Killing ThriftBroker.pid 3323
Unable to establish connection to range server
Shutdown master complete
Sending shutdown command
Shutdown range server complete
Sending shutdown command to DFS broker
Killing DfsBroker.local.pid 3096
Killing Hyperspace.pid 3065
Shutdown thrift broker complete
Shutdown hyperspace complete
Shutdown hypertable master complete
Shutdown DFS broker complete

Message has been deleted

Kenny F.

unread,

Aug 10, 2012, 5:27:37 AM8/10/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

That's all, it is faulty ((
Hypertable.Master can't start (((

When I start all-servers (or ./start-master.sh), it is written:

Hypertable.Master appears to be running (12416):
root 12416 12415 0 Aug09 ? 00:00:04 /opt/hypertable/0.9.6.0.

370edcf/bin/Hypertable.Master --pidfile /opt/hypertable/0.9.6.0.370edcf/run/Hypertable.Master.pid --verbose
but,
Hypertable.Master.log isn't created,

When I run shell:
# /opt/hypertable/0.9.6.0.95b0abc/bin/ht shell

1344595501 INFO hypertable : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050; Problem connecting to Master, will retry in 10000 milliseconds...

HyperSpace Logs:
1344595233 NOTICE Hyperspace.Master : (/root/src/hypertable/src/cc/Common/Config.cc:540) Initializing Hyperspace.Master (Hypertable 0.9.6.0.95b0abc (v0.9.6.0-21-g95b0abc))...
CPU cores count=8
CephBroker.MonAddr=10.0.1.1:6789
DfsBroker.Local.Root=fs/local
DfsBroker.Port=38030
HdfsBroker.Hadoop.ConfDir=/etc/hadoop/conf
Hyperspace.GracePeriod=200000
Hyperspace.KeepAlive.Interval=30000
Hyperspace.Lease.Interval=1000000
Hyperspace.Replica.Dir=hyperspace
Hyperspace.Replica.Host=[localhost]
Hyperspace.Replica.Port=38040
Hyperspace.Replica.Reactors=8
Hypertable.Master.Port=38050
Hypertable.RangeServer.Port=38060
Hypertable.Verbose=true
ThriftBroker.Port=38080
dir=hyperspace
keepalive=30000
lease-interval=1000000
pidfile=/opt/hypertable/0.9.6.0.95b0abc/run/Hyperspace.pid
port=38040
reactors=8
verbose=true
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:152) BerkeleyDB base directory = '/opt/hypertable/0.9.6.0.95b0abc/hyperspace'
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:110) localhost=178-162-111-111.local localip=178.162.111.111
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:124) Removing statedb
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:334) Replication master init done
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x830fe88
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8310048
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8310210
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x83103f8
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x83105a0
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x83107b8
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8310968
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8310b28
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8310ce8
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8310f20
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x83110d0
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8311290
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8311450
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8311610
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x83117d0
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8311990
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8311b50
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8311e08
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8311fc8
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x8312188
1344595233 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/BerkeleyDbFilesystem.cc:508) Created DB handles for thread: 0x83123f0
1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:264) Create session for 127.0.0.1:48052
1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:276) created session 1
1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerRenewSession.cc:74) Session handle 1 created
1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:95) Event: type=CONNECTION_ESTABLISHED from=178.162.111.111:43751
1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:338) Initialized session 1 (serverup)
1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 1
1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session 1(serverup)
1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 1 name=serverup
1344595238 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:264) Create session for 127.0.0.1:36204
1344595238 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:276) created session 2
1344595238 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerRenewSession.cc:74) Session handle 2 created
1344595238 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:95) Event: type=CONNECTION_ESTABLISHED from=178.162 .111.111 :55820
1344595238 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:338) Initialized session 2 (Hypertable.RangeServer)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2263) exists(session_id=2, name=/hypertable/namemap/names)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2269) exitting exists(session_id=2, name=/hypertable/namemap/names)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2263) exists(session_id=2, name=/hypertable/namemap/ids)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2269) exitting exists(session_id=2, name=/hypertable/namemap/ids)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1915) open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/master, flags=0x1, event_mask=0x1)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2044) handle 1 created ('/hypertable/master', session=2(Hypertable.RangeServer), flags=0x1, mask=0x1)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:698) exitting open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/master, flags=0x1, event_mask=0
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2368) attrget(session=2(Hypertable.RangeServer), handle=1, attr=address)
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:264) Create session for 127.0.0.1:49718
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:276) created session 3
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerRenewSession.cc:74) Session handle 3 created
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:95) Event: type=CONNECTION_ESTABLISHED from=178.162 .111.111 :34011
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:338) Initialized session 3 (ThriftBroker)
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2263) exists(session_id=3, name=/hypertable/namemap/names)
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2269) exitting exists(session_id=3, name=/hypertable/namemap/names)
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2263) exists(session_id=3, name=/hypertable/namemap/ids)
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2269) exitting exists(session_id=3, name=/hypertable/namemap/ids)
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1915) open(session_id=3, session_name = ThriftBroker, fname=/hypertable/master, flags=0x1, event_mask=0x1)
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2044) handle 2 created ('/hypertable/master', session=3(ThriftBroker), flags=0x1, mask=0x1)
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:698) exitting open(session_id=3, session_name = ThriftBroker, fname=/hypertable/master, flags=0x1, event_mask=0x1)
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2368) attrget(session=3(ThriftBroker), handle=2, attr=address)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2263) exists(session_id=2, name=/hypertable/servers)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2269) exitting exists(session_id=2, name=/hypertable/servers)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1915) open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/servers/rs1, flags=0xf, event_mask=0x0)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2044) handle 3 created ('/hypertable/servers/rs1', session=2(Hypertable.RangeServer), flags=0xf, mask=0x0)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:698) exitting open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/servers/rs1, flags=0xf, event_m
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1227) lock(session=2(Hypertable.RangeServer), handle=3, mode=0x2, try_lock=1)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1336) lock txn={BDbTxn m_handle_namespace_db=0x832e220, m_handle_state_db=0x832e630, m_db_txn=0x8335498} commited handle
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readdirattr(session=2(Hypertable.RangeServer), name=/hypertable/tables, attr=schema)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1915) open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/root, flags=0x1, event_mask=0x1)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2044) handle 4 created ('/hypertable/root', session=2(Hypertable.RangeServer), flags=0x1, mask=0x1)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:698) exitting open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/root, flags=0x1, event_mask=0x1
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1915) open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/tables/0/0, flags=0x1, event_mask=0x0)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2044) handle 5 created ('/hypertable/tables/0/0', session=2(Hypertable.RangeServer), flags=0x1, mask=0x0)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:698) exitting open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/tables/0/0, flags=0x1, event_ma
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2368) attrget(session=2(Hypertable.RangeServer), handle=5, attr=schema)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:750) close(session=2(Hypertable.RangeServer), handle=5)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=2(Hypertable.RangeServer), name=/hypertable/namemap/names/sys/METADATA, attr=id)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) attrget(session=2(Hypertable.RangeServer), name=/hypertable/tables/0/0, attr=schema)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1915) open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/root, flags=0x1, event_mask=0x0)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2044) handle 6 created ('/hypertable/root', session=2(Hypertable.RangeServer), flags=0x1, mask=0x0)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:698) exitting open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/root, flags=0x1, event_mask=0x0
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2371) attrget(session=2(Hypertable.RangeServer), handle=6)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2371) attrget(session=2(Hypertable.RangeServer), handle=6)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2371) attrget(session=2(Hypertable.RangeServer), handle=6)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:750) close(session=2(Hypertable.RangeServer), handle=6)
1344595249 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2368) attrget(session=2(Hypertable.RangeServer), handle=4, attr=Location)
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:264) Create session for 127.0.0.1:37479
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:276) created session 4
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerRenewSession.cc:74) Session handle 4 created
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:95) Event: type=CONNECTION_ESTABLISHED from=178.162.111.111:60052
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:338) Initialized session 4 (hypertable)
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2263) exists(session_id=4, name=/hypertable/namemap/names)
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2269) exitting exists(session_id=4, name=/hypertable/namemap/names)
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2263) exists(session_id=4, name=/hypertable/namemap/ids)
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2269) exitting exists(session_id=4, name=/hypertable/namemap/ids)
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1915) open(session_id=4, session_name = hypertable, fname=/hypertable/master, flags=0x1, event_mask=0x1)
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2044) handle 7 created ('/hypertable/master', session=4(hypertable), flags=0x1, mask=0x1)
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:698) exitting open(session_id=4, session_name = hypertable, fname=/hypertable/master, flags=0x1, event_mask=0x1)
1344595501 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2368) attrget(session=4(hypertable), handle=7, attr=address)
1344595516 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 4
1344595516 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session 4(hypertable)
1344595517 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 4 name=hypertable
1344595517 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 7

ThriftBroker Logs:
1344594474 NOTICE ThriftBroker : (/root/src/hypertable/src/cc/Common/Config.cc:540) Initializing ThriftBroker (Hypertable 0.9.6.0.95b0abc (v0.9.6.0-21-g95b0abc))...
CPU cores count=8
CephBroker.MonAddr=10.0.1.1:6789
DfsBroker.Local.Root=fs/local
DfsBroker.Port=38030
HdfsBroker.Hadoop.ConfDir=/etc/hadoop/conf
Hyperspace.GracePeriod=200000
Hyperspace.KeepAlive.Interval=30000
Hyperspace.Lease.Interval=1000000
Hyperspace.Replica.Dir=hyperspace
Hyperspace.Replica.Host=[localhost]
Hyperspace.Replica.Port=38040
Hypertable.Master.Port=38050
Hypertable.RangeServer.Port=38060
Hypertable.Verbose=true
ThriftBroker.Port=38080
pidfile=/opt/hypertable/0.9.6.0.95b0abc/run/ThriftBroker.pid
port=38080
reactors=8
verbose=true
1344594474 INFO ThriftBroker : (/root/src/hypertable/src/cc/Hyperspace/Session.cc:63) Hyperspace session setup to reconnect
1344594474 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050; Problem connecting to Master, will retry in 10000 milliseconds...
1344594484 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050; Problem connecting to Master, will retry in 10000 milliseconds...
1344594494 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050; Problem connecting to Master, will retry in 10000 milliseconds...
...............................................................
1344595843 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050; Problem connecting to Master, will retry in 10000 milliseconds...
1344595843 ERROR ThriftBroker : main (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2406): Hypertable::Exception: Waiting for Master connection - HYPERTABLE request timeout
at void Hypertable::Client::initialize() (/root/src/hypertable/src/cc/Hypertable/Lib/Client.cc:233)
[[EOF]]

RangeServers Logs:
1344594472 NOTICE Hypertable.RangeServer : (/root/src/hypertable/src/cc/Common/Config.cc:540) Initializing Hypertable.RangeServer (Hypertable 0.9.6.0.95b0abc (v0.9.6.0-21-g95b0abc))...
CPU cores count=8
CephBroker.MonAddr=10.0.1.1:6789
DfsBroker.Local.Root=fs/local
DfsBroker.Port=38030
HdfsBroker.Hadoop.ConfDir=/etc/hadoop/conf
Hyperspace.GracePeriod=200000
Hyperspace.KeepAlive.Interval=30000
Hyperspace.Lease.Interval=1000000
Hyperspace.Replica.Dir=hyperspace
Hyperspace.Replica.Host=[localhost]
Hyperspace.Replica.Port=38040
Hypertable.Master.Port=38050
Hypertable.RangeServer.Port=38060
Hypertable.RangeServer.Reactors=8
Hypertable.Verbose=true
ThriftBroker.Port=38080
dfs-port=38030
grace-period=200000
hs-host=[localhost]
hs-port=38040
keepalive=30000
lease-interval=1000000
master-port=38050
pidfile=/opt/hypertable/0.9.6.0.95b0abc/run/Hypertable.RangeServer.pid
port=38060
reactors=8
verbose=true
1344594472 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/HyperspaceSessionHandler.cc:31) Hyperspace session state change: SAFE
drive count = 2
maintenance threads = 8
1344594473 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050; Problem connecting to Master, will retry in 10000 milliseconds...
1344594473 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050
1344594473 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:95) Event: type=CONNECTION_ESTABLISHED from=178.162.111.111:56863
1344594473 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215) Event: type=DISCONNECT from=178.162.111.111:56863
1344594474 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:95) Event: type=CONNECTION_ESTABLISHED from=178.162.111.111:60107
1344594474 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215) Event: type=DISCONNECT from=178.162.111.111:60107
1344594483 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050; Problem connecting to Master, will retry in 10000 milliseconds...
1344594483 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050
1344594483 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:502) log_dir=/hypertable/servers/rs1/log
1344594483 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TableInfo.cc:252) Adding range 0/0[..0/0:..] to TableInfo
1344594483 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3675) Successfully replay loaded range 0/0[..0/0:..]
1344594483 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/root/0
1344594483 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/root/0
1344594483 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1007) Replayed 2 blocks of updates from '/hypertable/servers/rs1/log/root'
1344594483 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:113) Range reference for '/hypertable/servers/rs1/log/root' is NOT required
1344594483 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:218) Connection attempt to Root RangeServer at rs1 failed - COMM invalid proxy. Will retry again in 3000 milliseconds...
1344594488 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:218) Connection attempt to Root RangeServer at rs1 failed - COMM invalid proxy. Will retry again in 3000 milliseconds...
1344594492 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:218) Connection attempt to Root RangeServer at rs1 failed - COMM invalid proxy. Will retry again in 3000 milliseconds...
1344594493 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=178.162 .111.111 :38050; Problem connecting to Master, will retry in 10000 milliseconds...
1344594493 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050
1344594495 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:218) Connection attempt to Root RangeServer at rs1 failed - COMM invalid proxy. Will retry again in 3000 milliseconds...
1344594500 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:218) Connection attempt to Root RangeServer at rs1 failed - COMM invalid proxy. Will retry again in 3000 milliseconds...
1344594503 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=178.162 .111.111 :38050; Problem connecting to Master, will retry in 10000 milliseconds...
1344594503 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050
1344594504 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:218) Connection attempt to Root RangeServer at rs1 failed - COMM invalid proxy. Will retry again in 3000 milliseconds...
...

On Friday, August 10, 2012 12:30:09 AM UTC+3, Doug Judd wrote:

Ok, here are debug packages linked against glibc:

http://www.hypertable.com/debug/hypertable-0.9.6.0.95b0abc-linux-i386-debug.tar.bz2

http://www.hypertable.com/debug/hypertable-0.9.6.0.95b0abc-linux-i386-debug.deb

Please give it another try and report back and let us know if the RangeServer crashed and if so, whether or not it generated any more useful error messages. Thanks for your help!

- Doug

Kenny F.

unread,

Aug 10, 2012, 5:29:07 AM8/10/12

to hyperta...@googlegroups.com

Christoph Rupp

unread,

Aug 10, 2012, 5:30:59 AM8/10/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

Hi,

Can you kill the master? maybe it's hanging. I fixed a but yesterday where the master was hanging during shutdown.

first stop all servers (stop-servers.sh), then check if there are any lingering processes. Then kill them manually with the "kill" command. Then startup; if you want to have a clean setup, use "start-test-servers.sh --clear", otherwise "start-test-servers.sh" (this runs the servers on the local machine).

bye
Christoph

2012/8/10 Kenny F. <kfu...@gmail.com>

That's all, it is faulty ((
Hypertable.Master can't start (((

When I start all-servers (or ./start-master.sh), it is written:

Hypertable.Master appears to be running (12416):

root 12416 12415 0 Aug09 ? 00:00:04 /opt/hypertable/0.9.6.0.370edcf/bin/Hypertable.Master --pidfile /opt/hypertable/0.9.6.0.370edcf/run/Hypertable.Master.pid --verbose

1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:95) Event: type=CONNECTION_ESTABLISHED from=178.162.190.234:43751

1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:338) Initialized session 1 (serverup)
1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 1
1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session 1(serverup)
1344595236 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 1 name=serverup
1344595238 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:264) Create session for 127.0.0.1:36204
1344595238 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:276) created session 2
1344595238 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerRenewSession.cc:74) Session handle 2 created

1344595238 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:95) Event: type=CONNECTION_ESTABLISHED from=178.162.190.234:55820

1344595238 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:338) Initialized session 2 (Hypertable.RangeServer)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2263) exists(session_id=2, name=/hypertable/namemap/names)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2269) exitting exists(session_id=2, name=/hypertable/namemap/names)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2263) exists(session_id=2, name=/hypertable/namemap/ids)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2269) exitting exists(session_id=2, name=/hypertable/namemap/ids)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1915) open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/master, flags=0x1, event_mask=0x1)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2044) handle 1 created ('/hypertable/master', session=2(Hypertable.RangeServer), flags=0x1, mask=0x1)
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:698) exitting open(session_id=2, session_name = Hypertable.RangeServer, fname=/hypertable/master, flags=0x1, event_mask=0
1344595239 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2368) attrget(session=2(Hypertable.RangeServer), handle=1, attr=address)
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:264) Create session for 127.0.0.1:49718
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:276) created session 3
1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerRenewSession.cc:74) Session handle 3 created

1344595241 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:95) Event: type=CONNECTION_ESTABLISHED from=178.162.190.234:34011

1344594493 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=178.162.190.234:38050; Problem connecting to Master, will retry in 10000 milliseconds...

1344594493 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050
1344594495 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:218) Connection attempt to Root RangeServer at rs1 failed - COMM invalid proxy. Will retry again in 3000 milliseconds...
1344594500 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:218) Connection attempt to Root RangeServer at rs1 failed - COMM invalid proxy. Will retry again in 3000 milliseconds...

1344594503 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT "COMM connect error" from=178.162.190.234:38050; Problem connecting to Master, will retry in 10000 milliseconds...

1344594503 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215) Event: type=DISCONNECT "COMM connect error" from=178.162.111.111:38050
1344594504 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:218) Connection attempt to Root RangeServer at rs1 failed - COMM invalid proxy. Will retry again in 3000 milliseconds...
...

On Friday, August 10, 2012 12:30:09 AM UTC+3, Doug Judd wrote:

Ok, here are debug packages linked against glibc:

http://www.hypertable.com/debug/hypertable-0.9.6.0.95b0abc-linux-i386-debug.tar.bz2

http://www.hypertable.com/debug/hypertable-0.9.6.0.95b0abc-linux-i386-debug.deb

Please give it another try and report back and let us know if the RangeServer crashed and if so, whether or not it generated any more useful error messages. Thanks for your help!

- Doug

--
Doug Judd
CEO, Hypertable Inc.

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/F7k6Mu3veJoJ.

Kenny F.

unread,

Aug 10, 2012, 7:23:15 AM8/10/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com, ch...@hypertable.com

Hi Christoph,

Thank you, it helped.
Hypertable works strange a bit, I'll write a bit later.

Kenny F.

unread,

Aug 10, 2012, 10:07:30 AM8/10/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

here we are:

HyperSpace Logs:
...
1344602336 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 8
1344602336 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session 8(hypertable)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 8 name=hypertable
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 26
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 27
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:264) Create session for 127.0.0.1:38003
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:276) created session 9
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerRenewSession.cc:74) Session handle 9 created
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:95) Event: type=CONNECTION_ESTABLISHED from=178.162.111.111:51422
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:338) Initialized session 9 (hypertable)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2263) exists(session_id=9, name=/hypertable/namemap/names)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2269) exitting exists(session_id=9, name=/hypertable/namemap/names)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2263) exists(session_id=9, name=/hypertable/namemap/ids)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2269) exitting exists(session_id=9, name=/hypertable/namemap/ids)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1915) open(session_id=9, session_name = hypertable, fname=/hypertable/master, flags=0x1, event_mask=0x1)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2044) handle 29 created ('/hypertable/master', session=9(hypertable), flags=0x1, mask=0x1)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:698) exitting open(session_id=9, session_name = hypertable, fname=/hypertable/master, flags=0x1, event_mask=0x1)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2368) attrget(session=9(hypertable), handle=29, attr=address)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1915) open(session_id=9, session_name = hypertable, fname=/hypertable/root, flags=0x1, event_mask=0x1)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2044) handle 30 created ('/hypertable/root', session=9(hypertable), flags=0x1, mask=0x1)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:698) exitting open(session_id=9, session_name = hypertable, fname=/hypertable/root, flags=0x1, event_mask=0x1)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1915) open(session_id=9, session_name = hypertable, fname=/hypertable/tables/0/0, flags=0x1, event_mask=0x0)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2044) handle 31 created ('/hypertable/tables/0/0', session=9(hypertable), flags=0x1, mask=0x0)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:698) exitting open(session_id=9, session_name = hypertable, fname=/hypertable/tables/0/0, flags=0x1, event_mask=0x0)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2368) attrget(session=9(hypertable), handle=31, attr=schema)
1344602337 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:750) close(session=9(hypertable), handle=31)
1344602481 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=9(hypertable), name=/hypertable/namemap/names/some_name, attr=id)
1344602484 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 9
1344602484 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session 9(hypertable)
1344602485 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 9 name=hypertable
1344602485 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 29
1344602485 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 30
1344602486 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=4(ThriftBroker), name=/hypertable/namemap/names/some_name/some_table, attr=id)
1344602486 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) attrget(session=4(ThriftBroker), name=/hypertable/tables/2/8, attr=schema)
1344602503 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=2(Hypertable.Master), name=/hypertable/namemap/ids/2/4, attr=name)
1344602503 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=2(Hypertable.Master), name=/hypertable/namemap/ids/2/5, attr=name)
1344602503 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=2(Hypertable.Master), name=/hypertable/namemap/ids/2/6, attr=name)
1344602503 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=2(Hypertable.Master), name=/hypertable/namemap/ids/2/7, attr=name)
1344602503 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=2(Hypertable.Master), name=/hypertable/namemap/ids/2/8, attr=name)
1344602503 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=2(Hypertable.Master), name=/hypertable/namemap/ids/2/9, attr=name)
1344603673 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=3(Hypertable.RangeServer), name=/hypertable/namemap/names/sys/RS_METRICS, attr=id)
1344603673 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) attrget(session=3(Hypertable.RangeServer), name=/hypertable/tables/0/1, attr=schema)
1344604735 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=4(ThriftBroker), name=/hypertable/namemap/names/some_name/some_table2, attr=id)
1344604735 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) attrget(session=4(ThriftBroker), name=/hypertable/tables/2/9, attr=schema)
1344606764 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) readpathattr(session=4(ThriftBroker), name=/hypertable/namemap/names/some_name/some_table2, attr=id)
1344606764 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:2407) attrget(session=4(ThriftBroker), name=/hypertable/tables/2/5, attr=schema)
1344609930
1344609962 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 3
1344609962 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session 3(Hypertable.RangeServer)
1344609962 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 3 name=Hypertable.RangeServer
1344609962 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 9
1344609963 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 10
1344609963 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1523) Persisting lock released notifications
1344609963 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1534) Finished persisting lock released notifications
1344609963 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 11
1344610640 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 4
1344610640 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session 4(ThriftBroker)
1344610641 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 4 name=ThriftBroker
1344610641 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 14
1344610641 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 15
[[EOF]]

Master Logs:

...
sh: dot: not found

1344609883 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.95b0abc/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.95b0abc/run/monitoring
1344609883 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-578
1344609913 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-580 state=INITIAL
1344609913 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-581
1344609913 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-581
sh: dot: not found
1344609913 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.95b0abc/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.95b0abc/run/monitoring
1344609913 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-580
1344609943 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-582 state=INITIAL
1344609943 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-583
1344609943 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-583
sh: dot: not found
1344609943 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.95b0abc/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.95b0abc/run/monitoring
1344609962 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:237) socket read(27, len=38) failure : Connection reset by peer
1344609962 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/DispatchHandlerOperation.cc:111) Couldn't locate connection object for 178.162.111.111:35682
1344609962 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-582
1344609962 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:195) Event: type=DISCONNECT from=178.162.111.111:35682
1344609973 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-585 state=INITIAL
1344609973 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-586
1344609973 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344609973 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344609973 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-586
sh: dot: not found
1344609973 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.95b0abc/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.95b0abc/run/monitoring
1344609973 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-585
1344610003 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:38) Entering CollectGarbage-588
1344610003 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) Entering GatherStatistics-587 state=INITIAL
1344610003 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344610003 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344610003 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344610003 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344610003 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) Entering LoadBalancer-589
1344610003 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) Found non-live server rs1 wait till all servers are live before trying balance
1344610003 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) Leaving LoadBalancer-589
sh: dot: not found
1344610003 ERROR Hypertable.Master : (/root/src/hypertable/src/cc/Common/FileUtils.cc:451) rename("/opt/hypertable/0.9.6.0.95b0abc/run/monitoring/mop.tmp.jpg", "/opt/hypertable/0.9.6.0.95b0abc/run/monitoring
1344610003 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) Leaving GatherStatistics-587
1344610004 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344610004 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344610005 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344610005 WARN Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344610006 WARN Hypertable.Master : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
...

RangeServer Logs:
...
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/6/ZvNjprU1OyzcKPC3-1344340077/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/6/ZvNjprU1OyzcKPC3-1344340077/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/6/ZvNjprU1OyzcKPC3-1344340077/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/6/ZvNjprU1OyzcKPC3-1344340077/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/6/ZvNjprU1OyzcKPC3-1344340077/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/7/eIT0klUzUAfMOOxL-1344428336/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/7/CWzy5Rk4hS5LWW0W-1344428567/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/7/zYoeSP92oySV3PZ2-1344428671/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/7/zYoeSP92oySV3PZ2-1344428671/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/6/tv-ts2NdD5nHehtb-1344521131/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/6/tv-ts2NdD5nHehtb-1344521131/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/6/tv-ts2NdD5nHehtb-1344521131/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/6/tv-ts2NdD5nHehtb-1344521131/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLogReader.cc:130) Replaying commit log fragment /hypertable/servers/rs1/log/2/6/tv-ts2NdD5nHehtb-1344521131/0
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1007) Replayed 878049 blocks of updates from '/hypertable/servers/rs1/log/user'
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:111) Range reference for '/hypertable/servers/rs1/log/user' is required
1344602485 NOTICE Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:854) Replay finished
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:340) Prune thresholds - min=1000000000, max=4253024256
1344602485 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(51 168 11122 0.000019) updates=(0 0 0 0.000000 0)
1344602486 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0
1344602486 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0
1344602486 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0
1344602486 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/root' with latest revision older than 1312045326
1344602486 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/metadata' with latest revision older than 131204
1344602486 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/system' with latest revision older than 13120524
1344602486 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/user' with latest revision older than 1344513029
1344602486 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=1593.23, RSS=972.93, tracked=986.93, computed=1039.73 limit=4867.20
1344602486 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=0.06% BlockIndex=0.03% BloomFilter=0.40% CellCache=94.92% ShadowCache=0.00% QueryCache=4.59%
1344602486 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 1040421135 bytes
1344602503 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) Entering get_statistics()
1344602503 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) Successfully created scanner (id=0) on table '0/0', returning 0 k/v pairs, more=0
1344602503 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) Successfully created scanner (id=0) on table '0/0', returning 38 k/v pairs, more=0
1344602503 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) Exiting get_statistics()
...
1344608666 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 1801576768 bytes
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 17.300000, 13.588580
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:734) Finished Compaction of 2/7[some_key..some_key](default) to /hypertable/tables/2/7/default/yuAlHaOTjfGvXjUf/cs40
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TableInfo.cc:84) Changing end row 2/7 removing old row 'some_key 2' (start row 'some_key 1')
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TableInfo.cc:92) Changing end row 2/7 adding new row 'some_key2' (start row 'some_key1')
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:1194) Reporting newly split off range 2/7[some_key..some_key] to Master
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1570) Loading range: {TableIdentifier: id='2/7' generation=1} {RangeSpec: start='some_key0' end='some_key1'} {RangeState: state=STEADY timestamp=0 soft_limit=268435456 transfer_log='/hypertable/servers/rs1/log/2/7/a7pSwAjleWuKBIlk-13444608623' split_point='' old_boundary_row=''} needs_compaction=0
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TableInfo.cc:212) Staging range 2/7[some_key 2..some_key] to TableInfo
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:269) Loading CellStore 2/7/default/yuAlHaOTjfGvXjUf/cs40
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:810) Split Complete. New Range end_row=los tomatoes
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TableInfo.cc:237) Adding range 2/7[some_key..3] to TableInfo
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1780) Successfully loaded range 2/7[some_key..3]
1344608668 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1813) Acknowledging range: 2/7[some_key..3]
1344608670 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1436) Hypertable::Exception: (a) 2/7[some_key..4] - RANGE SERVER range not found
at void Hypertable::RangeServer::create_scanner(Hypertable::ResponseCallbackCreateScanner*, const Hypertable::TableIdentifier*, const Hypertable::RangeSpec*, const Hypertable::ScanSpec*, Hypertable::QueryCache::Key*) (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1319)
1344608670 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0
1344608670 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) Successfully created scanner (id=0) on table '0/0', returning 21 k/v pairs, more=0
1344608677 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 19.600000, 13.607077
1344608683 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) Entering get_statistics()
1344608683 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) Exiting get_statistics()
1344608685 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(3136 1334 1666618 0.038902) updates=(108 628 3420943 0.079852 12)
...
1344609186 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 1912457450 bytes
1344609186 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:520) Starting GC Compaction of 2/7[some_key..some_key](default)
1344609187 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 11.900000, 13.705962
1344609193 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) Entering get_statistics()
1344609193 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) Exiting get_statistics()
1344609194 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 19.400000, 13.721351
1344609199 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:734) Finished Compaction of 2/7[some_key..some_key](default) to /hypertable/tables/2/7/default/xgiJeBkzhNzw86Tk/cs12
1344609205 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 16.300000, 13.728302
1344609206 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(3019 1296 2830090 0.070752) updates=(351 1790 12537954 0.313449 341)
...
1344609386 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 1946100027 bytes
1344609391 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 14.500000, 13.723316
1344609403 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 10.600000, 13.715245
1344609403 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) Entering get_statistics()
1344609403 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) Successfully created scanner (id=0) on table '0/0', returning 0 k/v pairs, more=0
1344609403 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) Successfully created scanner (id=0) on table '0/0', returning 43 k/v pairs, more=0
1344609403 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) Exiting get_statistics()
1344609406 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(2614 1544 4025226 0.100626) updates=(302 1512 11274688 0.281853 300)
...
1344609826 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=2718.70, RSS=2003.48, tracked=1968.57, computed=1969.35 limit=4867.20
1344609826 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=49.79% BlockIndex=0.33% BloomFilter=0.35% CellCache=47.11% ShadowCache=0.00% QueryCache=2.38%
1344609826 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2064195914 bytes
1344609829 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 12.500000, 13.735392
1344609841 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 10.600000, 13.727962
1344609846 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(3656 2435 3158702 0.078968) updates=(430 2308 15022128 0.375553 419)
1344609846 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/root' with latest revision older than 1312045326
1344609846 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/metadata' with latest revision older than 131204
1344609846 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/system' with latest revision older than 13120524
1344609846 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/user' with latest revision older than 1344528114
1344609846 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=2729.74, RSS=2013.62, tracked=1977.38, computed=1977.36 limit=4867.20
1344609846 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=49.74% BlockIndex=0.33% BloomFilter=0.34% CellCache=47.18% ShadowCache=0.00% QueryCache=2.40%
1344609846 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2073431430 bytes
1344609851 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 10.600000, 13.720567
1344609853 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) Entering get_statistics()
1344609853 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) Exiting get_statistics()
1344609866 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(2935 1241 3080258 0.077006) updates=(327 1795 9937466 0.248437 325)
1344609866 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/root' with latest revision older than 1312045326
1344609866 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/metadata' with latest revision older than 131204
1344609866 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/system' with latest revision older than 13120524
1344609866 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/user' with latest revision older than 1344528114
1344609866 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=2737.76, RSS=2021.03, tracked=1982.93, computed=1982.90 limit=4867.20
1344609866 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=49.70% BlockIndex=0.33% BloomFilter=0.34% CellCache=47.22% ShadowCache=0.00% QueryCache=2.38%
1344609866 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2079255544 bytes
1344609867 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 10.700000, 13.713443
1344609881 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 15.200000, 13.716941
1344609883 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) Entering get_statistics()
1344609883 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) Exiting get_statistics()
1344609886 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(2935 1241 3080258 0.077006) updates=(327 1795 9937466 0.248437 325)
1344609886 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/root' with latest revision older than 1312045326
1344609886 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/metadata' with latest revision older than 131204
1344609886 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/system' with latest revision older than 13120524
1344609886 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/user' with latest revision older than 1344528114
1344609886 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=2743.81, RSS=2025.84, tracked=1989.69, computed=1989.78 limit=4867.20
1344609886 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=49.58% BlockIndex=0.33% BloomFilter=0.34% CellCache=47.36% ShadowCache=0.00% QueryCache=2.39%
1344609886 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2086343584 bytes
1344609898 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 11.700000, 13.712207
1344609906 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(2319 864 1204825 0.030120) updates=(255 1485 8140719 0.203513 251)
1344609906 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/root' with latest revision older than 1312045326
1344609906 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/metadata' with latest revision older than 131204
1344609906 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/system' with latest revision older than 13120524
1344609906 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/user' with latest revision older than 1344528114
1344609906 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=2748.83, RSS=2029.66, tracked=1994.18, computed=1994.23 limit=4867.20
1344609906 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=49.54% BlockIndex=0.33% BloomFilter=0.34% CellCache=47.40% ShadowCache=0.00% QueryCache=2.38%
1344609906 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2091050463 bytes
1344609913 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 12.400000, 13.709133
1344609913 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) Entering get_statistics()
1344609913 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) Exiting get_statistics()
1344609926 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(2319 864 1204825 0.030120) updates=(255 1485 8140719 0.203513 251)
1344609926 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/root' with latest revision older than 1312045326
1344609926 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/metadata' with latest revision older than 131204
1344609926 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/system' with latest revision older than 13120524
1344609926 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/user' with latest revision older than 1344528114
1344609926 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=2755.86, RSS=2036.22, tracked=2000.89, computed=2000.86 limit=4867.20
1344609926 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=49.48% BlockIndex=0.33% BloomFilter=0.34% CellCache=47.47% ShadowCache=0.00% QueryCache=2.38%
1344609926 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2098082973 bytes
1344609930 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 9.500000, 13.699299

terminate called recursively
terminate called recursively
[[EOF]]

Thrift Logs:
...
1344602486 ERROR ThriftBroker : TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 36146>Broken pipe
1344602486 ERROR ThriftBroker : TThreadedServer client died: write() send(): Broken pipe
1344602487 ERROR ThriftBroker : TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 39636>Broken pipe
1344602487 ERROR ThriftBroker : TThreadedServer client died: write() send(): Broken pipe
1344602487 ERROR ThriftBroker : TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 36201>Broken pipe
1344602487 ERROR ThriftBroker : TThreadedServer client died: write() send(): Broken pipe
1344609962 ERROR ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:237) socket read(29, len=38) failure : Connection reset by peer
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756) FileUtils::writev(29, len=167) failed : Broken pipe
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem flushing send queue - COMM broken connection
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756) FileUtils::writev(29, len=167) failed : Broken pipe
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem flushing send queue - COMM broken connection
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756) FileUtils::writev(29, len=167) failed : Broken pipe
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem flushing send queue - COMM broken connection
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756) FileUtils::writev(29, len=167) failed : Broken pipe
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem flushing send queue - COMM broken connection
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756) FileUtils::writev(29, len=167) failed : Broken pipe
.................................................................
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem flushing send queue - COMM broken connection
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756) FileUtils::writev(29, len=167) failed : Broken pipe
1344609962 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: type=DISCONNECT from=178.162.190.234:38060; Problem connecting to Root RangeServer, will retry in 3000 miliseconds...
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem flushing send queue - COMM broken connection
1344609962 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756) FileUtils::writev(29, len=167) failed : Broken pipe
......................
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344610639 ERROR ThriftBroker : TThreadedServer: Caught TException: pthread_create failed
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 - COMM not connected
1344610639 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678) Comm::send_request to rs1 failed - COMM not connected

terminate called after throwing an instance of 'std::bad_alloc'

what(): std::bad_alloc
[[EOF]]

Christoph Rupp

unread,

Aug 10, 2012, 10:09:26 AM8/10/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

Is that error coming immediately or after a certain while under heavy load?

can you please post your "uname -a" and "ulimit -a"?

Thanks
Christoph

2012/8/10 Kenny F. <kfu...@gmail.com>

here we are:

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/YyJHcM5V_IQJ.

Kenny F.

unread,

Aug 10, 2012, 10:11:33 AM8/10/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

Graphs (attached):

On Friday, August 10, 2012 12:30:09 AM UTC+3, Doug Judd wrote:

hyp_g_4.jpg

Kenny F.

unread,

Aug 10, 2012, 10:20:32 AM8/10/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com, ch...@hypertable.com

>Is that error coming immediately or after a certain while under heavy load?

as a rule ~ 2 hours of work. I don't think is "heavy" load )

# uname -a

Linux 178-162-111-111.local 2.6.28.7 #1 SMP Fri Mar 13 13:04:30 UTC 2009 i686 GNU/Linux

# ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Kenny F.

unread,

Aug 10, 2012, 10:46:55 AM8/10/12

to hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com, ch...@hypertable.com

# ulimit -Hn

262144

Doug Judd

unread,

Aug 10, 2012, 4:00:07 PM8/10/12

to hyperta...@googlegroups.com, Kenny F. (2), ch...@hypertable.com

Hi Kenny,

I suppose it's possible that your system is just running out of memory. Can you post the output of the 'free' command, for example:

$ free

total used free shared buffers cached

Mem: 24605128 4570252 20034876 0 232512 2818844

-/+ buffers/cache: 1518896 23086232

Swap: 49149800 9648 49140152

- Doug

On Fri, Aug 10, 2012 at 7:46 AM, Kenny F. <kfu...@gmail.com> wrote:

# ulimit -Hn

262144

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/Xif8ri18SwMJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 13, 2012, 3:35:52 AM8/13/12

to hyperta...@googlegroups.com, Kenny F. (2), ch...@hypertable.com, do...@hypertable.com

Hi Doug,

# free

total used free shared buffers cached

Mem:       8301272    8015704     285568          0     128476    5499072
-/+ buffers/cache:    2388156    5913116
Swap:     17775912     614476   17161436

Kenny F.

unread,

Aug 14, 2012, 5:01:39 AM8/14/12

to hyperta...@googlegroups.com, Kenny F. (2), ch...@hypertable.com, do...@hypertable.com

Hi Doug,

I think, may be the reason of crash is is there:

I had low setRecvTimeout/setSendTimeout for ThiftClient.
When system work slowly, ThiftClient try to connect a lot of times but halt the connection (low setRecvTimeout).

On Friday, August 10, 2012 11:00:07 PM UTC+3, Doug Judd wrote:

Hi Kenny,

I suppose it's possible that your system is just running out of memory. Can you post the output of the 'free' command, for example:

$ free

Kenny F.

unread,

Aug 14, 2012, 6:11:45 AM8/14/12

to hyperta...@googlegroups.com, Kenny F. (2), ch...@hypertable.com, do...@hypertable.com

RangeServer Logs:
...
1344945378 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2082449089 bytes
1344945380 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) Entering get_statistics()
1344945380 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) Exiting get_statistics()
1344945385 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 11.100000, 13.433875
1344945398 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(2770 2156 4691120 0.117278) updates=(316 1788 9761455 0.244036 316)
1344945398 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/root' with latest revision older than 1312045326
1344945398 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/metadata' with latest revision older than 131204
1344945398 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/system' with latest revision older than 13120524
1344945398 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log fragments from '/hypertable/servers/rs1/log/user' with latest revision older than 1344607601
1344945398 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) Memory Statistics (MB): VM=2763.03, RSS=2035.55, tracked=1991.85, computed=1992.76 limit=4867.20
1344945398 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) Memory Allocation: BlockCache=48.84% BlockIndex=0.32% BloomFilter=0.31% CellCache=48.13% ShadowCache=0.00% QueryCache=2.39%
1344945398 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2088603716 bytes
1344945401 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83) QueryCache hit rate over last 1000 lookups, cumulative = 10.100000, 13.426157
1344945405 FATAL Hypertable.RangeServer : operator() (/root/src/hypertable/src/cc/Hypertable/RangeServer/UpdateThread.cc:44): Hypertable::Exception: failed expectation: page - HYPERTABLE bad memory allocation
at Hypertable::PageArena<CharT, PageAllocatorT>::Page* Hypertable::PageArena<CharT, PageAllocatorT>::alloc_page(size_t, bool) [with CharT = unsigned char, PageAllocatorT = Hypertable::CellCachePageAllocator] (/root/src/hypertable/src/cc/Common/PageArena.h:122)

On Friday, August 10, 2012 11:00:07 PM UTC+3, Doug Judd wrote:

Hi Kenny,

I suppose it's possible that your system is just running out of memory. Can you post the output of the 'free' command, for example:

$ free
total used free shared buffers cached

Mem: 24605128 4570252 20034876 0 232512 2818844
-/+ buffers/cache: 1518896 23086232
Swap: 49149800 9648 49140152

- Doug

Kenny F.

unread,

Aug 14, 2012, 9:51:38 AM8/14/12

to hyperta...@googlegroups.com, Kenny F. (2), ch...@hypertable.com, do...@hypertable.com

0.9.6.0.95b0abc tracks:

# screen -r

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x99ec9b70 (LWP 15640)]
0xffffe424 in __kernel_vsyscall ()

# where

#0 0xffffe424 in __kernel_vsyscall ()
#1 0xb7b28781 in raise () from /lib/i686/cmov/libc.so.6
#2 0xb7b2bbb2 in abort () from /lib/i686/cmov/libc.so.6
#3 0xb7d34959 in __gnu_cxx::__verbose_terminate_handler() () from /opt/hypertable/0.9.6.0.95b0abc/lib/libstdc++.so.6
#4 0xb7d32865 in ?? () from /opt/hypertable/0.9.6.0.95b0abc/lib/libstdc++.so.6
#5 0xb7d328a2 in std::terminate() () from /opt/hypertable/0.9.6.0.95b0abc/lib/libstdc++.so.6
#6 0xb7d329da in __cxa_throw () from /opt/hypertable/0.9.6.0.95b0abc/lib/libstdc++.so.6
#7 0xb7d33033 in operator new(unsigned int) () from /opt/hypertable/0.9.6.0.95b0abc/lib/libstdc++.so.6
#8 0xb7d3311d in operator new[](unsigned int) () from /opt/hypertable/0.9.6.0.95b0abc/lib/libstdc++.so.6
#9 0x0869ef81 in Hypertable::DynamicBuffer::grow (this=0x99ec7ff4, new_size=1000004, nocopy=false) at /root/src/hypertable/src/cc/Common/DynamicBuffer.h:120
#10 0x0869f095 in Hypertable::DynamicBuffer::reserve (this=0x99ec7ff4, len=1000004, nocopy=false) at /root/src/hypertable/src/cc/Common/DynamicBuffer.h:72
#11 0x08773887 in Hypertable::FillScanBlock (scanner=..., dbuf=..., buffer_size=1000000) at /root/src/hypertable/src/cc/Hypertable/RangeServer/FillScanBlock.cc:104
#12 0x086628c1 in Hypertable::RangeServer::create_scanner (this=0x8c95460, cb=0x99ec92a8, table=0x99ec929c, range_spec=0x99ec9290, scan_spec=0x99ec9200, cache_key=0x99ec9278)
at /root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1371
#13 0x087d53e6 in Hypertable::RequestHandlerCreateScanner::run (this=0x331d9840) at /root/src/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerCreateScanner.cc:59
#14 0x0862debc in Hypertable::ApplicationQueue::Worker::operator() (this=0x8c83cf8) at /root/src/hypertable/src/cc/AsyncComm/ApplicationQueue.h:172
#15 0x0862df18 in boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run (this=0x8c83c28) at /usr/local/include/boost/thread/detail/thread.hpp:61
#16 0xb7ebfe68 in thread_proxy () from /opt/hypertable/0.9.6.0.95b0abc/lib/libboost_thread.so.1.44.0
#17 0xb7e05955 in start_thread () from /lib/i686/cmov/libpthread.so.0
#18 0xb7bca5ee in clone () from /lib/i686/cmov/libc.so.6

debug_errs_h_0.9.6.0.95b0abc.txt

Mehmet Ali Cetinkaya

unread,

Aug 14, 2012, 10:38:41 AM8/14/12

to hyperta...@googlegroups.com

Hello,

i have a Hadoop (0.20)+Hypertable (0.9.6) system that one master and two slaves.

we inserted 1 million data to hypertable succesfully.

But now, when use the select query ( select meta:eklemetarihi from urls; )

hypertable frozen after approximate 300.000 data show.

i didn't see any error in hypertable logs.

i read some document from internet. and their solution is "you must erase cell cache of hypertable".

but i didn't find how can i erase cell cache.

what can be done to resolvee this issue?

thanx,

mali

Doug Judd

unread,

Aug 14, 2012, 6:53:34 PM8/14/12

to hyperta...@googlegroups.com

Hi Mali,

This could be a number of things. First, check to make sure the RangeServers (slaves) are idle when the query appears to be hanging. I've witnessed situations where a query appears to hang, but is actually still being executed, scanning over a large section of data that, for example, does not contain the column "meta:eklemetarihi". The next thing to do is to double-check that there are no errors in the Hypertable logs. You can do this with:

grep -i ERROR /opt/hypertable/current/log/*.log

grep -i Except /opt/hypertable/current/log/*.log

Next double-check the Hadoop logs to be sure there are no errors (see your Hadoop distro documentation for location of logs). If none of the above turns up anything, let us know and we can help you capture stack traces of the RangeServers to determine if they're stuck in a deadlock.

- Doug

--

You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 16, 2012, 4:26:30 AM8/16/12

to hyperta...@googlegroups.com, do...@hypertable.com

Hi Doug,

As I understand, the problem is in the memory.
So, I've played with Hypertable.RangeServer.MemoryLimit
I looked at Virtual Memory, crash is on of ~3.2Gb load.
I made Hypertable.RangeServer.MemoryLimit=2800Mb and had a crash(at ~13:20) and looks like Hypertable even didn't notice the limit (see pictures).
I made Hypertable.RangeServer.MemoryLimit=2300Mb and had a crash(at ~16:00) and looks like Hypertable notice the limit at 3.2Gb. why not at 2.3Gb - I don't know..
I made Hypertable.RangeServer.MemoryLimit=1900Mb and finally have no crash, virtual memory stabilized at at 2.9Gb. why not at 1.9Gb - I don't know.

At "Virtual Memory/Resident Memory", we see that with limit=2.3Gb we capped limit, but have a crash; with limit=1.9Gb we capped limit.

How I see the reason: let's look at "Block Cache Max Memory".
When we have no limit or hight limit, Hypertable think he can use ALL memory for caching. And doesn't matter, that it is already fill at 80%-90%.
Only after reaching the cap limit it try not to use the rest of memory ))

Anyway - I added other graphs to compare.

Kenny F.

hyp_g_w1.jpg

hyp_g_w2.jpg

hyp_g_w3.jpg

hyp_g_w4.jpg

hyp_g_w5.jpg

Message has been deleted

Doug Judd

unread,

Aug 16, 2012, 8:02:39 AM8/16/12

to hyperta...@googlegroups.com, Kenny F. (2), ch...@hypertable.com

Hi Kenny,

One other thought. Is it possible that there is some other process (or set of processes) on the machine at the time of the crash that is using up all virtual memory? Try running 'free' at regular intervals during your test to see if the system is running out of swap space.

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/mV9fO9SeT_IJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 16, 2012, 8:51:55 AM8/16/12

to hyperta...@googlegroups.com, Kenny F. (2), ch...@hypertable.com, do...@hypertable.com

>Is it possible that there is some other process (or set of processes) on the machine at the time of the crash that is using up all virtual memory?

I don't think so.
First, I have 2 memory greed processes only: Hypertable and Apache.
Second, I didn't change the environment. And restarting the system didn't solve the issue.
Third, RangeServer didn't live more then several hours.
And I don't remember, when RangeServer crashes immediately.

Christoph Rupp

unread,

Aug 16, 2012, 9:04:31 AM8/16/12

to Kenny F., hyperta...@googlegroups.com, Kenny F. (2), do...@hypertable.com

Hi Kenny,

can you try reducing the memory usage of the RangeServer? By default the RangeServer uses 60% of the memory.

There are two properties:

Hypertable.RangeServer.MemoryLimit or Hypertable.RangeServer.MemoryLimit.Percentage.

You can use either of them.

Bye
Christoph

2012/8/16 Kenny F. <kfu...@gmail.com>

Kenny F.

unread,

Aug 16, 2012, 9:15:30 AM8/16/12

to hyperta...@googlegroups.com, Kenny F., Kenny F. (2), do...@hypertable.com, ch...@hypertable.com

Thanks!

I've already done it, it helped and post graphs (today) for you to interpret them.
Earlier I used Hypertable.RangeServer.MemoryLimit.Percentage=20 or 30 or 40 and it didn't help me at all.
Now I use Hypertable.RangeServer.MemoryLimit=1900Mb (see my post today with graphs)

I think, the reason is when we have no limit or high limit, RangeServer use ALL memory (see graph "Block Cache Max Memory"), even if it is fill at 95% or more.

Christoph Rupp

unread,

Aug 16, 2012, 9:19:53 AM8/16/12

to hyperta...@googlegroups.com, Kenny F., Kenny F. (2), do...@hypertable.com

Well, by default the RangeServer uses 60%, but that memory is used for caching CellStores.

If there's additional memory required to fetch data for a single request, then it can exceed the limit for a short while. It seems that this might have happened here, If you have very large scans with lots of data (and in addition some of the memory was allocated by apache).

But i am glad that we worked it out :)

bye
Christoph

2012/8/16 Kenny F. <kfu...@gmail.com>

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/CWqXo6VjsGQJ.

Kenny F.

unread,

Aug 17, 2012, 10:24:45 AM8/17/12

to hyperta...@googlegroups.com, Kenny F., Kenny F. (2), do...@hypertable.com, ch...@hypertable.com

))
I hope this issue help you to locate the problem.
Hypertable's RangeServers use ALL memory(not all FREE memory) when have no memory limit (or high memory limit).
It is easy to watch "Block Cache Max Memory" at RangeServer Statistics (monitoring).
This issue appeared between Hypertable 0.9.5.0 and Hypertable 0.9.5.4.
I hope this bug will be fix.

Doug Judd

unread,

Aug 17, 2012, 12:58:05 PM8/17/12

to hyperta...@googlegroups.com, Kenny F., Kenny F. (2), ch...@hypertable.com

Hi Kenny,

I think I may know what's going on here. Can you try your test again with the following changes:

1. Revert the Hypertable.RangeServer.MemoryLimit changes. In other words, leave the RangeServer memory properties at their defaults (60% of physical RAM).

2. Add the following property to your hypertable.cfg to disable the block cache:

Hypertable.RangeServer.BlockCache.MaxMemory=0

I suspect that the RangeServer is fighting with the OS file cache for RAM. Disabling the block cache is not as bad as you might think. The OS will cache the compressed blocks in its file cache.

- Doug

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/WDrfBDUZHZMJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 20, 2012, 4:04:31 AM8/20/12

to hyperta...@googlegroups.com, Kenny F., Kenny F. (2), ch...@hypertable.com, do...@hypertable.com

Let's try.

On Friday, August 17, 2012 7:58:05 PM UTC+3, Doug Judd wrote:

Hi Kenny,

- Doug

2012/8/16 Kenny F. <kfu...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/CWqXo6VjsGQJ.
To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/WDrfBDUZHZMJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 21, 2012, 3:44:13 AM8/21/12

to hyperta...@googlegroups.com, Kenny F., Kenny F. (2), ch...@hypertable.com, do...@hypertable.com

Hi Doug,

Here some results:
RangeServer works for about 24 hours without crash.

"Block Cache Fill", "Block Cache Hit Rate", "Block Cache Max Memory" are zero.

"Virtual Memory" slowly rose up, but it still under the "cap" level, so my be it fall down but a bit later.
At the picture we see restart ~10AM.

Or you already found the reason of crash.)

Regards,
Kenny F.

- Doug

2012/8/16 Kenny F. <kfu...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/CWqXo6VjsGQJ.
To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/WDrfBDUZHZMJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

hyp_g_w6.jpg

hyp_g_w7.jpg

Doug Judd

unread,

Aug 21, 2012, 12:25:51 PM8/21/12

to hyperta...@googlegroups.com

Hi Kenny,

So that looks pretty good. It looks like it solved your problem, correct? If so, we'll probably change the block cache default to disabled.

- Doug

To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/5aqMdL8wbTIJ.

To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

Kenny F.

unread,

Aug 31, 2012, 4:20:29 AM8/31/12

to hyperta...@googlegroups.com, do...@hypertable.com

Hi Doug!,

Hypertable works fine )

Reply all

Reply to author

Forward