RS1 (RangeServer) Lost

[mehmet ali cetinkaya]

Hello,

we setup a system with 3 data nodes + master + 2 rangeserver and current version is 0.9.7.5 (upgraded from 0.9.7.1). 

we have problem about rs1. when system running rs1 RangeServer.log file grew with like this logs;

RangeServer Logs -> 

1367998993 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TableInfo.cc:259) Adding range 2/8[be358b80-b9e6-4d63-bc0b-ccd2b0bb0b2a..be77787b-ca3f-4978-af48-62908e545ced] to TableInfo

1367998993 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:4467) Successfully replay loaded range 2/8[be358b80-b9e6-4d63-bc0b-ccd2b0bb0b2a..be77787b-ca3f-4978-af48-62908e545ced]

1367999004 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3113) Entering get_statistics()

1367999004 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3363) Exiting get_statistics()

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=172.16.200.59:62810

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:256) Loading cellstores for '0/0[..0/0:��]'

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:344) Finished loading cellstores for '0/0[..0/0:��]'

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:256) Loading cellstores for '0/0[0/0:��..��]'

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=172.16.200.58:61643

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:315) Loading CellStore 0/0/default/qyoNKN5rd__dbHKv/cs1

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:344) Finished loading cellstores for '0/0[0/0:��..��]'

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

1367999030 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

1367999031 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

1367999032 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

1367999032 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

1367999033 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

1367999033 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

1367999033 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

1367999033 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, more=0

i don't now why. Because system was running. after than rs1 down. we started again rs1 that it started. but rs1 RangeServer.log file grew again and rs1 down again. 

when i cap stop/start hypertable said "rangeserver is waiting..." (rs1) 

i can see rs2's datas in monitoring. But rs1's datas not there.

how can i resolve this issue with no data loss? this is logs;

Hypertable.Master.log

1368024541 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:225) Leaving RecoverServerRanges rs1 plan_generation=1 type=3 state=PHANTOM_LOAD

1368024541 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoveryBlocker.cc:44) Entering RecoveryBlocker-104089

1368024541 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoveryBlocker.cc:53) total_servers=2 connected_servers=2 quorum=2

1368024541 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoveryBlocker.cc:84) Leaving RecoveryBlocker-104089 state=COMPLETE

1368024541 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:64) Entering RecoverServerRanges (0x1756000) rs1 type=3 plan_generation=1 state=PHANTOM_LOAD

1368024541 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:328) Calling phantom_load(plan_generation=1, location=rs2) for 412 user ranges

1368024541 ERROR Hypertable.Master : phantom_load_ranges (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:335): Hypertable::Exception: RangeServer phantom_load() failure : Unable to verify schema for 3/4Problem getting attribute 'schema' of hyperspace file '/hypertable/tables/3/4' - HYPERSPACE file not found

        at void Hypertable::RangeServerClient::phantom_load(const Hypertable::CommAddress&, const Hypertable::String&, int, const std::vector<unsigned int, std::allocator<unsigned int> >&, const std::vector<Hypertable::QualifiedRangeSpec, std::allocator<Hypertable::QualifiedRangeSpec> >&, const std::vector<Hypertable::RangeState, std::allocator<Hypertable::RangeState> >&) (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)

1368024541 ERROR Hypertable.Master : phantom_load_ranges (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:340): Failed to issue phantom_load calls

1368024541 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:225) Leaving RecoverServerRanges rs1 plan_generation=1 type=3 state=PHANTOM_LOAD

Hypertable.Rangeserver (rs1) logs;

1368001942 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3113) Entering get_statistics()

1368001942 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3363) Exiting get_statistics()

1368001954 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(0 0 0 0.000000) updates=(0 0 0 0.000000 0)

1368001954 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:4572) Memory Usage: 0 bytes

1368001974 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(0 0 0 0.000000) updates=(0 0 0 0.000000 0)

1368001974 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:4572) Memory Usage: 0 bytes

1368001977 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=172.16.200.44:53512

1368001977 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:227) Event: type=DISCONNECT from=172.16.200.44:53512

1368001979 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:384) Received event Event: type=DISCONNECT from=172.16.200.57:38050

1368001979 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:424) Event: type=DISCONNECT from=172.16.200.57:38050; Problem connecting to Master, will retry in 5000 milliseconds...

1368001979 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:227) Event: type=DISCONNECT from=172.16.200.57:38050

1368001985 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:635) connect() completion error - Connection refused

1368001985 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:384) Received event Event: type=DISCONNECT from=172.16.200.57:38050

1368001985 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:424) Event: type=DISCONNECT from=172.16.200.57:38050; Problem connecting to Master, will retry in 5000 milliseconds...

1368001985 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:227) Event: type=DISCONNECT from=172.16.200.57:38050

1368001986 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=172.16.200.58:52091

1368001986 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:156) Received shutdown command

1368001986 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TimerHandler.cc:152) TimerHandler shutting down.

1368001986 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/GroupCommitTimerHandler.cc:61) CommitIntervalGroupCommitTimerHandler shutting down.

1368001986 ERROR Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:309) No connection for 172.16.200.58:52091 - COMM not connected

1368001986 ERROR Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:309) No connection for 172.16.200.58:52091 - COMM not connected

1368001986 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=172.16.200.58:60757

1368001988 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerShutdown.cc:43) Exiting RangeServer.

1368002202 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Common/Config.cc:577) Initializing Hypertable.RangeServer (Hypertable 0.9.7.5 (v0.9.7.5-0-g22129c7-dirty))...

ThrisBroker.log

1368001935 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:268) No connection for rs1 - COMM invalid proxy

1368001935 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:728) Comm::send_request to rs1 failed - COMM invalid proxy

1368001938 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:237) Connection attempt to RangeServer at rs1 failed - COMM invalid proxy.  Will retry again in 3000 milliseconds...

1368001939 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:268) No connection for rs1 - COMM invalid proxy

1368001939 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:728) Comm::send_request to rs1 failed - COMM invalid proxy

1368001941 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:237) Connection attempt to RangeServer at rs1 failed - COMM invalid proxy.  Will retry again in 3000 milliseconds...

1368001941 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:268) No connection for rs1 - COMM invalid proxy

1368001941 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:728) Comm::send_request to rs1 failed - COMM invalid proxy

1368001945 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:237) Connection attempt to RangeServer at rs1 failed - COMM invalid proxy.  Will retry again in 3000 milliseconds...

1368001945 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:268) No connection for rs1 - COMM invalid proxy

1368001945 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:728) Comm::send_request to rs1 failed - COMM invalid proxy

1368001947 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:268) No connection for rs1 - COMM invalid proxy

1368001947 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:728) Comm::send_request to rs1 failed - COMM invalid proxy

1368001949 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:237) Connection attempt to RangeServer at rs1 failed - COMM invalid proxy.  Will retry again in 3000 milliseconds...

1368001951 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:268) No connection for rs1 - COMM invalid proxy

1368001951 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:728) Comm::send_request to rs1 failed - COMM invalid proxy

1368001953 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:268) No connection for rs1 - COMM invalid proxy

1368001953 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:728) Comm::send_request to rs1 failed - COMM invalid proxy

1368001953 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:237) Connection attempt to RangeServer at rs1 failed - COMM invalid proxy.  Will retry again in 3000 milliseconds...

1368001956 INFO ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:237) Connection attempt to RangeServer at rs1 failed - COMM invalid proxy.  Will retry again in 3000 milliseconds...

1368001957 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:268) No connection for rs1 - COMM invalid proxy

1368001957 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:728) Comm::send_request to rs1 failed - COMM invalid proxy

1368001959 WARN ThriftBroker : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:268) No connection for rs1 - COMM invalid proxy

1368001959 WARN ThriftBroker : (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:728) Comm::send_request to rs1 failed - COMM invalid proxy

1368007317 INFO ThriftBroker : (/root/src/hypertable/src/cc/Common/Config.cc:577) Initializing ThriftBroker (Hypertable 0.9.7.5 (v0.9.7.5-0-g22129c7-dirty))...

CPU cores count=24

CephBroker.MonAddr=10.0.1.245:6789

DfsBroker.Local.Root=fs/local

DfsBroker.Port=38030

HdfsBroker.Hadoop.ConfDir=/etc/hadoop/conf

Hyperspace.GracePeriod=200000

Hyperspace.KeepAlive.Interval=30000

Hyperspace.Lease.Interval=1000000

Hyperspace.Replica.Dir=hyperspace

Hyperspace.Replica.Host=[hadoop57]

Hyperspace.Replica.Port=38040

Hypertable.Failover.Quorum.Percentage=100

Hypertable.Master.Port=38050

Hypertable.RangeServer.Port=38060

Hypertable.Verbose=true

ThriftBroker.Port=38080

pidfile=/opt/hypertable/current/run/ThriftBroker.pid

port=38080

reactors=24

verbose=true

1368007317 INFO ThriftBroker : (/root/src/hypertable/src/cc/Hyperspace/Session.cc:63) Hyperspace session setup to reconnect

1368007317 INFO ThriftBroker : (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:2497) Starting the server...

tcmalloc: large alloc 1668055040 bytes == 0xe4a000 @

tcmalloc: large alloc 1668055040 bytes == 0xe4a000 @

tcmalloc: large alloc 1668055040 bytes == 0xe4a000 @

tcmalloc: large alloc 1668055040 bytes == 0xe4a000 @

1368022577 INFO ThriftBroker : (/root/src/hypertable/src/cc/Common/Config.cc:577) Initializing ThriftBroker (Hypertable 0.9.7.5 (v0.9.7.5-0-g22129c7-dirty))...

Dfsbroker.log

May 8, 2013 5:16:01 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Create

INFO: Created file '/hypertable/servers/master/log/mml/21' handle = 2

May 8, 2013 5:16:13 PM org.hypertable.AsyncComm.IOHandlerData handle_message_body

WARNING: Received response for non-pending event (id=0,version=1,total_len=72)

May 8, 2013 5:16:15 PM org.hypertable.AsyncComm.IOHandlerData handle_message_body

WARNING: Received response for non-pending event (id=0,version=1,total_len=72)

May 8, 2013 5:21:01 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Remove

INFO: Removing file '/hypertable/tables/0/0/default/qyoNKN5rd__dbHKv/cs8

May 8, 2013 5:21:01 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Remove

INFO: Removing file '/hypertable/tables/0/0/default/qyoNKN5rd__dbHKv/cs9

May 8, 2013 5:21:01 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Remove

INFO: Removing file '/hypertable/tables/0/1/range/qyoNKN5rd__dbHKv/cs8

May 8, 2013 5:21:01 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Remove

INFO: Removing file '/hypertable/tables/0/1/range/qyoNKN5rd__dbHKv/cs9

May 8, 2013 5:49:01 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Close

INFO: Closing output file /hypertable/servers/master/log/mml/21 handle 2

May 8, 2013 5:49:01 PM org.hypertable.DfsBroker.hadoop.ConnectionHandler handle

INFO: [/172.16.200.57:56809 ; Wed May 08 17:49:01 EEST 2013] Disconnect - COMM broken connection : Closing all open handles from /172.16.200.57:56809

Closed 0 input streams and 0 output streams for client connection /172.16.200.57:56809

May 8, 2013 5:49:17 PM org.hypertable.AsyncComm.IOHandler DeliverEvent

INFO: [/172.16.200.57:59163 ; Wed May 08 17:49:17 EEST 2013] Connection Established

Closed 0 input streams and 0 output streams

May 8, 2013 5:49:17 PM org.hypertable.DfsBroker.hadoop.ConnectionHandler handle

INFO: [/172.16.200.57:59163 ; Wed May 08 17:49:17 EEST 2013] Disconnect - COMM broken connection : Closing all open handles from /172.16.200.57:59163

INFO: Removing file '/hypertable/tables/0/1/range/qyoNKN5rd__dbHKv/cs9

May 8, 2013 5:49:01 PM org.hypertable.DfsBroker.hadoop.HadoopBroker Close

INFO: Closing output file /hypertable/servers/master/log/mml/21 handle 2

May 8, 2013 5:49:01 PM org.hypertable.DfsBroker.hadoop.ConnectionHandler handle

INFO: [/172.16.200.57:56809 ; Wed May 08 17:49:01 EEST 2013] Disconnect - COMM broken connection : Closing all open handles from /172.16.200.57:56809

Closed 0 input streams and 0 output streams for client connection /172.16.200.57:56809

May 8, 2013 5:49:17 PM org.hypertable.AsyncComm.IOHandler DeliverEvent

INFO: [/172.16.200.57:59163 ; Wed May 08 17:49:17 EEST 2013] Connection Established

Closed 0 input streams and 0 output streams

May 8, 2013 5:49:17 PM org.hypertable.DfsBroker.hadoop.ConnectionHandler handle

INFO: [/172.16.200.57:59163 ; Wed May 08 17:49:17 EEST 2013] Disconnect - COMM broken connection : Closing all open handles from /172.16.200.57:59163

Closed 0 input streams and 0 output streams for client connection /172.16.200.57:59163

Hyperspace.log

1368024548 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:487) Destroying handle 14

1368024550 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 4

1368024550 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:317) destroyed session 4(Hypertable.RangeServer)

1368024550 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:468) Expiring session 4 name=Hypertable.RangeServer

1368024550 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:487) Destroying handle 10

1368024550 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:487) Destroying handle 12

1368024550 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1533) Persisting lock released notifications

1368024550 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1544) Finished persisting lock released notifications

1368024550 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42) Destroying session 5

1368024550 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:317) destroyed session 5(Hypertable.RangeServer)

best regards,

mali

[Doug Judd]

Hi Mali,

Can you tar up all of your logs and send them to me?  Ideally the entire log directory, but if that's too large, at least enough logs to cover the time period when the rs1 initially failed.  Also, can you copy the contents of the /hypertable/servers/master/log/mml and send it to me?  That directory is in HDFS, so you'll need to copy that with a command like:

hadoop fs -copyToLocal /hypertable/servers/master/log/mml/* .

Send those files to me as well and I'll take a look and try to figure out how to get you back up and running.

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-de...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Doug Judd

CEO, Hypertable Inc.

[mehmet ali cetinkaya]

Hi Doug,

My cluster's mml folder in the attach.

i hope recovery all data because we have 350gb data :(

mali

8 Mayıs 2013 Çarşamba 19:18:37 UTC+3 tarihinde Doug Judd yazdı:

[Doug Judd]

I'll need the log files as well (Hypertable.RangeServer.log for all RangeServers and Hypertable.Master.log).

- Doug

[Mehmet Ali Cetinkaya]

Hi Doug,

First of all, we are using same cluster with Hatice. And normaly we have same problem that she wrote a mail to you. And your answer are

"1. Stop Hypertable

2. Upgrade to 0.9.7.5 (see Upgrade)

3. Remove the /opt/hypertable/current/run/location file on the rs1 machine (the one giving you the ERROR message)

4. Restart Hypertable"

We maked your suggestions. 

1 machine is only hypertable master (hadoop57)

2 machine (rs1 and rs2) are rangeserver + datanode (hadoop58 and hadoop59)

1 machine is only hadoop datanode (hadoop54)

hadoop57's master, hyperspace and hadoop59's rangeserver logs are very huge. i used "tail -f " command. other logs are in mail attach and command result are;

hadoop57;

tail -f Hyperspace.log 

1368097675 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:487) Destroying handle 18

1368097675 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:487) Destroying handle 20

1368097675 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1533) Persisting lock released notifications

1368097675 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1544) Finished persisting lock released notifications

1368097675 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:487) Destroying handle 31

1368097675 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:487) Destroying handle 10

1368097675 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:487) Destroying handle 12

1368097675 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1533) Persisting lock released notifications

1368097675 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1544) Finished persisting lock released notifications

1368097675 INFO Hyperspace.Master : (/root/src/hypertable/src/cc/Hyperspace/Master.cc:487) Destroying handle 13

tail -f Hypertable.Master.log 

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:225) Leaving RecoverServerRanges rs1 plan_generation=1 type=3 state=PHANTOM_LOAD

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoveryBlocker.cc:44) Entering RecoveryBlocker-104093

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoveryBlocker.cc:53) total_servers=2 connected_servers=2 quorum=2

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoveryBlocker.cc:84) Leaving RecoveryBlocker-104093 state=COMPLETE

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:64) Entering RecoverServerRanges (0x18e8000) rs1 type=3 plan_generation=1 state=PHANTOM_LOAD

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:328) Calling phantom_load(plan_generation=1, location=rs2) for 412 user ranges

1368097650 ERROR Hypertable.Master : phantom_load_ranges (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:335): Hypertable::Exception: RangeServer phantom_load() failure : Unable to verify schema for 3/4Problem getting attribute 'schema' of hyperspace file '/hypertable/tables/3/4' - HYPERSPACE file not found

        at void Hypertable::RangeServerClient::phantom_load(const Hypertable::CommAddress&, const Hypertable::String&, int, const std::vector<unsigned int, std::allocator<unsigned int> >&, const std::vector<Hypertable::QualifiedRangeSpec, std::allocator<Hypertable::QualifiedRangeSpec> >&, const std::vector<Hypertable::RangeState, std::allocator<Hypertable::RangeState> >&) (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)

1368097650 ERROR Hypertable.Master : phantom_load_ranges (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:340): Failed to issue phantom_load calls

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:225) Leaving RecoverServerRanges rs1 plan_generation=1 type=3 state=PHANTOM_LOAD

hadoop59;

tail -f Hypertable.RangeServer.log 

1368097670 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:424) Event: type=DISCONNECT from=172.16.200.57:38050; Problem connecting to Master, will retry in 5000 milliseconds...

1368097670 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:227) Event: type=DISCONNECT from=172.16.200.57:38050

1368097672 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=172.16.200.59:52167

1368097672 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:156) Received shutdown command

1368097672 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TimerHandler.cc:152) TimerHandler shutting down.

1368097672 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/GroupCommitTimerHandler.cc:61) CommitIntervalGroupCommitTimerHandler shutting down.

1368097672 ERROR Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:309) No connection for 172.16.200.59:52167 - COMM not connected

1368097672 ERROR Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:309) No connection for 172.16.200.59:52167 - COMM not connected

1368097672 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=172.16.200.59:56615

1368097674 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerShutdown.cc:43) Exiting RangeServer.

---

From: Doug Judd <do...@hypertable.com>
To: hyperta...@googlegroups.com
Sent: Thursday, May 9, 2013 5:12 PM
Subject: Re: [hypertable-dev] RS1 (RangeServer) Lost

[Doug Judd]

Can you send me a tar or zip file instead of .rar?  I don't have a tool that can open that format.

- Doug

[Mehmet Ali Cetinkaya]

From: Doug Judd <do...@hypertable.com>
To: hyperta...@googlegroups.com

Sent: Thursday, May 9, 2013 5:48 PM

[Doug Judd]

The Hypertable.Master.log file does not exist in hadoop57_master_machine.tar.  Can you send me that file as well?

- Doug

[Mehmet Ali Cetinkaya]

All "small size" log files in attach. 

And hadoop57's master, hyperspace and hadoop59's rangeserver logs are very huge. i used "tail -f " command. other logs are in mail attach and command result are;

Sent: Thursday, May 9, 2013 6:33 PM

[Doug Judd]

The hadoop57_master_machine.tar file does not contain the Hypertable.Master.log file.  Can you send me that as well?

[Mehmet Ali Cetinkaya]

i see Doug, but master log is very huge (4.9 gb) and i used "tail -f" command but if it is not enough i will make try tar or maybe split and send to you last part of master.log ok?

tail -f Hypertable.Master.log 

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:225) Leaving RecoverServerRanges rs1 plan_generation=1 type=3 state=PHANTOM_LOAD

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoveryBlocker.cc:44) Entering RecoveryBlocker-104093

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoveryBlocker.cc:53) total_servers=2 connected_servers=2 quorum=2

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoveryBlocker.cc:84) Leaving RecoveryBlocker-104093 state=COMPLETE

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:64) Entering RecoverServerRanges (0x18e8000) rs1 type=3 plan_generation=1 state=PHANTOM_LOAD

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:328) Calling phantom_load(plan_generation=1, location=rs2) for 412 user ranges

1368097650 ERROR Hypertable.Master : phantom_load_ranges (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:335): Hypertable::Exception: RangeServer phantom_load() failure : Unable to verify schema for 3/4Problem getting attribute 'schema' of hyperspace file '/hypertable/tables/3/4' - HYPERSPACE file not found

        at void Hypertable::RangeServerClient::phantom_load(const Hypertable::CommAddress&, const Hypertable::String&, int, const std::vector<unsigned int, std::allocator<unsigned int> >&, const std::vector<Hypertable::QualifiedRangeSpec, std::allocator<Hypertable::QualifiedRangeSpec> >&, const std::vector<Hypertable::RangeState, std::allocator<Hypertable::RangeState> >&) (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)

1368097650 ERROR Hypertable.Master : phantom_load_ranges (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:340): Failed to issue phantom_load calls

1368097650 INFO Hypertable.Master : (/root/src/hypertable/src/cc/Hypertable/Master/OperationRecoverRanges.cc:225) Leaving RecoverServerRanges rs1 plan_generation=1 type=3 state=PHANTOM_LOAD

From: Doug Judd <do...@hypertable.com>
To: hyperta...@googlegroups.com

Sent: Friday, May 10, 2013 9:40 AM

[Doug Judd]

Ok, then run "grep ERROR Hypertable.Master.log" and send me the output.

- Doug

[mehmet ali cetinkaya]

i run "grep ERROR Hypertable.Master.log >> master_only_error_log.txt" command and new file size is 1.2 gb.

i run "cap start" and "cap stop". You can find all clusters log in attach.

i hope this logs are enough for issue this problem...

10 Mayıs 2013 Cuma 09:59:59 UTC+3 tarihinde Doug Judd yazdı:

[Doug Judd]

Also, please run the Hyperspace shell as follows:

/opt/hypertable/current/bin/ht hyperspace

and run the following command:

hyperspace> dump / "~/hsdump.txt";

and send me the ~/hsdump.txt file.

- Doug

[Hatice Çobanoğlu]

hi Doug,

i work with mehmet ali. i send you this file in attach.

bye

Hatice

2013/5/11 Doug Judd <do...@hypertable.com>

[Doug Judd]

Hi Hatice,

There is one more thing that I'd like you to do.  Please do the following and send me the output:

1. Stop hypertable (cap stop)

2. One the master machine, start the DFS broker:

/opt/hypertable/current/bin/start-dfsbroker.sh hadoop

3. Run the following command, change the "6" in the below command to match the largest range server in your setup (you can figure it out by running hadoop fs -ls /hypertable/servers.

for ((i=1; i<=6; i++)); do echo "rs$i"; /opt/hypertable/doug/current/bin/ht metalog_dump /hypertable/servers/rs$i/log/rsml; echo ""; done

4. Send me the output of the above command.

- Doug

P.S. Leave Hypertable stopped until I get back to you..

[Doug Judd]

I think I figured out what's going on.  It looks like rs1, the range server being recovered, contains a range for table '3/4' which has been dropped, which is causing the recovery to fail.  Clearly this range should have been dropped when the table was dropped.  I've filed issue 1069 to resolve this problem.  In the meantime, I can fix your MML to purge this bad range so that you can get your database back up and running.  I will do this on Monday and send you instructions on how to resolve the problem.  But please follow the instructions from my previous e-mail.  However, you only need to dump the rsml for rs1:

/opt/hypertable/doug/current/bin/ht metalog_dump /hypertable/servers/rs1/log/rsml

- Doug

[Hatice Çobanoğlu]

sorry Doug,
i'm not in office now.i am at another city.  i send you the outputs on monday morning.

thank you very much

Bye 

Hatice
12 Mayıs 2013 Pazar tarihinde Doug Judd adlı kullanıcı şöyle yazdı:

[Mehmet Ali Cetinkaya]

We will send the files to you tomorrow Doug. 

Thank you for all answers.

mali

---

From: Doug Judd <do...@hypertable.com>
To: hyperta...@googlegroups.com

Sent: Sunday, May 12, 2013 2:50 AM

[Hatice Çobanoğlu]

hi Doug,
 I got a error when i use metalog_dump tool. as follow:

root@hadoop57:~# /opt/hypertable/current/bin/ht metalog_dump /hypertable/servers/rs1/log/rsml
1368425526 ERROR metalog_dump : main (/root/src/hypertable/src/cc/Tools/metalog_dump/metalog_dump.cc:247): Hypertable::Exception: Error reading 16 bytes from DFS fd 7 - DFS BROKER i/o error
    at virtual size_t Hypertable::DfsBroker::Client::read(int32_t, void*, size_t) (/root/src/hypertable/src/cc/DfsBroker/Lib/Client.cc:278)
    at virtual size_t Hypertable::DfsBroker::Client::read(int32_t, void*, size_t) (/root/src/hypertable/src/cc/DfsBroker/Lib/Client.cc:272): java.io.IOException: File handle 7 not open for reading

2013/5/12 Mehmet Ali Cetinkaya <malice...@yahoo.com>

[Doug Judd]

Are you sure all of the Hypertable processes are stopped?  Be sure to run 'cap stop' and then verify that all the Hypertable processes are stopped with:

cap shell

cap> ps auxww | fgrep -i hyper | fgrep -v java

If not, manually kill the processes.  If you're still getting the same thing, send me the output of:

hadoop fs -ls /hypertable/servers/rs*/log/rsml

- Doug

[Hatice Çobanoğlu]

yes, we'r sure. i send you  log files in attach.

thanks,

Hatice

2013/5/13 Doug Judd <do...@hypertable.com>

[Doug Judd]

Ok, I'm not sure why that command didn't work for you, but I managed to run it successfully.  I'll send you the fixed MML file today.  BTW, can you also send me the contents of this directory:

/hypertable/servers/master/log/mml

Also, don't try to restart Hypertable until I get you the patched MML file.

- Doug

[Hatice Çobanoğlu]

i send you MML file in attach.

thanks

Hatice

[Doug Judd]

Thanks!

[Doug Judd]

Hi Hatice,

Attached is MML file 27 (SHA1 bdbad7794a9d40e6f7def224562cb05d19ad2e06).  Before you install it, make sure that the largest numerically name file in the mml directory (/hypertable/servers/master/log/mml) is 26.  If it is not 26, then stop Hypertable and send me the largest numerically named fragment file in that directory and do not restart Hypertable.  Each time you start Hypertable it will create a new file which I'll have to modify and send back to you.  If 26 is the largest file, then copy the attached file into /hypertable/servers/master/log/mml and then start Hypertable.  Be prepared to send me the log files if the system still has problems recovering.

- Doug

[Hatice Çobanoğlu]

hi Doug,

yes, file 26 is largest numerically name file in the mml directory. i copy file 27 into /hypertable/servers/master/log/mml and then start hypertable. I waited for a while. the system still has problems recovering. i send you logs file in attach.

(Did i copy file 27 into /opt/hypertable/current/run/log_backup/mml/master_38050/ ? )

2013/5/14 Doug Judd <do...@hypertable.com>

[Doug Judd]

Hi Hatice,

It looks like one of the RangeServers asserted.  The best way for me to get to the bottom of this would be for me to log in and work on your database directly.  Would that be possible?

- Doug

[Hatice Çobanoğlu]

Hi Doug

To access our servers or database hasn't been possible before. But i will ask my projects manager tomorrow.

Bye

Hatice
14 Mayıs 2013 Salı tarihinde Doug Judd adlı kullanıcı şöyle yazdı:

[Hatice Çobanoğlu]

sorry Doug, we can't. our servers working on only local. Do you have another resolve solition this problem ? what can i do?
the worst-case scenario is "cap cleandb" for our.

2013/5/14 Hatice Çobanoğlu <hattice....@gmail.com>

[Doug Judd]

Did the RangeServer that crashed create a core file?  If so, can you load the core file into gdb as follows:

gdb /opt/hypertable/current/bin/Hypertable.RangeServer <core-file>

and then run the following command and post the output:

(gdb) thread apply all where

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-de...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Hatice Çobanoğlu]

Hi Doug,

following output.
gdb /opt/hypertable/current/bin/Hypertable.RangeServer ~/core

GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /opt/hypertable/0.9.7.5/bin/Hypertable.RangeServer...done.

warning: core file may not match specified executable file.
[New LWP 3183]
[New LWP 3186]
[New LWP 3188]
[New LWP 3189]
[New LWP 3187]
[New LWP 3191]
[New LWP 3184]
[New LWP 3195]
[New LWP 3196]
[New LWP 3202]
[New LWP 3185]
[New LWP 3200]
[New LWP 3190]
[New LWP 3193]
[New LWP 3198]
[New LWP 3062]
[New LWP 3192]
[New LWP 3064]
[New LWP 3203]
[New LWP 3208]
[New LWP 3154]
[New LWP 3211]
[New LWP 3057]
[New LWP 3204]
[New LWP 3206]
[New LWP 3199]
[New LWP 3151]
[New LWP 3173]
[New LWP 3052]
[New LWP 3210]
[New LWP 3081]
[New LWP 3159]
[New LWP 3080]
[New LWP 3165]
[New LWP 3205]
[New LWP 3137]
[New LWP 2994]
[New LWP 3103]
[New LWP 3213]
[New LWP 3068]
[New LWP 3060]
[New LWP 3147]
[New LWP 3073]
[New LWP 3146]
[New LWP 3055]
[New LWP 3156]
[New LWP 3071]
[New LWP 3079]
[New LWP 3157]
[New LWP 3086]
[New LWP 3145]
[New LWP 3090]
[New LWP 3141]
[New LWP 3215]
[New LWP 3220]
[New LWP 3070]
[New LWP 3104]
[New LWP 3131]
[New LWP 3172]
[New LWP 3061]
[New LWP 3152]
[New LWP 3102]
[New LWP 3113]
[New LWP 3083]
[New LWP 3170]
[New LWP 3118]
[New LWP 3114]
[New LWP 3093]
[New LWP 3066]
[New LWP 3051]
[New LWP 3150]
[New LWP 3067]
[New LWP 3120]
[New LWP 3116]
[New LWP 3182]
[New LWP 3084]
[New LWP 3108]
[New LWP 3207]
[New LWP 3139]
[New LWP 3096]
[New LWP 3155]
[New LWP 3143]
[New LWP 3212]
[New LWP 3091]
[New LWP 3097]
[New LWP 3162]
[New LWP 3178]
[New LWP 3130]
[New LWP 3197]
[New LWP 3125]
[New LWP 3088]
[New LWP 3140]
[New LWP 3176]
[New LWP 3209]
[New LWP 3133]
[New LWP 3177]
[New LWP 3171]
[New LWP 3148]
[New LWP 3174]
[New LWP 3214]
[New LWP 3087]
[New LWP 3217]
[New LWP 3126]
[New LWP 3107]
[New LWP 3164]
[New LWP 3136]
[New LWP 3053]
[New LWP 3168]
[New LWP 3180]
[New LWP 3132]
[New LWP 3122]
[New LWP 3095]
[New LWP 3094]
[New LWP 3092]
[New LWP 3166]
[New LWP 3221]
[New LWP 3169]
[New LWP 3160]
[New LWP 3158]
[New LWP 3163]
[New LWP 3216]
[New LWP 3082]
[New LWP 3219]
[New LWP 3109]
[New LWP 3089]
[New LWP 3063]
[New LWP 3105]
[New LWP 3106]
[New LWP 3111]
[New LWP 3218]
[New LWP 3181]
[New LWP 3134]
[New LWP 3179]
[New LWP 3194]
[New LWP 3138]
[New LWP 3112]
[New LWP 3117]
[New LWP 3167]
[New LWP 3056]
[New LWP 3129]
[New LWP 3175]
[New LWP 3069]
[New LWP 3072]
[New LWP 3153]
[New LWP 3050]
[New LWP 3121]
[New LWP 3085]
[New LWP 3115]
[New LWP 3074]
[New LWP 3119]
[New LWP 3058]
[New LWP 3149]
[New LWP 3161]
[New LWP 3110]
[New LWP 3127]
[New LWP 3144]
[New LWP 3059]
[New LWP 3075]
[New LWP 3065]
[New LWP 3201]
[New LWP 3078]
[New LWP 3123]
[New LWP 3128]
[New LWP 3142]
[New LWP 3054]
[New LWP 3135]
[New LWP 3124]
Core was generated by `/opt/hypertable/current/bin/Hypertable.Master --pidfile /opt/hypertable/current'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000583604 in boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::gregorian::bad_day_of_month> >::rethrow() const ()

(gdb) thread apply all where

Thread 167 (LWP 3124):
#0  0x00007ff1dd37dd84 in ?? ()
#1  0x000047d100000000 in ?? ()
#2  0x00000000023cf060 in ?? ()
#3  0x00000000023cf038 in ?? ()
#4  0x00000000000ffb10 in ?? ()
#5  0x0000000002564478 in ?? ()
#6  0x00000000005a6f8a in Hypertable::TableInfo::add_range(boost::intrusive_ptr<Hypertable::Range>&, bool) ()
#7  0x0000000000000000 in ?? ()

Thread 166 (LWP 3135):
#0  0x00007ff1dd37dd84 in ?? ()
#1  0x000047d100000000 in ?? ()
#2  0x00000000023cf060 in ?? ()
#3  0x00000000023cf038 in ?? ()
#4  0x00000000000ffb10 in ?? ()
#5  0x0000000003ccbb98 in ?? ()
#6  0x00000000005a6f8a in Hypertable::TableInfo::add_range(boost::intrusive_ptr<Hypertable::Range>&, bool) ()
#7  0x0000000000000000 in ?? ()

Thread 165 (LWP 3054):
---Type <return> to continue, or q <return> to quit---

2013/5/15 Doug Judd <do...@hypertable.com>

[Doug Judd]

I'll build you a debug version which will give better debugging output.  If you could upgrade to that version, and try again, that will help me figure out what's going on.  I'll try to get the to you later today.

- Doug