0.9.7.5 on Ubuntu Server 10.10 64-bit, complete database loss?

[Gábor Dicső]

Hi All,

I took the following steps during the last few weeks:

- installed Hypertable 0.9.7.4 on Ubuntu Server 10.10 64-bit (open file limit previously set to 65536), created two tables in "/" namespace, inserted a few million rows

- upgraded it to 0.9.7.5, created the same two tables in "/" namespace (not sure, but I think my previous tables were not there)

- inserted a few million rows of data in both tables

- my application that's using Hypertable could not start TCP listening because of "Too many open files" errors afterwards, so I decided to reboot the server

- after reboot, my tables were gone: Hypertable shell does not show any tables in "/" namespace

What I've checked/tried:

- Hypertable restart - did not help

- checked directory sizes for different Hypertable versions in /opt/hypertable:

--- 0.9.4.3: 29Gb (application data collected so far)

--- 0.9.7.4: 7.5Gb (application data partially copied before 0.9.7.5 released, directory size seems to be a bit much)

--- 0.9.7.5: 263Mb (application data partially copied after upgrading to 0.9.7.5, should be around 6-8Gb)

- checked Hypertable log files, excerpt below

Not 100% sure, but I think this was the first server reboot since I upgraded from 0.9.7.4 to 0.9.7.5. Can it be that after the upgrade, 0.9.7.5 somehow still saved data in 0.9.7.4's folder, then after the reboot it expected them to be in the 0.9.7.5 folder? Or are my data completely lost? If so, why? Thanks,

Gábor

----------

Hypertable.RangeServer.log excerpt, I can send all logs if necessary:

(...)

1368961570 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:4572) Memory Usage: 1572864 bytes

1368961570 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=10.0.201.7:65161

1368961570 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:156) Received shutdown command

1368961570 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TimerHandler.cc:152) TimerHandler shutting down.

1368961570 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/GroupCommitTimerHandler.cc:61) CommitIntervalGroupCommitTimerHandler shutting down.

1368961570 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:242) socket read(13, len=38) failure : Connection reset by peer

1368961570 ERROR Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:309) No connection for 10.0.201.7:65161 - COMM not connected

1368961571 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=10.0.201.7:65161

1368961572 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerShutdown.cc:43) Exiting RangeServer.

1368961666 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Common/Config.cc:577) Initializing Hypertable.RangeServer (Hypertable 0.9.7.5 (v0.9.7.5-0-g22129c7-dirty))...

CPU cores count=2

CephBroker.MonAddr=10.0.1.245:6789

DfsBroker.Local.Root=fs/local

DfsBroker.Port=38030

HdfsBroker.Hadoop.ConfDir=/etc/hadoop/conf

Hyperspace.GracePeriod=200000

Hyperspace.KeepAlive.Interval=30000

Hyperspace.Lease.Interval=1000000

Hyperspace.Replica.Dir=hyperspace

Hyperspace.Replica.Host=[localhost]

Hyperspace.Replica.Port=38040

Hypertable.Master.Port=38050

Hypertable.RangeServer.Port=38060

Hypertable.RangeServer.Reactors=2

Hypertable.Verbose=true

ThriftBroker.Port=38080

dfs-port=38030

grace-period=200000

hs-host=[localhost]

hs-port=38040

keepalive=30000

lease-interval=1000000

master-port=38050

pidfile=/opt/hypertable/current/run/Hypertable.RangeServer.pid

port=38060

reactors=2

verbose=true

1368961666 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/HyperspaceSessionHandler.cc:31) Hyperspace session state change:  SAFE

drive count = 1

maintenance threads = 2

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:227) Event: type=CONNECTION_ESTABLISHED from=10.0.201.7:38050

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:511) log_dir=/hypertable/servers/rs1/log

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:4454) adding range

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TableInfo.cc:259) Adding range 0/0[..0/0:ÿÿ] to TableInfo

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:4467) Successfully replay loaded range 0/0[..0/0:ÿÿ]

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1014) Replayed 2 blocks of updates from '/hypertable/servers/rs1/log/root'

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:112) Range reference for '/hypertable/servers/rs1/log/root' is required

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:4454) adding range

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TableInfo.cc:259) Adding range 0/0[0/0:ÿÿ..ÿÿ] to TableInfo

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:4467) Successfully replay loaded range 0/0[0/0:ÿÿ..ÿÿ]

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1014) Replayed 2 blocks of updates from '/hypertable/servers/rs1/log/metadata'

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TableInfo.cc:259) Adding range 0/0[0/0:ÿÿ..ÿÿ] to TableInfo

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:112) Range reference for '/hypertable/servers/rs1/log/metadata' is required

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:4454) adding range

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/TableInfo.cc:259) Adding range 0/1[..ÿÿ] to TableInfo

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:4467) Successfully replay loaded range 0/1[..ÿÿ]

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1014) Replayed 1 blocks of updates from '/hypertable/servers/rs1/log/system'

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:112) Range reference for '/hypertable/servers/rs1/log/system' is required

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:112) Range reference for '/hypertable/servers/rs1/log/user' is required

1368961667 NOTICE Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:864) Replay finished

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:356) Prune thresholds - min=1000000000, max=2030043136

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) Maintenance stats scans=(0 0 0 0.000000) updates=(0 0 0 0.000000 0)

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:356) purge('/hypertable/servers/rs1/log/root') breaking on fragment 0 (rev=1368960974712153001 gen=1); rev=1368960974641573001 gen=6; ref_req=yes rm_ok=yes

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:356) purge('/hypertable/servers/rs1/log/metadata') breaking on fragment 0 (rev=1368960974886108001 gen=3); rev=1368960974850700001 gen=6; ref_req=yes rm_ok=yes

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:356) purge('/hypertable/servers/rs1/log/system') breaking on fragment 0 (rev=1368961208985471006 gen=5); rev=1368961208985471001 gen=6; ref_req=yes rm_ok=yes

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:286) Memory Statistics (MB): VM=545.15, RSS=9.11, tracked=1.50, computed=49.18 limit=2323.20

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:291) Memory Allocation: BlockCache=0.00% BlockIndex=0.00% BloomFilter=0.00% CellCache=3.05% ShadowCache=0.00% QueryCache=96.95%

1368961667 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:4572) Memory Usage: 1572864 bytes

1368961669 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/AsyncComm/IOHandler.h:149) Event: type=CONNECTION_ESTABLISHED from=10.0.201.7:59352

(...)

--

Üdvözlettel / Best Regards

Gabor Dicso | +36308237933 | Gabor...@inepex.com
http://inepex.com | IT Development and Location Based Services

http://inetrack.com | The customizable fleet tracking platform

[Gábor Dicső]

For the sake of completeness, it is a standalone installation.

2013/5/19 Gábor Dicső <gabor...@inepex.com>

[Doug Judd]

Hi Gabor,

Can you send me the log files as well as the output of the following commands:

ls -lR /opt/hypertable/0.9.7.5/fs/local

and the ~/hs-dump file generated with the following command:

echo "dump / \"~/hs-dump\";" | /opt/hypertable/0.9.7.5/bin/ht hyperspace --batch

With all of that information, I should have enough to figure out what's going on.

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Doug Judd

CEO, Hypertable Inc.

[Gábor Dicső]

Hi Doug,

I sent you a mail with the requested info. Thanks,

Gábor

2013/5/19 Doug Judd <do...@hypertable.com>