Hypertable on Hadoop Cluster

vive...@gmail.com

unread,

Dec 10, 2009, 12:03:59 AM12/10/09

to Hypertable Development

Hi,

I have installed Hypertable and Hadoop. I want to use Hypertable on a
Hadoop cluster of 3 machines.
I want to run Namenode on 192.168.2.69, Jobtracker on 192.168.2.85,
Tasktracker and Datanode on 192.168.2.86

Please check whether my configurations are right or not.
Files in directory /opt/hypertable/0.9.2.7/conf are : Capfile.cluser,
Capfile.localhost, hypertable.cfg and Metadata.xml

--------------------------------------------------------------------------
Content of Capfile.cluster
set :source_machine, "192.168.2.69"
set :install_dir, "/opt/hypertable"
set :hypertable_version, "0.9.2.7"
set :default_dfs, "hadoop"
set :default_config, "/opt/hypertable/0.9.2.7/conf/hypertable.cfg"

role :master, "192.168.2.69"
role :slave, "192.168.2.69", "192.168.2.85", "192.168.2.86"
role :localhost, "192.168.2.69"
--------------------------------------------------------------------------
Content of Capfile.localhost (Not modified because cluster setup is
needed)
set :source_machine, "localhost"
set :install_dir, "/opt/hypertable"
set :hypertable_version, "0.9.2.7"
set :default_dfs, "local"
set :default_config, "/opt/hypertable/#{hypertable_version}/conf/
hypertable.cfg"

role :master, "localhost"
role :slave, "localhost"
role :localhost, "localhost"
--------------------------------------------------------------------------
Content of Hypertable.cfg
#
# hypertable.cfg
#

# Global properties
Hypertable.Request.Timeout=180000

# HDFS Broker
HdfsBroker.Port=38030
HdfsBroker.fs.default.name=hdfs://192.168.2.69:9000
HdfsBroker.Workers=20

# Ceph Broker
CephBroker.Port=38030
CephBroker.Workers=20
CephBroker.MonAddr=10.0.1.245:6789

# Local Broker
DfsBroker.Local.Port=38030
DfsBroker.Local.Root=fs/local

# DFS Broker - for clients
DfsBroker.Host=192.168.2.69
DfsBroker.Port=38030

# Hyperspace
Hyperspace.Master.Host=192.168.2.69
Hyperspace.Master.Port=38040
Hyperspace.Master.Dir=hyperspace
Hyperspace.Master.Workers=20

# Hypertable.Master
Hypertable.Master.Host=192.168.2.69
Hypertable.Master.Port=38050
Hypertable.Master.Workers=20

# Hypertable.RangeServer
Hypertable.RangeServer.Port=38060

Hyperspace.KeepAlive.Interval=30000
Hyperspace.Lease.Interval=1000000
Hyperspace.GracePeriod=200000

# ThriftBroker
ThriftBroker.Port=38080
------------------------------------------------------------------------------

When i am running "cap shell" i get cap prompt, but when i am trying
to execute date (to sync all machines) it is not showing any result.
Do i need to "capify /opt/hypertable/0.9.2.7/conf", if so then what
should be added in deploy.rb file because it throws error while doing
"cap deploy" (Do i really need to do that).
Also when i am doing "cap dist" is says "the task 'dist' does not
exist".

Please guide me to start cluster machine through Capistrano.

Thanks
Vivek

Doug Judd

unread,

Dec 10, 2009, 12:13:42 AM12/10/09

to hyperta...@googlegroups.com

The Capfile.cluster file in the conf directory of the tarball is meant to be an example. You need to copy it somewhere and rename it to "Capfile" and make your edits. Then run the 'cap' command in the same directory that contains 'Capfile' (or you can run 'cap -f <location-of-Capfile>'). With the Capfile setup correctly you should be able to start the system with 'cap start', stop the system with 'cap stop', and scrub the database clean with 'cap cleandb'

- Doug

--

You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

vive...@gmail.com

unread,

Dec 10, 2009, 1:14:30 AM12/10/09

to Hypertable Development

Thanks Doug,

It worked.
When i am running Date command on cap then i am getting following
result:
--------------------------------------------------------------------
cap> date
** [out :: 192.168.2.85] Thu Dec 10 11:29:45 IST 2009
** [out :: 192.168.2.86] Thu Dec 10 11:29:18 IST 2009
** [out :: 192.168.2.69] Thu Dec 10 11:29:53 IST 2009
--------------------------------------------------------------------

Which shows that time on all the servers are not same. Do we need to
do something else to sync them or it should have been taken care by
Date command itself?

later we tried doing cap dist, and here is the output for it:
--------------------------------------------------------------------
[root@localhost conf]# cap dist
* executing `dist'
** transaction: start
* executing `copy_config'
* executing "rsync /opt/hypertable/0.9.2.7/conf/hypertable.cfg /opt/
hypertable/0.9.2.7/conf"
servers: ["192.168.2.69"]
[192.168.2.69] executing command
command finished
* executing `rsync_installation'
* executing "rsync -av --exclude=log --exclude=run --exclude=demo --
exclude=fs --exclude=hyperspace 192.168.2.69:/opt/hypertable/0.9.2.7 /
opt/hypertable"
servers: ["192.168.2.69", "192.168.2.85", "192.168.2.86"]
Password:
[192.168.2.69] executing command
[192.168.2.86] executing command
[192.168.2.85] executing command
*** [err :: 192.168.2.85] Host key verification failed.
*** [err :: 192.168.2.85] rsync: connection unexpectedly closed (0
bytes received so far) [receiver]
*** [err :: 192.168.2.85] rsync error: unexplained error (code 255) at
io.c(632) [receiver=3.0.4]
*** [err :: 192.168.2.86] Host key verification failed.
*** [err :: 192.168.2.86] rsync: connection unexpectedly closed (0
bytes received so far) [receiver]
*** [err :: 192.168.2.86]
*** [err :: 192.168.2.86] rsync error: unexplained error (code 255) at
io.c(632) [receiver=3.0.4]
*** [err :: 192.168.2.86]
** [out :: 192.168.2.69] receiving incremental file list
** [out :: 192.168.2.69]
** [out :: 192.168.2.69] sent 343 bytes received 68029 bytes
45581.33 bytes/sec
** [out :: 192.168.2.69] total size is 635939178 speedup is 9301.16
command finished
failed: "sh -c 'rsync -av --exclude=log --exclude=run --exclude=demo --
exclude=fs --exclude=hyperspace 192.168.2.69:/opt/hypertable/0.9.2.7 /
opt/hypertable'" on 192.168.2.85,192.168.2.86
--------------------------------------------------------------------

What may be the problem?

Thanks
Vivek

On Dec 10, 10:13 am, Doug Judd <nuggetwh...@gmail.com> wrote:
> The Capfile.cluster file in the conf directory of the tarball is meant to be
> an example. You need to copy it somewhere and rename it to "Capfile" and
> make your edits. Then run the 'cap' command in the same directory that
> contains 'Capfile' (or you can run 'cap -f <location-of-Capfile>'). With
> the Capfile setup correctly you should be able to start the system with 'cap
> start', stop the system with 'cap stop', and scrub the database clean with
> 'cap cleandb'
>
> - Doug
>

> > hypertable-de...@googlegroups.com<hypertable-dev%2Bunsu...@googlegroups.com>
> > .

Doug Judd

unread,

Dec 10, 2009, 11:08:07 AM12/10/09

to hyperta...@googlegroups.com

Vivek,

You need to run NTP on all of your machines to get the clocks in sync. After you get NTP setup it will sometimes take a while (15 minutes) before the clocks synchronize, so be patient. After getting NTP setup, verify that it is running on all machines with:

$ ps auxww | fgrep ntp
doug 1145 0.0 0.0 61116 588 pts/0 S+ 08:06 0:00 fgrep ntp
ntp 2435 0.0 0.0 23368 5020 ? SLs Aug22 0:34 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g

As far as the 'cap dist' problem goes, make sure you can ssh to and from each machine without a password. You may need to add entries to the ~/.ssh/authorized_keys files to make this happen.

- Doug

To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.

Don Marti

unread,

Dec 10, 2009, 11:29:28 AM12/10/09

to hyperta...@googlegroups.com

begin Doug Judd quotation of Thu, Dec 10, 2009 at 08:08:07AM -0800:

> You need to run NTP on all of your machines to get the clocks in sync. After
> you get NTP setup it will sometimes take a while (15 minutes) before the clocks
> synchronize, so be patient. After getting NTP setup, verify that it is running
> on all machines with:
>
> $ ps auxww | fgrep ntp
> doug 1145 0.0 0.0 61116 588 pts/0 S+ 08:06 0:00 fgrep ntp
> ntp 2435 0.0 0.0 23368 5020 ? SLs Aug22 0:34 ntpd -u
> ntp:ntp -p /var/run/ntpd.pid -g

You can also ssh to all your hosts, drop the NTP
service with the init script, run
ntpdate [TIMESERVER]
as root, and restart NTP.

(If NTP is critical as it is here, it's a good idea to
have two systems on your LAN set up as the timeservers
for your other hosts, make them NTP peers of each
other, and don't have anyone except your timeservers
use an outside NTP server. That way things are less
likely to get wacky.)

--
Don Marti +1 510-332-1587 mobile
http://zgp.org/~dmarti/
dma...@zgp.org

vive...@gmail.com

unread,

Dec 11, 2009, 12:07:46 AM12/11/09

to Hypertable Development

Thanks and Doug and Don. I will install NTP on all machines and let
you know.

vive...@gmail.com

unread,

Dec 11, 2009, 5:05:12 AM12/11/09

to Hypertable Development

Hi,

I have installed OpenSSL, but I am getting following error while doing
"make install" for NTP
----------------------------------------------------------------------------------------------------------------------
make[6]: Leaving directory `/root/Download/ntp-4.2.4p8/sntp'
test -z "/usr/local/share/man/man1" || /bin/mkdir -p "/usr/local/share/
man/man1"
/usr/bin/install -c -m 644 sntp.1 sntp.1 '/usr/local/share/man/man1'
/usr/bin/install: will not overwrite just-created `/usr/local/share/
man/man1/sntp.1' with `sntp.1'
make[5]: *** [install-man1] Error 1
make[5]: Leaving directory `/root/Download/ntp-4.2.4p8/sntp'
make[4]: *** [install-am] Error 2
make[4]: Leaving directory `/root/Download/ntp-4.2.4p8/sntp'
make[3]: *** [install-recursive] Error 1
make[3]: Leaving directory `/root/Download/ntp-4.2.4p8/sntp'
make[2]: *** [install] Error 2
make[2]: Leaving directory `/root/Download/ntp-4.2.4p8/sntp'
make[1]: *** [install-recursive] Error 1
make[1]: Leaving directory `/root/Download/ntp-4.2.4p8'
make: *** [install] Error 2
----------------------------------------------------------------------------------------------------------------------

Please help.

Thanks
Vivek

On Dec 11, 10:07 am, "vivek8...@gmail.com" <vivek8...@gmail.com>
wrote:

Nilesh K

unread,

Dec 16, 2009, 1:00:22 AM12/16/09

to Hypertable Development

Hi,
Now we(Vivek,Me and Team) have installed NTP and successfully
synchronised clocks of connected machines. Also we are able to
distribute hypertable through cap dist. Now while trying to start, we
are getting message as "Waiting for Hypertable.RangeServer to come
up...".

Console has following output.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++
[root@localhost conf]# cap start
* executing `start'
** transaction: start
* executing `start_master'
* executing "/opt/hypertable/0.9.2.7/bin/start-dfsbroker.sh
hadoop --config=/opt/hypertable/0.9.2.7/conf/hypertable.cfg &&\\
\n /opt/hypertable/0.9.2.7/bin/start-hyperspace.sh --config=/
opt/hypertable/0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/
0.9.2.7/bin/start-master.sh --config=/opt/hypertable/0.9.2.7/conf/
hypertable.cfg;"

servers:
["192.168.2.69"]
[192.168.2.69] executing
command

** [out :: 192.168.2.69] DFS broker: available file descriptors:
1024
** [out :: 192.168.2.69] DfsBroker.hadoop appears to be running
(20333):
** [out :: 192.168.2.69] root 20333 20332 0 Dec15 pts/0 00:00:30 java
-classpath /opt/hypertable/0.9.2.7:/opt/hypertable/0.9.2.7/build/
classes:/opt/hypertable/0.9.2.7/build:/opt/hypertable/0.9.2.7/build/
test/classes:/opt/hypertable/0.9.2.7/lib/*.jar:/opt/hypertable/0.9.2.7/
lib/java/commons-logging-1.0.4.jar:/opt/hypertable/0.9.2.7/lib/java/
hadoop-0.20.1-core.jar:/opt/hypertable/0.9.2.7/lib/java/
hypertable-0.9.2.7.jar:/opt/hypertable/0.9.2.7/lib/java/
junit-4.3.1.jar:/opt/hypertable/0.9.2.7/lib/java/libthrift.jar:/opt/
hypertable/0.9.2.7/lib/java/log4j-1.2.13.jar:/opt/hypertable/0.9.2.7/
lib/jetty-ext/*.jar org.hypertable.DfsBroker.hadoop.main --
verbose
** [out :: 192.168.2.69] Hyperspace appears to be running
(20330):
** [out :: 192.168.2.69] root 20330 20329 0 Dec15 pts/0 00:00:37 /opt/
hypertable/0.9.2.7/bin/Hyperspace.Master --pidfile /opt/hypertable/
0.9.2.7/run/Hyperspace.pid --verbose
** [out :: 192.168.2.69] Started
Hypertable.Master
command
finished
* executing
`start_slaves'
* executing "/opt/hypertable/0.9.2.7/bin/random-wait.sh 5 &&\\\n /
opt/hypertable/0.9.2.7/bin/start-dfsbroker.sh hadoop --config=/
opt/hypertable/0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/
0.9.2.7/bin/start-rangeserver.sh --config=/opt/hypertable/
0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/0.9.2.7/bin/start-
thriftbroker.sh --config=/opt/hypertable/0.9.2.7/conf/
hypertable.cfg"

servers: ["192.168.2.69", "192.168.2.85",
"192.168.2.86"]

[192.168.2.69] executing
command
[192.168.2.86] executing
command
[192.168.2.85] executing
command

** [out :: 192.168.2.69] DFS broker: available file descriptors:
1024
** [out :: 192.168.2.69] DfsBroker.hadoop appears to be running
(20333):
** [out :: 192.168.2.69] root 20333 20332 0 Dec15 pts/0 00:00:30 java
-classpath /opt/hypertable/0.9.2.7:/opt/hypertable/0.9.2.7/build/
classes:/opt/hypertable/0.9.2.7/build:/opt/hypertable/0.9.2.7/build/
test/classes:/opt/hypertable/0.9.2.7/lib/*.jar:/opt/hypertable/0.9.2.7/
lib/java/commons-logging-1.0.4.jar:/opt/hypertable/0.9.2.7/lib/java/
hadoop-0.20.1-core.jar:/opt/hypertable/0.9.2.7/lib/java/
hypertable-0.9.2.7.jar:/opt/hypertable/0.9.2.7/lib/java/
junit-4.3.1.jar:/opt/hypertable/0.9.2.7/lib/java/libthrift.jar:/opt/
hypertable/0.9.2.7/lib/java/log4j-1.2.13.jar:/opt/hypertable/0.9.2.7/
lib/jetty-ext/*.jar org.hypertable.DfsBroker.hadoop.main --
verbose
** [out :: 192.168.2.86] DFS broker: available file descriptors:
1024
** [out :: 192.168.2.86] DfsBroker.hadoop appears to be running
(16585):
** [out :: 192.168.2.86] root 16585 1 0 Dec15 ? 00:00:19 java -
classpath /opt/hypertable/0.9.2.7:/opt/hypertable/0.9.2.7/build/
classes:/opt/hypertable/0.9.2.7/build:/opt/hypertable/0.9.2.7/build/
test/classes:/opt/hypertable/0.9.2.7/lib/*.jar:/opt/hypertable/0.9.2.7/
lib/java/commons-logging-1.0.4.jar:/opt/hypertable/0.9.2.7/lib/java/
hadoop-0.20.1-core.jar:/opt/hypertable/0.9.2.7/lib/java/
hypertable-0.9.2.7.jar:/opt/hypertable/0.9.2.7/lib/java/
junit-4.3.1.jar:/opt/hypertable/0.9.2.7/lib/java/libthrift.jar:/opt/
hypertable/0.9.2.7/lib/java/log4j-1.2.13.jar:/opt/hypertable/0.9.2.7/
lib/jetty-ext/*.jar org.hypertable.DfsBroker.hadoop.main --verbose --
config=/opt/hypertable/0.9.2.7/conf/hypertable.cfg
** [out :: 192.168.2.85] DFS broker: available file descriptors:
1024
** [out :: 192.168.2.85] DfsBroker.hadoop appears to be running
(9748):
** [out :: 192.168.2.85] root 9748 1 0 Dec15 ? 00:00:16 java -
classpath /opt/hypertable/0.9.2.7:/opt/hypertable/0.9.2.7/build/
classes:/opt/hypertable/0.9.2.7/build:/opt/hypertable/0.9.2.7/build/
test/classes:/opt/hypertable/0.9.2.7/lib/*.jar:/opt/hypertable/0.9.2.7/
lib/java/commons-logging-1.0.4.jar:/opt/hypertable/0.9.2.7/lib/java/
hadoop-0.20.1-core.jar:/opt/hypertable/0.9.2.7/lib/java/
hypertable-0.9.2.7.jar:/opt/hypertable/0.9.2.7/lib/java/
junit-4.3.1.jar:/opt/hypertable/0.9.2.7/lib/java/libthrift.jar:/opt/
hypertable/0.9.2.7/lib/java/log4j-1.2.13.jar:/opt/hypertable/0.9.2.7/
lib/jetty-ext/*.jar org.hypertable.DfsBroker.hadoop.main --verbose --
config=/opt/hypertable/0.9.2.7/conf/hypertable.cfg
** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come
up...
*** [err :: 192.168.2.86] /opt/hypertable/0.9.2.7/bin/ht-env.sh: line
99: 19817 Segmentation fault (core dumped) $VALGRIND
$HYPERTABLE_HOME/bin/$servercmd --pidfile $pidfile "$@" >&
$logfile
** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.85] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.69] ERROR: Hypertable.RangeServer did not come
up
** [out :: 192.168.2.86] ERROR: Hypertable.RangeServer did not come
up
** [out :: 192.168.2.85] ERROR: Hypertable.RangeServer did not come
up
command finished
failed: "sh -c '/opt/hypertable/0.9.2.7/bin/random-wait.sh 5 &&\\\n /
opt/hypertable/0.9.2.7/bin/start-dfsbroker.sh hadoop --config=/
opt/hypertable/0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/
0.9.2.7/bin/start-rangeserver.sh --config=/opt/hypertable/
0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/0.9.2.7/bin/start-
thriftbroker.sh --config=/opt/hypertable/0.9.2.7/conf/
hypertable.cfg'" on 192.168.2.69,192.168.2.85,192.168.2.86

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

and please let me know which log you will need.

Thanks,
Nilesh Katkar

Doug Judd

unread,

Dec 16, 2009, 1:04:14 AM12/16/09

to hyperta...@googlegroups.com

Try running 'cap shell' and then executing the command:

tail /opt/hypertable/0.9.2.7/log/Hypertable.RangeServer.log

and post the output.

- Doug

Nilesh K

unread,

Dec 16, 2009, 1:21:15 AM12/16/09

to Hypertable Development

Upon starting from shell the output is like this
+++++++++++++++++++++++++++++++++++++++++++++++
[root@localhost conf]# cap shell
* executing `shell'
====================================================================
Welcome to the interactive Capistrano shell! This is an experimental
feature, and is liable to change in future releases. Type 'help' for
a summary of how to use the shell.
--------------------------------------------------------------------
cap> start
[establishing connection(s) to 192.168.2.69, 192.168.2.85,
192.168.2.86]
*** [err :: 192.168.2.86] start: missing job name
*** [err :: 192.168.2.86] Try `start --help' for more information.
*** [err :: 192.168.2.85] start: missing job name
*** [err :: 192.168.2.85] Try `start --help' for more information.
*** [err :: 192.168.2.69] start: missing job name
*** [err :: 192.168.2.69] Try `start --help' for more information.
error: failed: "sh -c 'start'" on
192.168.2.69,192.168.2.85,192.168.2.86

+++++++++++++++++++++++++++++++++++++++++++++++

and RangeServer log is like this
+++++++++++++++++++++++++++++++++++++++++++++++++++++
1260941869 NOTICE Hypertable.RangeServer : (/root/src/
hypertable-0.9.2.7-alpha/src/cc/Common/Config.cc:409) Initializing
Hypertable.RangeServer (Hypertable 0.9.2.7 (tarball))...
CPU cores count=2
CephBroker.MonAddr=10.0.1.245:6789
CephBroker.Port=38030
CephBroker.Workers=20
DfsBroker.Host=localhost
DfsBroker.Local.Port=38030
DfsBroker.Local.Root=fs/local
DfsBroker.Port=38030
HdfsBroker.Port=38030
HdfsBroker.Workers=20
HdfsBroker.fs.default.name=hdfs://192.168.2.69:9000
Hyperspace.GracePeriod=200000
Hyperspace.KeepAlive.Interval=30000
Hyperspace.Lease.Interval=1000000
Hyperspace.Master.Dir=hyperspace
Hyperspace.Master.Host=192.168.2.69
Hyperspace.Master.Port=38040
Hyperspace.Master.Workers=20
Hypertable.Master.Host=192.168.2.69
Hypertable.Master.Port=38050
Hypertable.Master.Workers=20
Hypertable.RangeServer.Port=38060
Hypertable.RangeServer.Reactors=2
Hypertable.Request.Timeout=180000
Hypertable.Verbose=true
ThriftBroker.Port=38080
config=/opt/hypertable/0.9.2.7/conf/hypertable.cfg
dfs-host=localhost
dfs-port=38030
grace-period=200000
hs-host=192.168.2.69
hs-port=38040
keepalive=30000
lease-interval=1000000
master-host=192.168.2.69
master-port=38050
pidfile=/opt/hypertable/0.9.2.7/run/Hypertable.RangeServer.pid
port=38060
reactors=2
timeout=180000
verbose=true
1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-
alpha/src/cc/Hypertable/RangeServer/HyperspaceSessionHandler.cc:31)
Hyperspace session state change: SAFE
1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-
alpha/src/cc/Hypertable/RangeServer/RangeServer.cc:292) log_dir=/
hypertable/servers/192.168.2.69_38060/log
1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-
alpha/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:200) Event:
type=CONNECTION_ESTABLISHED from=192.168.2.69:38050
1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-
alpha/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:201) Stopping
maintenance queue
1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-
alpha/src/cc/AsyncComm/ConnectionManager.cc:272) Event:
type=DISCONNECT from=192.168.2.69:38050; Problem connecting to Master,
will retry in 15000 milliseconds...
1260941869 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-
alpha/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:211) Event:
type=DISCONNECT from=192.168.2.69:38050
1260941870 ERROR Hypertable.RangeServer : local_recover (/root/src/
hypertable-0.9.2.7-alpha/src/cc/Hypertable/RangeServer/RangeServer.cc:
472): Hypertable::Exception: failed expectation: nread ==
RSML_HEADER_SIZE - METALOG bad range server metalog header
at Hypertable::RangeServerMetaLogReader::RangeServerMetaLogReader
(Hypertable::Filesystem*, const Hypertable::String&) (/root/src/
hypertable-0.9.2.7-alpha/src/cc/Hypertable/Lib/
RangeServerMetaLogReader.cc:210)

+++++++++++++++++++++++++++++++++++++++++++++++++++++
Thanks,
Nilesh Katkar

On Dec 16, 11:04 am, Doug Judd <nuggetwh...@gmail.com> wrote:
> Try running 'cap shell' and then executing the command:
>
> tail /opt/hypertable/0.9.2.7/log/Hypertable.RangeServer.log
>
> and post the output.
>
> - Doug
>

> > lib/jetty-ext/*.jar<http://0.9.2.7/build:/opt/hypertable/0.9.2.7/build/%0Atest/classes:/o...>org.hypertable.DfsBroker.hadoop.main --

> > verbose
> > ** [out :: 192.168.2.86] DFS broker: available file descriptors:
> > 1024
> > ** [out :: 192.168.2.86] DfsBroker.hadoop appears to be running
> > (16585):
> > ** [out :: 192.168.2.86] root 16585 1 0 Dec15 ? 00:00:19 java -
> > classpath /opt/hypertable/0.9.2.7:/opt/hypertable/0.9.2.7/build/
> > classes:/opt/hypertable/0.9.2.7/build:/opt/hypertable/0.9.2.7/build/
> > test/classes:/opt/hypertable/0.9.2.7/lib/*.jar:/opt/hypertable/0.9.2.7/
> > lib/java/commons-logging-1.0.4.jar:/opt/hypertable/0.9.2.7/lib/java/
> > hadoop-0.20.1-core.jar:/opt/hypertable/0.9.2.7/lib/java/
> > hypertable-0.9.2.7.jar:/opt/hypertable/0.9.2.7/lib/java/
> > junit-4.3.1.jar:/opt/hypertable/0.9.2.7/lib/java/libthrift.jar:/opt/
> > hypertable/0.9.2.7/lib/java/log4j-1.2.13.jar:/opt/hypertable/0.9.2.7/

> > lib/jetty-ext/*.jar<http://0.9.2.7/build:/opt/hypertable/0.9.2.7/build/%0Atest/classes:/o...>org.hypertable.DfsBroker.hadoop.main --verbose --

> > config=/opt/hypertable/0.9.2.7/conf/hypertable.cfg
> > ** [out :: 192.168.2.85] DFS broker: available file descriptors:
> > 1024
> > ** [out :: 192.168.2.85] DfsBroker.hadoop appears to be running
> > (9748):
> > ** [out :: 192.168.2.85] root 9748 1 0 Dec15 ? 00:00:16 java -
> > classpath /opt/hypertable/0.9.2.7:/opt/hypertable/0.9.2.7/build/
> > classes:/opt/hypertable/0.9.2.7/build:/opt/hypertable/0.9.2.7/build/
> > test/classes:/opt/hypertable/0.9.2.7/lib/*.jar:/opt/hypertable/0.9.2.7/
> > lib/java/commons-logging-1.0.4.jar:/opt/hypertable/0.9.2.7/lib/java/
> > hadoop-0.20.1-core.jar:/opt/hypertable/0.9.2.7/lib/java/
> > hypertable-0.9.2.7.jar:/opt/hypertable/0.9.2.7/lib/java/
> > junit-4.3.1.jar:/opt/hypertable/0.9.2.7/lib/java/libthrift.jar:/opt/
> > hypertable/0.9.2.7/lib/java/log4j-1.2.13.jar:/opt/hypertable/0.9.2.7/

> > lib/jetty-ext/*.jar<http://0.9.2.7/build:/opt/hypertable/0.9.2.7/build/%0Atest/classes:/o...>org.hypertable.DfsBroker.hadoop.main --verbose --

> ...
>
> read more »

Doug Judd

unread,

Dec 16, 2009, 1:36:36 AM12/16/09

to hyperta...@googlegroups.com

Looks like the database was shut down uncleanly. It appears that the meta log is corrupt due to the HDFS fsync issue. You can fix the problem with the following sequence of commands:

cap stop
cap cleandb
cap start

That should clear the database and bring it back up in a working state.

- Doug

Nilesh K

unread,

Dec 16, 2009, 5:31:11 AM12/16/09

to Hypertable Development

Thank you for reply,
but still problem persists, now log for rangeserver is as
==================================

1260957750 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-

alpha/src/cc/Hypertable/RangeServer/HyperspaceSessionHandler.cc:31)
Hyperspace session state change: SAFE

1260957750 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-
alpha/src/cc/Hypertable/RangeServer/RangeServer.cc:292) log_dir=/
hypertable/servers/192.168.2.69_38060/log
1260957750 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-

alpha/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:201) Stopping
maintenance queue

1260957750 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-

alpha/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:200) Event:
type=CONNECTION_ESTABLISHED from=192.168.2.69:38050

1260957750 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-
alpha/src/cc/Hypertable/RangeServer/RangeServer.cc:812) Loading range:
{TableIdentifier: name='METADATA' id=0 generation=1} {RangeSpec:
start='' end='0:��'}
1260957750 INFO Hypertable.RangeServer : (/root/src/hypertable-0.9.2.7-
alpha/src/cc/Hypertable/RangeServer/RangeServer.cc:2138) Waiting for
recovery to complete...
1260957750 ERROR Hypertable.RangeServer : local_recover (/root/src/
hypertable-0.9.2.7-alpha/src/cc/Hypertable/RangeServer/RangeServer.cc:
472): Hypertable::Exception: Error appending 17 bytes to DFS fd 2 -
DFS BROKER i/o error
at virtual size_t Hypertable::DfsBroker::Client::append(int32_t,
Hypertable::StaticBuffer&, uint32_t) (/root/src/hypertable-0.9.2.7-
alpha/src/cc/DfsBroker/Lib/Client.cc:301)
at virtual size_t Hypertable::DfsBroker::Client::append(int32_t,
Hypertable::StaticBuffer&, uint32_t) (/root/src/hypertable-0.9.2.7-
alpha/src/cc/DfsBroker/Lib/Client.cc:290):
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /
hypertable/servers/192.168.2.69_38060/log/range_txn/0 could only be
replicated to 0
==================================
And Output on console is as like below
+++++++++++++++++++++++++++++++++++++++++++++++++

cap start
* executing `start'
** transaction: start
* executing `start_master'
* executing "/opt/hypertable/0.9.2.7/bin/start-dfsbroker.sh
hadoop --config=/opt/hypertable/0.9.2.7/conf/hypertable.cfg &&\\
\n /opt/hypertable/0.9.2.7/bin/start-hyperspace.sh --config=/
opt/hypertable/0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/
0.9.2.7/bin/start-master.sh --config=/opt/hypertable/0.9.2.7/conf/
hypertable.cfg;"
servers:
["192.168.2.69"]
[192.168.2.69] executing
command
** [out :: 192.168.2.69] DFS broker: available file descriptors:
1024

** [out :: 192.168.2.69] Started DFS Broker
(hadoop)
** [out :: 192.168.2.69] Started
Hyperspace

** [out :: 192.168.2.69] Started
Hypertable.Master
command
finished
* executing
`start_slaves'
* executing "/opt/hypertable/0.9.2.7/bin/random-wait.sh 5 &&\\\n /
opt/hypertable/0.9.2.7/bin/start-dfsbroker.sh hadoop --config=/
opt/hypertable/0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/
0.9.2.7/bin/start-rangeserver.sh --config=/opt/hypertable/
0.9.2.7/conf/hypertable.cfg &&\\\n /opt/hypertable/0.9.2.7/bin/start-
thriftbroker.sh --config=/opt/hypertable/0.9.2.7/conf/
hypertable.cfg"
servers: ["192.168.2.69", "192.168.2.85",
"192.168.2.86"]
[192.168.2.69] executing
command

[192.168.2.85] executing
command
[192.168.2.86] executing
command

** [out :: 192.168.2.85] DFS broker: available file descriptors:
1024

** [out :: 192.168.2.69] DFS broker: available file descriptors:
1024
** [out :: 192.168.2.69] DfsBroker.hadoop appears to be running

(4650):
** [out :: 192.168.2.69] root 4650 4648 8 15:32 ? 00:00:00 java -

classpath /opt/hypertable/0.9.2.7:/opt/hypertable/0.9.2.7/build/
classes:/opt/hypertable/0.9.2.7/build:/opt/hypertable/0.9.2.7/build/
test/classes:/opt/hypertable/0.9.2.7/lib/*.jar:/opt/hypertable/0.9.2.7/
lib/java/commons-logging-1.0.4.jar:/opt/hypertable/0.9.2.7/lib/java/
hadoop-0.20.1-core.jar:/opt/hypertable/0.9.2.7/lib/java/
hypertable-0.9.2.7.jar:/opt/hypertable/0.9.2.7/lib/java/
junit-4.3.1.jar:/opt/hypertable/0.9.2.7/lib/java/libthrift.jar:/opt/
hypertable/0.9.2.7/lib/java/log4j-1.2.13.jar:/opt/hypertable/0.9.2.7/

lib/jetty-ext/*.jar org.hypertable.DfsBroker.hadoop.main --verbose --
config=/opt/hypertable/0.9.2.7/conf/hypertable.cfg

** [out :: 192.168.2.86] DFS broker: available file descriptors:
1024

** [out :: 192.168.2.86] Started DFS Broker
(hadoop)

** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come
up...
*** [err :: 192.168.2.86] /opt/hypertable/0.9.2.7/bin/ht-env.sh: line

99: 3630 Segmentation fault (core dumped) $VALGRIND

$HYPERTABLE_HOME/bin/$servercmd --pidfile $pidfile "$@" >&
$logfile

** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.69] Waiting for Hypertable.RangeServer to come
up...
** [out :: 192.168.2.86] Waiting for Hypertable.RangeServer to come
up...