I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.
Here are some logs from master (foo is one of the tables not found):
2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
There are also a number of the following types of error logs:
2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.
Any ideas how to find and remove any references to these non-existent tables?
There seems to be some problem with the regionserver hosting the
table. Had you disabled or deleted "foo"?And try to see what "hbck"
says. And RegionAlreadyInTransitionException: is normally thrown if a
region server is asked to open or close a region but it's already
processing that region. BTW, did you find anything abnormal with your
HDFS??
Regards,
Mohammad Tariq
On Sat, Aug 11, 2012 at 2:52 AM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> Hi there
> I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.
> Here are some logs from master (foo is one of the tables not found):
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
> There are also a number of the following types of error logs:
> 2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.
> Any ideas how to find and remove any references to these non-existent tables?
> I have a few tables which show up in a "list" in the shell, but produce
> "table not found" when performing any operation on them. There is no
> reference of them in the .META. table. It seems to be resulting in some of
> the hbase services being killed every so often.
> Here are some logs from master (foo is one of the tables not found):
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485.
> state=PENDING_OPEN, ts=1344570044277,
> server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 ..
> Cannot transit it to OFFLINE.
> There are also a number of the following types of error logs:
> 2012-08-09 20:10:04,308 ERROR
> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> Received:OPEN for the
> region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are
> already trying to OPEN.
> Any ideas how to find and remove any references to these non-existent
> tables?
6 is the number of tables that appear in "list" but cannot be operated on (which btw, includes not being able to run disable/drop on them - both ops say table not found). I also just noticed "foo" does not occur in a table list, although I did create it at one point but was able to clear it from .META. when it also was reporting table not found when trying to disable/drop it. All these come from when I ^C'ed (i.e. killed) table creation when I was trying to get lzo compression working and table creation was hanging.
Is there any way to repair this? I see hbck has repair options, but I want to proceed with caution.
> Did anything disastrous happen to cluster?
> Can you try using hbck utility of HBase.
> Run: 'hbase hbck -help' to get all the available options.
> ~Anil
> On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)>wrote:
> > Hi there
> > I have a few tables which show up in a "list" in the shell, but produce
> > "table not found" when performing any operation on them. There is no
> > reference of them in the .META. table. It seems to be resulting in some of
> > the hbase services being killed every so often.
> > Here are some logs from master (foo is one of the tables not found):
Are you running a distributed cluster?
If yes, do you have localhost in /etc/hosts file?
You are getting reference to localhost in hbck output:
ERROR: Region { meta => null, hdfs =>
hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
deployed => } on HDFS, but not listed in META or deployed on any region
server
~Anil
On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <ma...@gallotta.co.za>wrote:
> 6 is the number of tables that appear in "list" but cannot be operated on
> (which btw, includes not being able to run disable/drop on them - both ops
> say table not found). I also just noticed "foo" does not occur in a table
> list, although I did create it at one point but was able to clear it from
> .META. when it also was reporting table not found when trying to
> disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> creation when I was trying to get lzo compression working and table
> creation was hanging.
> Is there any way to repair this? I see hbck has repair options, but I want
> to proceed with caution.
> On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > Hi Marco,
> > Did anything disastrous happen to cluster?
> > Can you try using hbck utility of HBase.
> > Run: 'hbase hbck -help' to get all the available options.
> > ~Anil
> > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <ma...@gallotta.co.za(mailto:
> ma...@gallotta.co.za)>wrote:
> > > Hi there
> > > I have a few tables which show up in a "list" in the shell, but produce
> > > "table not found" when performing any operation on them. There is no
> > > reference of them in the .META. table. It seems to be resulting in
> some of
> > > the hbase services being killed every so often.
> > > Here are some logs from master (foo is one of the tables not found):
> > > There are also a number of the following types of error logs:
> > > 2012-08-09 20:10:04,308 ERROR
> > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
> > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
> > > Received:OPEN for the
> > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we
> are
> > > already trying to OPEN.
> > > Any ideas how to find and remove any references to these non-existent
> > > tables?
@Anil : Good Point.
@Marco : First make sure that all the AMIs running region servers are
reachable and there is no problem in DNS resolution.(As I see you are
using AWS).
On Sat, Aug 11, 2012 at 4:00 AM, anil gupta <anilgupt...@gmail.com> wrote:
> Are you running a distributed cluster?
> If yes, do you have localhost in /etc/hosts file?
> You are getting reference to localhost in hbck output:
> ERROR: Region { meta => null, hdfs =>
> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> deployed => } on HDFS, but not listed in META or deployed on any region
> server
> ~Anil
> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <ma...@gallotta.co.za>wrote:
>> 6 is the number of tables that appear in "list" but cannot be operated on
>> (which btw, includes not being able to run disable/drop on them - both ops
>> say table not found). I also just noticed "foo" does not occur in a table
>> list, although I did create it at one point but was able to clear it from
>> .META. when it also was reporting table not found when trying to
>> disable/drop it. All these come from when I ^C'ed (i.e. killed) table
>> creation when I was trying to get lzo compression working and table
>> creation was hanging.
>> Is there any way to repair this? I see hbck has repair options, but I want
>> to proceed with caution.
>> On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>> > Hi Marco,
>> > Did anything disastrous happen to cluster?
>> > Can you try using hbck utility of HBase.
>> > Run: 'hbase hbck -help' to get all the available options.
>> > ~Anil
>> > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <ma...@gallotta.co.za(mailto:
>> ma...@gallotta.co.za)>wrote:
>> > > Hi there
>> > > I have a few tables which show up in a "list" in the shell, but produce
>> > > "table not found" when performing any operation on them. There is no
>> > > reference of them in the .META. table. It seems to be resulting in
>> some of
>> > > the hbase services being killed every so often.
>> > > Here are some logs from master (foo is one of the tables not found):
>> > > There are also a number of the following types of error logs:
>> > > 2012-08-09 20:10:04,308 ERROR
>> > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in:
>> > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to
>> org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException:
>> > > Received:OPEN for the
>> > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we
>> are
>> > > already trying to OPEN.
>> > > Any ideas how to find and remove any references to these non-existent
>> > > tables?
> Are you running a distributed cluster?
> If yes, do you have localhost in /etc/hosts file?
> You are getting reference to localhost in hbck output:
> ERROR: Region { meta => null, hdfs =>
> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> deployed => } on HDFS, but not listed in META or deployed on any region
> server
> ~Anil
> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)>wrote:
> > 6 is the number of tables that appear in "list" but cannot be operated on
> > (which btw, includes not being able to run disable/drop on them - both ops
> > say table not found). I also just noticed "foo" does not occur in a table
> > list, although I did create it at one point but was able to clear it from
> > .META. when it also was reporting table not found when trying to
> > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> > creation when I was trying to get lzo compression working and table
> > creation was hanging.
> > Is there any way to repair this? I see hbck has repair options, but I want
> > to proceed with caution.
> > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > Hi Marco,
> > > Did anything disastrous happen to cluster?
> > > Can you try using hbck utility of HBase.
> > > Run: 'hbase hbck -help' to get all the available options.
> > > ~Anil
> > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > Hi there
> > > > I have a few tables which show up in a "list" in the shell, but produce
> > > > "table not found" when performing any operation on them. There is no
> > > > reference of them in the .META. table. It seems to be resulting in
> > some of
> > > > the hbase services being killed every so often.
> > > > Here are some logs from master (foo is one of the tables not found):
On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <ma...@gallotta.co.za> wrote:
> It's not a distributed cluster. I'm not processing enough data yet. So the reference to localhost is correct.
> On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>> Are you running a distributed cluster?
>> If yes, do you have localhost in /etc/hosts file?
>> You are getting reference to localhost in hbck output:
>> ERROR: Region { meta => null, hdfs =>
>> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>> deployed => } on HDFS, but not listed in META or deployed on any region
>> server
>> ~Anil
>> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)>wrote:
>> > 6 is the number of tables that appear in "list" but cannot be operated on
>> > (which btw, includes not being able to run disable/drop on them - both ops
>> > say table not found). I also just noticed "foo" does not occur in a table
>> > list, although I did create it at one point but was able to clear it from
>> > .META. when it also was reporting table not found when trying to
>> > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
>> > creation when I was trying to get lzo compression working and table
>> > creation was hanging.
>> > Is there any way to repair this? I see hbck has repair options, but I want
>> > to proceed with caution.
>> > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>> > > Hi Marco,
>> > > Did anything disastrous happen to cluster?
>> > > Can you try using hbck utility of HBase.
>> > > Run: 'hbase hbck -help' to get all the available options.
>> > > ~Anil
>> > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
>> > > > Hi there
>> > > > I have a few tables which show up in a "list" in the shell, but produce
>> > > > "table not found" when performing any operation on them. There is no
>> > > > reference of them in the .META. table. It seems to be resulting in
>> > some of
>> > > > the hbase services being killed every so often.
>> > > > Here are some logs from master (foo is one of the tables not found):
> Could you please share your /etc/hosts file??Meantime, do a manual
> compaction and see if ti works.
> Regards,
> Mohammad Tariq
> On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)> wrote:
> > It's not a distributed cluster. I'm not processing enough data yet. So the reference to localhost is correct.
> > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > Are you running a distributed cluster?
> > > If yes, do you have localhost in /etc/hosts file?
> > > You are getting reference to localhost in hbck output:
> > > ERROR: Region { meta => null, hdfs =>
> > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > deployed => } on HDFS, but not listed in META or deployed on any region
> > > server
> > > ~Anil
> > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)>wrote:
> > > > 6 is the number of tables that appear in "list" but cannot be operated on
> > > > (which btw, includes not being able to run disable/drop on them - both ops
> > > > say table not found). I also just noticed "foo" does not occur in a table
> > > > list, although I did create it at one point but was able to clear it from
> > > > .META. when it also was reporting table not found when trying to
> > > > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> > > > creation when I was trying to get lzo compression working and table
> > > > creation was hanging.
> > > > Is there any way to repair this? I see hbck has repair options, but I want
> > > > to proceed with caution.
> > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > Hi Marco,
> > > > > Did anything disastrous happen to cluster?
> > > > > Can you try using hbck utility of HBase.
> > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > ~Anil
> > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > > Hi there
> > > > > > I have a few tables which show up in a "list" in the shell, but produce
> > > > > > "table not found" when performing any operation on them. There is no
> > > > > > reference of them in the .META. table. It seems to be resulting in
> > > > some of
> > > > > > the hbase services being killed every so often.
> > > > > > Here are some logs from master (foo is one of the tables not found):
Is it a standalone installation or pseudo-distributed?
I faced a similar problem a few days back in a distributed cluster and used
hbck -repair option. You might give it a try.
On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <donta...@gmail.com> wrote:
> Could you please share your /etc/hosts file??Meantime, do a manual
> compaction and see if ti works.
> Regards,
> Mohammad Tariq
> On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <ma...@gallotta.co.za>
> wrote:
> > It's not a distributed cluster. I'm not processing enough data yet. So
> the reference to localhost is correct.
> > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> >> Are you running a distributed cluster?
> >> If yes, do you have localhost in /etc/hosts file?
> >> You are getting reference to localhost in hbck output:
> >> ERROR: Region { meta => null, hdfs =>
> >> hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> >> deployed => } on HDFS, but not listed in META or deployed on any region
> >> server
> >> ~Anil
> >> On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <ma...@gallotta.co.za(mailto:
> ma...@gallotta.co.za)>wrote:
> >> > 6 is the number of tables that appear in "list" but cannot be
> operated on
> >> > (which btw, includes not being able to run disable/drop on them -
> both ops
> >> > say table not found). I also just noticed "foo" does not occur in a
> table
> >> > list, although I did create it at one point but was able to clear it
> from
> >> > .META. when it also was reporting table not found when trying to
> >> > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> >> > creation when I was trying to get lzo compression working and table
> >> > creation was hanging.
> >> > Is there any way to repair this? I see hbck has repair options, but I
> want
> >> > to proceed with caution.
> >> > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> >> > > Hi Marco,
> >> > > Did anything disastrous happen to cluster?
> >> > > Can you try using hbck utility of HBase.
> >> > > Run: 'hbase hbck -help' to get all the available options.
> >> > > ~Anil
> >> > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> >> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> >> > > > Hi there
> >> > > > I have a few tables which show up in a "list" in the shell, but
> produce
> >> > > > "table not found" when performing any operation on them. There is
> no
> >> > > > reference of them in the .META. table. It seems to be resulting in
> >> > some of
> >> > > > the hbase services being killed every so often.
> >> > > > Here are some logs from master (foo is one of the tables not
> found):
It's a pseudo-distributed cluster, as I plan to add more nodes as we start gathering more data.
I get the following error when running hbck -repair, and then it stalls:
12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking after fix...
Version: 0.94.0
12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from HDFS...
Exception in thread "main" java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Threa dPoolExecutor.java:1956)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:133 7)
at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:10 59)
at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java: 504)
at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck .java:304)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> Is it a standalone installation or pseudo-distributed?
> I faced a similar problem a few days back in a distributed cluster and used
> hbck -repair option. You might give it a try.
> ~Anil
> On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <donta...@gmail.com (mailto:donta...@gmail.com)> wrote:
> > Could you please share your /etc/hosts file??Meantime, do a manual
> > compaction and see if ti works.
> > Regards,
> > Mohammad Tariq
> > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)>
> > wrote:
> > > It's not a distributed cluster. I'm not processing enough data yet. So
> > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > Are you running a distributed cluster?
> > > > If yes, do you have localhost in /etc/hosts file?
> > > > You are getting reference to localhost in hbck output:
> > > > ERROR: Region { meta => null, hdfs =>
> > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > deployed => } on HDFS, but not listed in META or deployed on any region
> > > > server
> > > > ~Anil
> > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > 6 is the number of tables that appear in "list" but cannot be
> > operated on
> > > > > (which btw, includes not being able to run disable/drop on them -
> > both ops
> > > > > say table not found). I also just noticed "foo" does not occur in a
> > table
> > > > > list, although I did create it at one point but was able to clear it
> > from
> > > > > .META. when it also was reporting table not found when trying to
> > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed) table
> > > > > creation when I was trying to get lzo compression working and table
> > > > > creation was hanging.
> > > > > Is there any way to repair this? I see hbck has repair options, but I
> > want
> > > > > to proceed with caution.
> > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > > Hi Marco,
> > > > > > Did anything disastrous happen to cluster?
> > > > > > Can you try using hbck utility of HBase.
> > > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > > ~Anil
> > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > > > Hi there
> > > > > > > I have a few tables which show up in a "list" in the shell, but
> > produce
> > > > > > > "table not found" when performing any operation on them. There is
> > no
> > > > > > > reference of them in the .META. table. It seems to be resulting in
> > > > > some of
> > > > > > > the hbase services being killed every so often.
> > > > > > > Here are some logs from master (foo is one of the tables not
> > found):
> It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> gathering more data.
> I get the following error when running hbck -repair, and then it stalls:
> 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> after fix...
> Version: 0.94.0
> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> HDFS...
> Exception in thread "main" java.util.concurrent.RejectedExecutionException
> at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Threa dPoolExecutor.java:1956)
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:133 7)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:10 59)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java: 504)
> at
> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck .java:304)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > Is it a standalone installation or pseudo-distributed?
> > I faced a similar problem a few days back in a distributed cluster and
> used
> > hbck -repair option. You might give it a try.
> > ~Anil
> > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <donta...@gmail.com(mailto:
> donta...@gmail.com)> wrote:
> > > Could you please share your /etc/hosts file??Meantime, do a manual
> > > compaction and see if ti works.
> > > Regards,
> > > Mohammad Tariq
> > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <ma...@gallotta.co.za(mailto:
> ma...@gallotta.co.za)>
> > > wrote:
> > > > It's not a distributed cluster. I'm not processing enough data yet.
> So
> > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > > Are you running a distributed cluster?
> > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > You are getting reference to localhost in hbck output:
> > > > > ERROR: Region { meta => null, hdfs =>
> > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > deployed => } on HDFS, but not listed in META or deployed on any
> region
> > > > > server
> > > > > ~Anil
> > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > > 6 is the number of tables that appear in "list" but cannot be
> > > operated on
> > > > > > (which btw, includes not being able to run disable/drop on them -
> > > both ops
> > > > > > say table not found). I also just noticed "foo" does not occur
> in a
> > > table
> > > > > > list, although I did create it at one point but was able to
> clear it
> > > from
> > > > > > .META. when it also was reporting table not found when trying to
> > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
> table
> > > > > > creation when I was trying to get lzo compression working and
> table
> > > > > > creation was hanging.
> > > > > > Is there any way to repair this? I see hbck has repair options,
> but I
> > > want
> > > > > > to proceed with caution.
> > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > > > Hi Marco,
> > > > > > > Did anything disastrous happen to cluster?
> > > > > > > Can you try using hbck utility of HBase.
> > > > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > > > ~Anil
> > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > > > > Hi there
> > > > > > > > I have a few tables which show up in a "list" in the shell,
> but
> > > produce
> > > > > > > > "table not found" when performing any operation on them.
> There is
> > > no
> > > > > > > > reference of them in the .META. table. It seems to be
> resulting in
> > > > > > some of
> > > > > > > > the hbase services being killed every so often.
> > > > > > > > Here are some logs from master (foo is one of the tables not
> > > found):
Have you specified "hadoop.tmp.dir" property in your core-site.xml and
"dfs.data.dir" and "dfs.name.dir" properties in your hdfs-site.xml
files??
If not you will loose all your data along with you meta information as
Anil has said.
On Sat, Aug 11, 2012 at 5:01 AM, anil gupta <anilgupt...@gmail.com> wrote:
> Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> rebooted your machined then you will have problems.
> On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <ma...@gallotta.co.za>wrote:
>> It's a pseudo-distributed cluster, as I plan to add more nodes as we start
>> gathering more data.
>> I get the following error when running hbck -repair, and then it stalls:
>> 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
>> after fix...
>> Version: 0.94.0
>> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
>> 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
>> HDFS...
>> Exception in thread "main" java.util.concurrent.RejectedExecutionException
>> at
>> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Threa dPoolExecutor.java:1956)
>> at
>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
>> at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:133 7)
>> at
>> org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:10 59)
>> at
>> org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java: 504)
>> at
>> org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck .java:304)
>> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
>> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
>> On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>> > Is it a standalone installation or pseudo-distributed?
>> > I faced a similar problem a few days back in a distributed cluster and
>> used
>> > hbck -repair option. You might give it a try.
>> > ~Anil
>> > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <donta...@gmail.com(mailto:
>> donta...@gmail.com)> wrote:
>> > > Could you please share your /etc/hosts file??Meantime, do a manual
>> > > compaction and see if ti works.
>> > > Regards,
>> > > Mohammad Tariq
>> > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <ma...@gallotta.co.za(mailto:
>> ma...@gallotta.co.za)>
>> > > wrote:
>> > > > It's not a distributed cluster. I'm not processing enough data yet.
>> So
>> > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>> > > > > Are you running a distributed cluster?
>> > > > > If yes, do you have localhost in /etc/hosts file?
>> > > > > You are getting reference to localhost in hbck output:
>> > > > > ERROR: Region { meta => null, hdfs =>
>> > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>> > > > > deployed => } on HDFS, but not listed in META or deployed on any
>> region
>> > > > > server
>> > > > > ~Anil
>> > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>> ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>> > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
>> > > > > > 6 is the number of tables that appear in "list" but cannot be
>> > > operated on
>> > > > > > (which btw, includes not being able to run disable/drop on them -
>> > > both ops
>> > > > > > say table not found). I also just noticed "foo" does not occur
>> in a
>> > > table
>> > > > > > list, although I did create it at one point but was able to
>> clear it
>> > > from
>> > > > > > .META. when it also was reporting table not found when trying to
>> > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
>> table
>> > > > > > creation when I was trying to get lzo compression working and
>> table
>> > > > > > creation was hanging.
>> > > > > > Is there any way to repair this? I see hbck has repair options,
>> but I
>> > > want
>> > > > > > to proceed with caution.
>> > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>> > > > > > > Hi Marco,
>> > > > > > > Did anything disastrous happen to cluster?
>> > > > > > > Can you try using hbck utility of HBase.
>> > > > > > > Run: 'hbase hbck -help' to get all the available options.
>> > > > > > > ~Anil
>> > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
>> > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>> > > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
>> > > > > > > > Hi there
>> > > > > > > > I have a few tables which show up in a "list" in the shell,
>> but
>> > > produce
>> > > > > > > > "table not found" when performing any operation on them.
>> There is
>> > > no
>> > > > > > > > reference of them in the .META. table. It seems to be
>> resulting in
>> > > > > > some of
>> > > > > > > > the hbase services being killed every so often.
>> > > > > > > > Here are some logs from master (foo is one of the tables not
>> > > found):
> > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > > Is it a standalone installation or pseudo-distributed?
> > > I faced a similar problem a few days back in a distributed cluster and
> > used
> > > hbck -repair option. You might give it a try.
> > > ~Anil
> > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <donta...@gmail.com (mailto:donta...@gmail.com)(mailto:
> > donta...@gmail.com (mailto:donta...@gmail.com))> wrote:
> > > > Could you please share your /etc/hosts file??Meantime, do a manual
> > > > compaction and see if ti works.
> > > > Regards,
> > > > Mohammad Tariq
> > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>
> > > > wrote:
> > > > > It's not a distributed cluster. I'm not processing enough data yet.
> > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > > > Are you running a distributed cluster?
> > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > > You are getting reference to localhost in hbck output:
> > > > > > ERROR: Region { meta => null, hdfs =>
> > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > > deployed => } on HDFS, but not listed in META or deployed on any
> > region
> > > > > > server
> > > > > > ~Anil
> > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > > > 6 is the number of tables that appear in "list" but cannot be
> > > > operated on
> > > > > > > (which btw, includes not being able to run disable/drop on them -
> > > > both ops
> > > > > > > say table not found). I also just noticed "foo" does not occur
> > in a
> > > > table
> > > > > > > list, although I did create it at one point but was able to
> > clear it
> > > > from
> > > > > > > .META. when it also was reporting table not found when trying to
> > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
> > table
> > > > > > > creation when I was trying to get lzo compression working and
> > table
> > > > > > > creation was hanging.
> > > > > > > Is there any way to repair this? I see hbck has repair options,
> > but I
> > > > want
> > > > > > > to proceed with caution.
> > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > > > > Hi Marco,
> > > > > > > > Did anything disastrous happen to cluster?
> > > > > > > > Can you try using hbck utility of HBase.
> > > > > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > > > > ~Anil
> > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > > > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > > > > > Hi there
> > > > > > > > > I have a few tables which show up in a "list" in the shell,
> > but
> > > > produce
> > > > > > > > > "table not found" when performing any operation on them.
> > There is
> > > > no
> > > > > > > > > reference of them in the .META. table. It seems to be
> > resulting in
> > > > > > > some of
> > > > > > > > > the hbase services being killed every so often.
> > > > > > > > > Here are some logs from master (foo is one of the tables not
> > > > found):
> Have you specified "hadoop.tmp.dir" property in your core-site.xml and
> "dfs.data.dir" and "dfs.name.dir" properties in your hdfs-site.xml
> files??
> If not you will loose all your data along with you meta information as
> Anil has said.
> Regards,
> Mohammad Tariq
> On Sat, Aug 11, 2012 at 5:01 AM, anil gupta <anilgupt...@gmail.com (mailto:anilgupt...@gmail.com)> wrote:
> > Where are you storing your hdfs data? Is it /tmp? If it's /tmp and you have
> > rebooted your machined then you will have problems.
> > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)>wrote:
> > > It's a pseudo-distributed cluster, as I plan to add more nodes as we start
> > > gathering more data.
> > > I get the following error when running hbck -repair, and then it stalls:
> > > 12/08/10 16:17:27 INFO util.HBaseFsck: Sleeping 10000ms before re-checking
> > > after fix...
> > > Version: 0.94.0
> > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading regioninfos HDFS
> > > 12/08/10 16:17:37 INFO util.HBaseFsck: Loading HBase regioninfo from
> > > HDFS...
> > > Exception in thread "main" java.util.concurrent.RejectedExecutionException
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Threa dPoolExecutor.java:1956)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:133 7)
> > > at
> > > org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionDirs(HBaseFsck.java:10 59)
> > > at
> > > org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java: 504)
> > > at
> > > org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck .java:304)
> > > at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:377)
> > > at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3139)
> > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > > > > Are you running a distributed cluster?
> > > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > > > You are getting reference to localhost in hbck output:
> > > > > > > ERROR: Region { meta => null, hdfs =>
> > > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > > > deployed => } on HDFS, but not listed in META or deployed on any
> > > region
> > > > > > > server
> > > > > > > ~Anil
> > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > > > > 6 is the number of tables that appear in "list" but cannot be
> > > > > operated on
> > > > > > > > (which btw, includes not being able to run disable/drop on them -
> > > > > both ops
> > > > > > > > say table not found). I also just noticed "foo" does not occur
> > > in a
> > > > > table
> > > > > > > > list, although I did create it at one point but was able to
> > > clear it
> > > > > from
> > > > > > > > .META. when it also was reporting table not found when trying to
> > > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
> > > table
> > > > > > > > creation when I was trying to get lzo compression working and
> > > table
> > > > > > > > creation was hanging.
> > > > > > > > Is there any way to repair this? I see hbck has repair options,
> > > but I
> > > > > want
> > > > > > > > to proceed with caution.
> > > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > > > > > Hi Marco,
> > > > > > > > > Did anything disastrous happen to cluster?
> > > > > > > > > Can you try using hbck utility of HBase.
> > > > > > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > > > > > ~Anil
> > > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > > > > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > > > > > > Hi there
> > > > > > > > > > I have a few tables which show up in a "list" in the shell,
> > > but
> > > > > produce
> > > > > > > > > > "table not found" when performing any operation on them.
> > > There is
> > > > > no
> > > > > > > > > > reference of them in the .META. table. It seems to be
> > > resulting in
> > > > > > > > some of
> > > > > > > > > > the hbase services being killed every so often.
> > > > > > > > > > Here are some logs from master (foo is one of the tables not
> > > > > found):
>> > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>> > > Is it a standalone installation or pseudo-distributed?
>> > > I faced a similar problem a few days back in a distributed cluster and
>> > used
>> > > hbck -repair option. You might give it a try.
>> > > ~Anil
>> > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <donta...@gmail.com (mailto:donta...@gmail.com)(mailto:
>> > donta...@gmail.com (mailto:donta...@gmail.com))> wrote:
>> > > > Could you please share your /etc/hosts file??Meantime, do a manual
>> > > > compaction and see if ti works.
>> > > > Regards,
>> > > > Mohammad Tariq
>> > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>
>> > > > wrote:
>> > > > > It's not a distributed cluster. I'm not processing enough data yet.
>> > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>> > > > > > Are you running a distributed cluster?
>> > > > > > If yes, do you have localhost in /etc/hosts file?
>> > > > > > You are getting reference to localhost in hbck output:
>> > > > > > ERROR: Region { meta => null, hdfs =>
>> > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>> > > > > > deployed => } on HDFS, but not listed in META or deployed on any
>> > region
>> > > > > > server
>> > > > > > ~Anil
>> > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>> > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
>> > > > > > > 6 is the number of tables that appear in "list" but cannot be
>> > > > operated on
>> > > > > > > (which btw, includes not being able to run disable/drop on them -
>> > > > both ops
>> > > > > > > say table not found). I also just noticed "foo" does not occur
>> > in a
>> > > > table
>> > > > > > > list, although I did create it at one point but was able to
>> > clear it
>> > > > from
>> > > > > > > .META. when it also was reporting table not found when trying to
>> > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
>> > table
>> > > > > > > creation when I was trying to get lzo compression working and
>> > table
>> > > > > > > creation was hanging.
>> > > > > > > Is there any way to repair this? I see hbck has repair options,
>> > but I
>> > > > want
>> > > > > > > to proceed with caution.
>> > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>> > > > > > > > Hi Marco,
>> > > > > > > > Did anything disastrous happen to cluster?
>> > > > > > > > Can you try using hbck utility of HBase.
>> > > > > > > > Run: 'hbase hbck -help' to get all the available options.
>> > > > > > > > ~Anil
>> > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
>> > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>> > > > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
>> > > > > > > > > Hi there
>> > > > > > > > > I have a few tables which show up in a "list" in the shell,
>> > but
>> > > > produce
>> > > > > > > > > "table not found" when performing any operation on them.
>> > There is
>> > > > no
>> > > > > > > > > reference of them in the .META. table. It seems to be
>> > resulting in
>> > > > > > > some of
>> > > > > > > > > the hbase services being killed every so often.
>> > > > > > > > > Here are some logs from master (foo is one of the tables not
>> > > > found):
>>> > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>>> > > Is it a standalone installation or pseudo-distributed?
>>> > > I faced a similar problem a few days back in a distributed cluster and
>>> > used
>>> > > hbck -repair option. You might give it a try.
>>> > > ~Anil
>>> > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <donta...@gmail.com (mailto:donta...@gmail.com)(mailto:
>>> > donta...@gmail.com (mailto:donta...@gmail.com))> wrote:
>>> > > > Could you please share your /etc/hosts file??Meantime, do a manual
>>> > > > compaction and see if ti works.
>>> > > > Regards,
>>> > > > Mohammad Tariq
>>> > > > On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>>> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>
>>> > > > wrote:
>>> > > > > It's not a distributed cluster. I'm not processing enough data yet.
>>> > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>>> > > > > > Are you running a distributed cluster?
>>> > > > > > If yes, do you have localhost in /etc/hosts file?
>>> > > > > > You are getting reference to localhost in hbck output:
>>> > > > > > ERROR: Region { meta => null, hdfs =>
>>> > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>>> > > > > > deployed => } on HDFS, but not listed in META or deployed on any
>>> > region
>>> > > > > > server
>>> > > > > > ~Anil
>>> > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>>> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>>> > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
>>> > > > > > > 6 is the number of tables that appear in "list" but cannot be
>>> > > > operated on
>>> > > > > > > (which btw, includes not being able to run disable/drop on them -
>>> > > > both ops
>>> > > > > > > say table not found). I also just noticed "foo" does not occur
>>> > in a
>>> > > > table
>>> > > > > > > list, although I did create it at one point but was able to
>>> > clear it
>>> > > > from
>>> > > > > > > .META. when it also was reporting table not found when trying to
>>> > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
>>> > table
>>> > > > > > > creation when I was trying to get lzo compression working and
>>> > table
>>> > > > > > > creation was hanging.
>>> > > > > > > Is there any way to repair this? I see hbck has repair options,
>>> > but I
>>> > > > want
>>> > > > > > > to proceed with caution.
>>> > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
>>> > > > > > > > Hi Marco,
>>> > > > > > > > Did anything disastrous happen to cluster?
>>> > > > > > > > Can you try using hbck utility of HBase.
>>> > > > > > > > Run: 'hbase hbck -help' to get all the available options.
>>> > > > > > > > ~Anil
>>> > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
>>> > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>>> > > > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
>>> > > > > > > > > Hi there
>>> > > > > > > > > I have a few tables which show up in a "list" in the shell,
>>> > but
>>> > > > produce
>>> > > > > > > > > "table not found" when performing any operation on them.
>>> > There is
>>> > > > no
>>> > > > > > > > > reference of them in the .META. table. It seems to be
>>> > resulting in
>>> > > > > > > some of
>>> > > > > > > > > the hbase services being killed every so often.
>>> > > > > > > > > Here are some logs from master (foo is one of the tables not
>>> > > > found):
> On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)> wrote:
> > It's in /var which is persistent across reboots.
> > > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > > > > > Are you running a distributed cluster?
> > > > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > > > > You are getting reference to localhost in hbck output:
> > > > > > > > ERROR: Region { meta => null, hdfs =>
> > > > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > > > > deployed => } on HDFS, but not listed in META or deployed on any
> > > > region
> > > > > > > > server
> > > > > > > > ~Anil
> > > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > > > > > 6 is the number of tables that appear in "list" but cannot be
> > > > > > operated on
> > > > > > > > > (which btw, includes not being able to run disable/drop on them -
> > > > > > both ops
> > > > > > > > > say table not found). I also just noticed "foo" does not occur
> > > > in a
> > > > > > table
> > > > > > > > > list, although I did create it at one point but was able to
> > > > clear it
> > > > > > from
> > > > > > > > > .META. when it also was reporting table not found when trying to
> > > > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
> > > > table
> > > > > > > > > creation when I was trying to get lzo compression working and
> > > > > > > > > Is there any way to repair this? I see hbck has repair options,
> > > > but I
> > > > > > want
> > > > > > > > > to proceed with caution.
> > > > > > > > > On Friday 10 August 2012 at 2:49 PM, anil gupta wrote:
> > > > > > > > > > Hi Marco,
> > > > > > > > > > Did anything disastrous happen to cluster?
> > > > > > > > > > Can you try using hbck utility of HBase.
> > > > > > > > > > Run: 'hbase hbck -help' to get all the available options.
> > > > > > > > > > ~Anil
> > > > > > > > > > On Fri, Aug 10, 2012 at 2:22 PM, Marco Gallotta <
> > > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > > > > > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > > > > > > > Hi there
> > > > > > > > > > > I have a few tables which show up in a "list" in the shell,
> > > > but
> > > > > > produce
> > > > > > > > > > > "table not found" when performing any operation on them.
> > > > There is
> > > > > > no
> > > > > > > > > > > reference of them in the .META. table. It seems to be
> This is pretty strange. I mean everything seems to be in place, but we
> are stuck. Please make a check once if your Hdfs is in safemode.
> Regards,
> Mohammad Tariq
> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <donta...@gmail.com (mailto:donta...@gmail.com)> wrote:
> > What about fs.default.name (http://fs.default.name)?????
> > Regards,
> > Mohammad Tariq
> > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)> wrote:
> > > It's in /var which is persistent across reboots.
> > > > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
> > > > > > > > > Are you running a distributed cluster?
> > > > > > > > > If yes, do you have localhost in /etc/hosts file?
> > > > > > > > > You are getting reference to localhost in hbck output:
> > > > > > > > > ERROR: Region { meta => null, hdfs =>
> > > > > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
> > > > > > > > > deployed => } on HDFS, but not listed in META or deployed on any
> > > > > region
> > > > > > > > > server
> > > > > > > > > ~Anil
> > > > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
> > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > > > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
> > > > > > > > > > 6 is the number of tables that appear in "list" but cannot be
> > > > > > > operated on
> > > > > > > > > > (which btw, includes not being able to run disable/drop on them -
> > > > > > > both ops
> > > > > > > > > > say table not found). I also just noticed "foo" does not occur
> > > > > in a
> > > > > > > table
> > > > > > > > > > list, although I did create it at one point but was able to
> > > > > clear it
> > > > > > > from
> > > > > > > > > > .META. when it also was reporting table not found when trying to
> > > > > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
> > > > > table
> > > > > > > > > > creation when I was trying to get lzo compression working and
> > > > > > > > > > Is there any way to repair this? I see hbck has repair options,
> > > > > but I
> > > > > > > want
> > > > > > > > > > to proceed with caution.
You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
point your web browser to http://localhost:9000. It'll show all the
details of your HDFS.
> On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
>> This is pretty strange. I mean everything seems to be in place, but we
>> are stuck. Please make a check once if your Hdfs is in safemode.
>> Regards,
>> Mohammad Tariq
>> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <donta...@gmail.com (mailto:donta...@gmail.com)> wrote:
>> > What about fs.default.name (http://fs.default.name)?????
>> > Regards,
>> > Mohammad Tariq
>> > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)> wrote:
>> > > It's in /var which is persistent across reboots.
>> > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>> > > > > > Is it a standalone installation or pseudo-distributed?
>> > > > > > I faced a similar problem a few days back in a distributed cluster and
>> > > > > used
>> > > > > > hbck -repair option. You might give it a try.
>> > > > > > ~Anil
>> > > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <donta...@gmail.com (mailto:donta...@gmail.com)(mailto:
>> > > > > donta...@gmail.com (mailto:donta...@gmail.com))> wrote:
>> > > > > > > Could you please share your /etc/hosts file??Meantime, do a manual
>> > > > > > > compaction and see if ti works.
>> > > > > > > > On Friday 10 August 2012 at 3:30 PM, anil gupta wrote:
>> > > > > > > > > Are you running a distributed cluster?
>> > > > > > > > > If yes, do you have localhost in /etc/hosts file?
>> > > > > > > > > You are getting reference to localhost in hbck output:
>> > > > > > > > > ERROR: Region { meta => null, hdfs =>
>> > > > > > > > > hdfs://localhost:9000/hbase/test2/b0d4a5f294809c94fccb3d4ce10c3b23,
>> > > > > > > > > deployed => } on HDFS, but not listed in META or deployed on any
>> > > > > region
>> > > > > > > > > server
>> > > > > > > > > ~Anil
>> > > > > > > > > On Fri, Aug 10, 2012 at 3:08 PM, Marco Gallotta <
>> > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>> > > > > > > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>wrote:
>> > > > > > > > > > 6 is the number of tables that appear in "list" but cannot be
>> > > > > > > operated on
>> > > > > > > > > > (which btw, includes not being able to run disable/drop on them -
>> > > > > > > both ops
>> > > > > > > > > > say table not found). I also just noticed "foo" does not occur
>> > > > > in a
>> > > > > > > table
>> > > > > > > > > > list, although I did create it at one point but was able to
>> > > > > clear it
>> > > > > > > from
>> > > > > > > > > > .META. when it also was reporting table not found when trying to
>> > > > > > > > > > disable/drop it. All these come from when I ^C'ed (i.e. killed)
>> > > > > table
>> > > > > > > > > > creation when I was trying to get lzo compression working and
>> > > > > > > > > > Is there any way to repair this? I see hbck has repair options,
>> > > > > but I
>> > > > > > > want
>> > > > > > > > > > to proceed with caution.
On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <donta...@gmail.com> wrote:
> You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
> point your web browser to http://localhost:9000. It'll show all the
> details of your HDFS.
> Regards,
> Mohammad Tariq
> On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <ma...@gallotta.co.za>
wrote:
>> How do you check that?
>> On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
>>> This is pretty strange. I mean everything seems to be in place, but we
>>> are stuck. Please make a check once if your Hdfs is in safemode.
>>> Regards,
>>> Mohammad Tariq
>>> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <donta...@gmail.com(mailto:
donta...@gmail.com)> wrote:
>>> > What about fs.default.name (http://fs.default.name)?????
>>> > Regards,
>>> > Mohammad Tariq
>>> > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <ma...@gallotta.co.za(mailto:
ma...@gallotta.co.za)> wrote:
>>> > > It's in /var which is persistent across reboots.
>>> > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>>> > > > > > Is it a standalone installation or pseudo-distributed?
>>> > > > > > I faced a similar problem a few days back in a distributed
cluster and
>>> > > > > used
>>> > > > > > hbck -repair option. You might give it a try.
>>> > > > > > ~Anil
>>> > > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <
>>> > > > > > > > > > 6 is the number of tables that appear in "list" but
cannot be
>>> > > > > > > operated on
>>> > > > > > > > > > (which btw, includes not being able to run
>>> > > > > > > both ops
>>> > > > > > > > > > say table not found). I also just noticed "foo" does
not occur
>>> > > > > in a
>>> > > > > > > table
>>> > > > > > > > > > list, although I did create it at one point but was
able to
>>> > > > > clear it
>>> > > > > > > from
>>> > > > > > > > > > .META. when it also was reporting table not found
when trying to
>>> > > > > > > > > > disable/drop it. All these come from when I ^C'ed
(i.e. killed)
>>> > > > > table
>>> > > > > > > > > > creation when I was trying to get lzo compression
working and
> Name: 127.0.0.1:50010
> Decommission Status : Normal
> Configured Capacity: 31111143424 (28.97 GB)
> DFS Used: 510435328 (486.79 MB)
> Non DFS Used: 25801388032 (24.03 GB)
> DFS Remaining: 4799320064(4.47 GB)
> DFS Used%: 1.64%
> DFS Remaining%: 15.43%
> Last contact: Sat Aug 11 05:19:18 IST 2012
> See the line in red color.
> Regards,
> Mohammad Tariq
> On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <donta...@gmail.com (mailto:donta...@gmail.com)> wrote:
> > You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
> > point your web browser to http://localhost:9000. It'll show all the
> > details of your HDFS.
> > Regards,
> > Mohammad Tariq
> > On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)>
> wrote:
> > > How do you check that?
> > > On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
> > > > This is pretty strange. I mean everything seems to be in place, but we
> > > > are stuck. Please make a check once if your Hdfs is in safemode.
> > > > Regards,
> > > > Mohammad Tariq
> > > > On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <donta...@gmail.com (mailto:donta...@gmail.com)(mailto:
> donta...@gmail.com (mailto:donta...@gmail.com))> wrote:
> > > > > What about fs.default.name (http://fs.default.name)?????
> > > > > Regards,
> > > > > Mohammad Tariq
> > > > > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))> wrote:
> > > > > > It's in /var which is persistent across reboots.
> > > > > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
> > > > > > > Where are you storing your hdfs data? Is it /tmp? If it's /tmp
> and you have
> > > > > > > rebooted your machined then you will have problems.
> > > > > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <
> ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)>wrote:
> > > > > > > > It's a pseudo-distributed cluster, as I plan to add more nodes
> as we start
> > > > > > > > gathering more data.
> > > > > > > > I get the following error when running hbck -repair, and then
> it stalls:
Yeah, I feel the same. I am still in the learning phase, so it is quite
possible that I might be missing something important. But we have several
experts on the list and I hope they post their response for you.
Regards,
Mohammad Tariq
On Sat, Aug 11, 2012 at 5:30 AM, Marco Gallotta <ma...@gallotta.co.za>wrote:
> > Name: 127.0.0.1:50010
> > Decommission Status : Normal
> > Configured Capacity: 31111143424 (28.97 GB)
> > DFS Used: 510435328 (486.79 MB)
> > Non DFS Used: 25801388032 (24.03 GB)
> > DFS Remaining: 4799320064(4.47 GB)
> > DFS Used%: 1.64%
> > DFS Remaining%: 15.43%
> > Last contact: Sat Aug 11 05:19:18 IST 2012
> > See the line in red color.
> > Regards,
> > Mohammad Tariq
> > On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <donta...@gmail.com(mailto:
> donta...@gmail.com)> wrote:
> > > You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
> > > point your web browser to http://localhost:9000. It'll show all the
> > > details of your HDFS.
> > > Regards,
> > > Mohammad Tariq
> > > On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <ma...@gallotta.co.za(mailto:
> ma...@gallotta.co.za)>
> > wrote:
> > > > How do you check that?
> > > > On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
> > > > > This is pretty strange. I mean everything seems to be in place,
> but we
> > > > > are stuck. Please make a check once if your Hdfs is in safemode.
> > > > > Regards,
> > > > > Mohammad Tariq
> > > > > On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <
> donta...@gmail.com (mailto:donta...@gmail.com)(mailto:
> > donta...@gmail.com (mailto:donta...@gmail.com))> wrote:
> > > > > > What about fs.default.name (http://fs.default.name)?????
> > > > > > Regards,
> > > > > > Mohammad Tariq
> > > > > > On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <
> ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))> wrote:
> > > > > > > It's in /var which is persistent across reboots.
> > > > > > > On Friday 10 August 2012 at 4:31 PM, anil gupta wrote:
> > > > > > > > Where are you storing your hdfs data? Is it /tmp? If it's
> /tmp
> > and you have
> > > > > > > > rebooted your machined then you will have problems.
> > > > > > > > On Fri, Aug 10, 2012 at 4:19 PM, Marco Gallotta <
> > ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)>wrote:
> > > > > > > > > It's a pseudo-distributed cluster, as I plan to add more
> nodes
> > as we start
> > > > > > > > > gathering more data.
> > > > > > > > > I get the following error when running hbck -repair, and
> then
> > it stalls:
> > > > > > > > > On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
> > > > > > > > > > Is it a standalone installation or pseudo-distributed?
> > > > > > > > > > I faced a similar problem a few days back in a
> distributed
> > cluster and
> > > > > > > > > used
> > > > > > > > > > hbck -repair option. You might give it a try.
> > > > > > > > > > ~Anil
> > > > > > > > > > On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <
> > donta...@gmail.com (mailto:donta...@gmail.com)(mailto:
> > > > > > > > > donta...@gmail.com (mailto:donta...@gmail.com))> wrote:
> > > > > > > > > > > Could you please share your /etc/hosts file??Meantime,
> do a
> > manual
> > > > > > > > > > > compaction and see if ti works.
Can you try to reboot the machine and run repair again. Might not sound logical but I would give it a shot.
PS: My personal experience is that hbase and hadoop has never been reliable in my standalone environment. I always trust the distributed cluster environment. AFAIK, these things are tested extensively in distributed mode.
Best Regards,
Anil
On Aug 10, 2012, at 5:00 PM, Marco Gallotta <ma...@gallotta.co.za> wrote:
>> Name: 127.0.0.1:50010
>> Decommission Status : Normal
>> Configured Capacity: 31111143424 (28.97 GB)
>> DFS Used: 510435328 (486.79 MB)
>> Non DFS Used: 25801388032 (24.03 GB)
>> DFS Remaining: 4799320064(4.47 GB)
>> DFS Used%: 1.64%
>> DFS Remaining%: 15.43%
>> Last contact: Sat Aug 11 05:19:18 IST 2012
>> See the line in red color.
>> Regards,
>> Mohammad Tariq
>> On Sat, Aug 11, 2012 at 5:20 AM, Mohammad Tariq <donta...@gmail.com (mailto:donta...@gmail.com)> wrote:
>>> You can use "bin/hadoop dfsadmin -report" to do that. Alternatively
>>> point your web browser to http://localhost:9000. It'll show all the
>>> details of your HDFS.
>>> Regards,
>>> Mohammad Tariq
>>> On Sat, Aug 11, 2012 at 5:16 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)>
>> wrote:
>>>> How do you check that?
>>>> On Friday 10 August 2012 at 4:44 PM, Mohammad Tariq wrote:
>>>>> This is pretty strange. I mean everything seems to be in place, but we
>>>>> are stuck. Please make a check once if your Hdfs is in safemode.
>>>>> Regards,
>>>>> Mohammad Tariq
>>>>> On Sat, Aug 11, 2012 at 5:13 AM, Mohammad Tariq <donta...@gmail.com (mailto:donta...@gmail.com)(mailto:
>> donta...@gmail.com (mailto:donta...@gmail.com))> wrote:
>>>>>> What about fs.default.name (http://fs.default.name)?????
>>>>>> Regards,
>>>>>> Mohammad Tariq
>>>>>> On Sat, Aug 11, 2012 at 5:10 AM, Marco Gallotta <ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>> ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))> wrote:
>>>>>>> It's in /var which is persistent across reboots.
>>>>>>> --
>>>>>>> Marco Gallotta | Mountain View, California
>>>>>>> Software Engineer, Infrastructure | Loki Studios
>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
>>>>>>>>> On Friday 10 August 2012 at 4:09 PM, anil gupta wrote:
>>>>>>>>>> Is it a standalone installation or pseudo-distributed?
>>>>>>>>>> I faced a similar problem a few days back in a distributed
>> cluster and
>>>>>>>>> used
>>>>>>>>>> hbck -repair option. You might give it a try.
>>>>>>>>>> ~Anil
>>>>>>>>>> On Fri, Aug 10, 2012 at 3:39 PM, Mohammad Tariq <
>> donta...@gmail.com (mailto:donta...@gmail.com)(mailto:
>>>>>>>>> donta...@gmail.com (mailto:donta...@gmail.com))> wrote:
>>>>>>>>>>> Could you please share your /etc/hosts file??Meantime, do a
>> manual
>>>>>>>>>>> compaction and see if ti works.
>>>>>>>>>>> Regards,
>>>>>>>>>>> Mohammad Tariq
>>>>>>>>>>> On Sat, Aug 11, 2012 at 4:07 AM, Marco Gallotta <
>> ma...@gallotta.co.za (mailto:ma...@gallotta.co.za)(mailto:
>>>>>>>>> ma...@gallotta.co.za (mailto:ma...@gallotta.co.za))>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> It's not a distributed cluster. I'm not processing enough
>> data yet.
>>>>>>>>> So
>>>>>>>>>>> the reference to localhost is correct.
>>>>>>>>>>>> --
>>>>>>>>>>>> Marco Gallotta | Mountain View, California
>>>>>>>>>>>> Software Engineer, Infrastructure | Loki Studios
>>>>>>>>>>>> fb.me/marco.gallotta (http://fb.me/marco.gallotta) |
----- Original Message -----
From: Marco Gallotta <ma...@gallotta.co.za>
To: u...@hbase.apache.org
Cc: Sent: Friday, August 10, 2012 2:22 PM
Subject: Table listed in "list", but not in .META.
Hi there
I have a few tables which show up in a "list" in the shell, but produce "table not found" when performing any operation on them. There is no reference of them in the .META. table. It seems to be resulting in some of the hbase services being killed every so often.
Here are some logs from master (foo is one of the tables not found):
2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. state=PENDING_OPEN, ts=1344570044277, server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. Cannot transit it to OFFLINE.
There are also a number of the following types of error logs:
2012-08-09 20:10:04,308 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are already trying to OPEN.
Any ideas how to find and remove any references to these non-existent tables?
