RangeServer/DfsBroker failled to come up after restart

[Swapnil]

After a restart to hypertable cluster which is running on hdfs, Dfsbroker is failling to come up on 2 servers out of 6. When looking at the logs we found it's failing due to not able to find files.
One thing we did unusual is, restarting the cluster before all the rangeserver came up .

Below are Dfsbroker and Rangeserver logs:

DfsBroker Log:
-----------------------
INFO: Opening file '/hypertable/tables/2/0/permission/YkZf1M3gXM727K0n/cs26' flags=1 bs=0 handle = 82
Aug 14, 2012 1:40:25 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Length
INFO: Getting length of file '/hypertable/tables/2/0/permission/YkZf1M3gXM727K0n/cs27' (accurate: false)
Aug 14, 2012 1:40:25 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Open
INFO: Opening file '/hypertable/tables/2/0/permission/YkZf1M3gXM727K0n/cs27' flags=1 bs=0 handle = 83
Aug 14, 2012 1:40:25 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Length
INFO: Getting length of file '/hypertable/tables/2/0/profile/YkZf1M3gXM727K0n/cs115' (accurate: false)
Aug 14, 2012 1:40:25 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Length
SEVERE: File not found: /hypertable/tables/2/0/profile/YkZf1M3gXM727K0n/cs115
Aug 14, 2012 1:40:25 AM org.hypertable.DfsBroker.hadoop.ConnectionHandler handle
INFO: [/10.20.42.43:49686 ; Tue Aug 14 01:40:25 UTC 2012] Disconnect - COMM broken connection : Closing all open handles from /10.20.42.43:49686
Closed 0 input streams and 3 output streams for client connection /10.20.42.43:49686

RangeServer Log:
-------------------------
1344908425 INFO Hypertable.RangeServer : (/home/herbert/src-hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1407) Successfully created scanner (id=0) on table '0/0', returning 10 k/v pairs, more=0
1344908425 INFO Hypertable.RangeServer : (/home/herbert/src-hypertable/src/cc/Hypertable/RangeServer/Range.cc:269) Loading CellStore 2/0/permission/YkZf1M3gXM727K0n/cs27
1344908425 INFO Hypertable.RangeServer : (/home/herbert/src-hypertable/src/cc/Hypertable/RangeServer/Range.cc:269) Loading CellStore 2/0/profile/YkZf1M3gXM727K0n/cs115
1344908425 FATAL Hypertable.RangeServer : (/home/herbert/src-hypertable/src/cc/Hypertable/RangeServer/Range.cc:282) Problem opening CellStore file '2/0/profile/YkZf1M3gXM727K0n/cs115' - DFS BROKER file not found

The file  /hypertable/tables/2/0/profile/YkZf1M3gXM727K0n/cs115 is not present on hdfs.

What can be done to resolve this issue?

Thanks,
Swapnil

[Christoph Rupp]

this looks like your hdfs encountered a data loss.

What you can do is instructing the RangeServer to skip files that do not exist, instead of terminating immediately. To do that, add
Hypertable.RangeServer.CellStore.SkipNotFound=true
to your hypertable.cfg, then start the RangeServer.

Bye
Christoph


2012/8/14 Swapnil <swapnil...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Hypertable Development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/hypertable-dev/-/0BgfdIqvEVsJ.
To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-de...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.

[herb]

Christoph-

How would one go about deleting a range that landed in this data loss?
Ideally we could do this wholesale as we run into these CellStores.

On Aug 13, 9:17 pm, Christoph Rupp <ch...@hypertable.com> wrote:
> this looks like your hdfs encountered a data loss.
>
> What you can do is instructing the RangeServer to skip files that do not
> exist, instead of terminating immediately. To do that, add
> Hypertable.RangeServer.CellStore.SkipNotFound=true
> to your hypertable.cfg, then start the RangeServer.
>
> Bye
> Christoph
>

> 2012/8/14 Swapnil <swapnil.tai...@gmail.com>

>
>
>
>
>
>
>
> > After a restart to hypertable cluster which is running on hdfs, Dfsbroker
> > is failling to come up on 2 servers out of 6. When looking at the logs we
> > found it's failing due to not able to find files.
> > One thing we did unusual is, restarting the cluster before all the
> > rangeserver came up .
>
> > Below are Dfsbroker and Rangeserver logs:
>
> > DfsBroker Log:
> > -----------------------
> > INFO: Opening file
> > '/hypertable/tables/2/0/permission/YkZf1M3gXM727K0n/cs26' flags=1 bs=0
> > handle = 82
> > Aug 14, 2012 1:40:25 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Length
> > INFO: Getting length of file
> > '/hypertable/tables/2/0/permission/YkZf1M3gXM727K0n/cs27' (accurate: false)
> > Aug 14, 2012 1:40:25 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Open
> > INFO: Opening file
> > '/hypertable/tables/2/0/permission/YkZf1M3gXM727K0n/cs27' flags=1 bs=0
> > handle = 83
> > Aug 14, 2012 1:40:25 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Length
> > INFO: Getting length of file
> > '/hypertable/tables/2/0/profile/YkZf1M3gXM727K0n/cs115' (accurate: false)
> > Aug 14, 2012 1:40:25 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Length

> > *SEVERE: File not found:
> > /hypertable/tables/2/0/profile/YkZf1M3gXM727K0n/cs115*

> > Aug 14, 2012 1:40:25 AM org.hypertable.DfsBroker.hadoop.ConnectionHandler
> > handle
> > INFO: [/10.20.42.43:49686 ; Tue Aug 14 01:40:25 UTC 2012] Disconnect -
> > COMM broken connection : Closing all open handles from /10.20.42.43:49686
> > Closed 0 input streams and 3 output streams for client connection /
> > 10.20.42.43:49686
>
> > RangeServer Log:
> > -------------------------
> > 1344908425 INFO Hypertable.RangeServer :
> > (/home/herbert/src-hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc: 1407)
> > Successfully created scanner (id=0) on table '0/0', returning 10 k/v pairs,
> > more=0
> > 1344908425 INFO Hypertable.RangeServer :
> > (/home/herbert/src-hypertable/src/cc/Hypertable/RangeServer/Range.cc:269)
> > Loading CellStore 2/0/permission/YkZf1M3gXM727K0n/cs27
> > 1344908425 INFO Hypertable.RangeServer :
> > (/home/herbert/src-hypertable/src/cc/Hypertable/RangeServer/Range.cc:269)
> > Loading CellStore 2/0/profile/YkZf1M3gXM727K0n/cs115
> > 1344908425 FATAL Hypertable.RangeServer :

> > (/home/herbert/src-hypertable/src/cc/Hypertable/RangeServer/Range.cc:282)*P roblem opening CellStore file '2/0/profile/YkZf1M3gXM727K0n/cs115' - DFS
> > BROKER file not found*