BUILD_REQUESTED issues

[Dave Stuart]

Hi,

Im sure this is just me but I have put a small index of 10k items into
hbase via lily. I hadn't defined a index at that point. Now I have and
want to test a full index into Solr 3.1 I am permissions errors.

I am running this command: sudo -u hdfs bin/lily-update-index -n
indexerPOC1 --build-state BUILD_REQUESTED (I have also tried su - hdfs
and run the command from there also)
then when I run bin/lily-list-indexes

I get this output

indexerPOC1
+ General state: ACTIVE
+ Update state: SUBSCRIBE_AND_LISTEN
+ Batch build state: INACTIVE
+ Queue subscription ID: IndexUpdater_indexerPOC1
+ Solr shards:
+ shard1: http://192.168.56.1:8080/solr7/core0
+ Last batch build:
+ Hadoop Job ID: failed-1306788294793
+ Submitted at: 2011-05-30T21:44:54.803+01:00
+ Success: false
+ Job state: failed to start, check logs on ubuntu-dev-1
+ Tracking URL: null
+ Map input records: null
+ Launched map tasks: null
+ Failed map tasks: null
+ Index failures: null


The only log I can find with details is the lily-server log which says

Caused by: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=root, access=WRITE, inode="":hdfs:supergroup:rwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:
203)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:
184)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:
132)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:
4976)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:
4950)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:
1971)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:
1940)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs(NameNode.java:
770)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:
1415)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:
1411)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
1115)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1409)

But I am running it as hdfs so am a little confused

Regards,

Dave

[Dave Stuart]

I have found a workaround for this problem which I'm not sure what the
long term effects downsides will be

you can update your hdfs-site.xml and add the property below. It will
disable permissions
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

[Dave Stuart]

Now im not sure whether turning permission off has caused my next
problem but here it is. Lily seems to be stuck in a BUILDING state,
how can I rest it?

I ran sudo -u hdfs bin/lily-update-index -n indexerPOC1 --build-state
BUILD_REQUESTED again and the job request was submitted but it seemed
to hang and not do anything after about 30mins of waiting I stopped
the hadoop server. When I started it back up again it states I have no
job pending but when I do a

lily-list-indexes i get the following

indexerPOC1
+ General state: ACTIVE
+ Update state: SUBSCRIBE_AND_LISTEN

+ Batch build state: BUILDING

+ Queue subscription ID: IndexUpdater_indexerPOC1
+ Solr shards:
+ shard1: http://192.168.56.1:8080/solr7/core0

+ Active batch build:
+ Hadoop Job ID: job_201105302320_0001
+ Submitted at: 2011-05-30T23:21:17.121+01:00
+ Tracking URL: http://master:50030/jobdetails.jsp?jobid=job_201105302320_0001
+ Last batch build:
+ Hadoop Job ID: failed-1306793610455
+ Submitted at: 2011-05-30T23:13:30.455+01:00

[Evert Arckens]

Hi Dave,

First of all, it should not be necessary to run the lily commands (e.g. bin/lily-update-index) with sudo -u hdfs .
Those commands should just be run logged in as the same user with which you installed started the lily server (bin/lily-server) .
Can you check that you get the same kind of permission problems when you run it like this?

When you ran it with the permissions disabled, the BUILDING also had a failure while starting. Can you check in the logs what that failure was?

Regards,
Evert Arckens.

--
Evert Arckens

http://outerthought.org/
Scalable Smart Data
Makers of Kauri, Daisy CMS and Lily

[Dave Stuart]

Hi Evert,

When I run the job as root (i know, i know) I get this error. I
commented out the dfs.permission false too.

org.apache.hadoop.ipc.Server: IPC Server listener on 8020:
readAndProcess threw exception java.lang.RuntimeException:
java.lang.NoSuchMethodException:
org.apache.hadoop.fs.permission.FsPermission$2.<init>(). Count of
bytes read: 0
java.lang.RuntimeException: java.lang.NoSuchMethodException:
org.apache.hadoop.fs.permission.FsPermission$2.<init>()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:
115)
at
org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:
53)
at
org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:
236)
at org.apache.hadoop.ipc.RPC$Invocation.readFields(RPC.java:
104)
at org.apache.hadoop.ipc.Server
$Connection.processData(Server.java:1330)
at org.apache.hadoop.ipc.Server
$Connection.processOneRpc(Server.java:1309)
at org.apache.hadoop.ipc.Server
$Connection.readAndProcess(Server.java:1209)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:
566)
at org.apache.hadoop.ipc.Server$Listener
$Reader.run(Server.java:363)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.fs.permission.FsPermission$2.<init>()
at java.lang.Class.getConstructor0(Class.java:2706)
at java.lang.Class.getDeclaredConstructor(Class.java:1985)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:
109)
... 8 more

Regards,

Dave

[Bruno Dumon]

Hi Dave,

When the job does no longer exist in Hadoop, the index should go automatically out of the building state. This might be a bug, I created a ticket that we should verify this: http://dev.outerthought.org/trac/outerthought_lilyproject/ticket/293

It could however also be that the Lily server is not running. The commands like lily-list-indexes read the information directly from ZooKeeper, so also work when Lily is not running.

On Tue, May 31, 2011 at 7:26 PM, Dave Stuart <da...@axistwelve.com> wrote:

--
Bruno Dumon
Outerthought
http://outerthought.org/