Having trouble with alluxio,please help?

[Sambhu R]

I have been trying to run the 'Hadoop wordcount' example with alluxio following this documentaion--> "http://www.alluxio.org/docs/master/en/Running-Hadoop-MapReduce-on-Alluxio.html" and i am facing some issues,

when i execute the command: hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount -libjars /home/sambhu/Alluxio/alluxio-1.2.0/core/client/target/original-alluxio-core-client-1.2.0-jar-with-dependencies.jar alluxio://localhost:19998/home/sambhu/hadoopneww/Word_Count_input.txt alluxio://localhost:19998/wordcount/output

I am getting this:

16/08/05 11:35:09 INFO logger.type: initialize(alluxio://localhost:19998/home/sambhu/hadoopneww/Word_Count_input.txt, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Alluxio: alluxio://localhost:19998/home/sambhu/hadoopneww/Word_Count_input.txt
16/08/05 11:35:09 INFO logger.type: alluxio://localhost:19998 alluxio://localhost:19998
16/08/05 11:35:09 INFO logger.type: Loading Alluxio properties from Hadoop configuration: {}
16/08/05 11:35:10 INFO logger.type: getWorkingDirectory: /
16/08/05 11:35:11 INFO logger.type: getFileStatus(alluxio://localhost:19998/wordcount/output)
16/08/05 11:35:11 INFO logger.type: Alluxio client (version 1.2.0) is trying to connect with FileSystemMasterClient master @ localhost/127.0.0.1:19998
16/08/05 11:35:11 INFO logger.type: Client registered with FileSystemMasterClient master @ localhost/127.0.0.1:19998
16/08/05 11:35:11 INFO logger.type: getFileStatus(alluxio://localhost:19998/home/sambhu/hadoopneww/Word_Count_input.txt)
16/08/05 11:35:11 INFO input.FileInputFormat: Total input paths to process : 1
16/08/05 11:35:12 INFO mapred.JobClient: Running job: job_201608031240_0017
16/08/05 11:35:12 INFO mapred.JobClient: Job complete: job_201608031240_0017
16/08/05 11:35:12 INFO mapred.JobClient: Counters: 0

what does this mean? Is the job successful ?

and after that when i run: ./bin/alluxio fs ls /wordcount/output
I'm getting an error like this: Path /wordcount/output does not exist

What is happening here? Someone please help I'm new to Alluxio.....

[Calvin Jia]

Hi,

Although there is no error reported, I don't think the job was successful. Could you check your jobtracker for the mapper logs? There should be some errors in there which need to be fixed before the job will succeed. 

Hope this helps,

Calvin

[Sambhu R]

Thank You for the reply i was trying to connect to http://localhost:19999/home, but i am getting the following error.

HTTP ERROR 500

Problem accessing /home. Reason:

    Server Error

Caused by:

java.net.ConnectException: Call From sambhu/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
	at org.apache.hadoop.ipc.Client.call(Client.java:1472)
	at org.apache.hadoop.ipc.Client.call(Client.java:1399)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
	at $Proxy14.getFsStats(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getStats(ClientNamenodeProtocolTranslatorPB.java:593)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at $Proxy15.getStats(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getDiskStatus(DFSClient.java:2360)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getDiskStatus(DistributedFileSystem.java:946)
	at alluxio.underfs.hdfs.HdfsUnderFileSystem.getSpace(HdfsUnderFileSystem.java:258)
	at alluxio.web.WebInterfaceGeneralServlet.populateValues(WebInterfaceGeneralServlet.java:195)
	at alluxio.web.WebInterfaceGeneralServlet.doGet(WebInterfaceGeneralServlet.java:125)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:556)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:369)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
	at org.apache.hadoop.ipc.Client.call(Client.java:1438)
	... 43 more

Caused by:

java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
	at org.apache.hadoop.ipc.Client.call(Client.java:1438)
	at org.apache.hadoop.ipc.Client.call(Client.java:1399)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
	at $Proxy14.getFsStats(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getStats(ClientNamenodeProtocolTranslatorPB.java:593)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at $Proxy15.getStats(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.getDiskStatus(DFSClient.java:2360)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getDiskStatus(DistributedFileSystem.java:946)
	at alluxio.underfs.hdfs.HdfsUnderFileSystem.getSpace(HdfsUnderFileSystem.java:258)
	at alluxio.web.WebInterfaceGeneralServlet.populateValues(WebInterfaceGeneralServlet.java:195)
	at alluxio.web.WebInterfaceGeneralServlet.doGet(WebInterfaceGeneralServlet.java:125)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:556)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:369)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:722)

---

Powered by Jetty://

[Calvin Jia]

Hi,

This error means Alluxio cannot connect with HDFS. Could you make sure HDFS is running on your machine and listening on port 9000 for ipc requests?

Thanks,

Calvin

[Sambhu R]

the error is fixed now it up and running... But i still couldn't find any logs..

and what does this(mentioned in bold) means "hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount -libjars /home/sambhu/alluxio-1.2.0/core/client/target/original-alluxio-core-client-1.2.0-jar-with-dependencies.jar alluxio://localhost:19998/wordcount/input.txt alluxio://localhost:19998/wordcount/output"

how can i view alluxio input/output folder? Pardon me its kinda basic question, I'm completely noob to this....

[Calvin Jia]

Hi,

You will probably see the logs for the job tracker in your Hadoop folder.

For the lines in bold, it means to use alluxio://localhost:19998/wordcount/input.txt as the input file for the job and alluxio://localhost:19998/wordcount/output as the output folder. You can view the input file in alluxio through the web ui on port 19999 on your master node.

Hope this helps,

Calvin

[Calvin Jia]

Hi,

Were you able to find the cause of your problem?

Thanks,

Calvin

[Sambhu R]

yes the problem was i didn't load the file into alluxio file system..... the file was on my local machine... It wokerd after loading he input files to alluxio file system using 'Copy' command.

[Calvin Jia]

Glad you were able to solve it, and thanks for posting your solution!