How to configure

[binh...@gmail.com]

Hi,

After building project I start the CLI and get this warning

>WARNING: Configuration of DataTorrent has not been complete. Please proceed with caution and only in development environment!

If I use DT-RTS I could configure it from web UI but in this case I do not have Gateway service so how can I configure it?

Thanks

-Binh

[Yogi Devendra]

Binh,

I can see similar WARNING message when I launch dtcli from my repo.

But, I am able to launch demo applications as well as my own applications using this dtcli. 

I feel that, this WARNING is harmless. Anyways it explicitly says "Please proceed" for "development environment ". 

So, we should not worry about it.

Can you launch the applications from your dtcli?

~ Yogi

[binh...@gmail.com]

Hi Yogi,

No I haven't tried it yet but I am thinking about non-development environments.

How can I configure them? Do I have to use package that provided by Datatorrent?

-Binh

[Thomas Weise]

Hey Binh,

You can download pre-built distributions from DataTorrent, the community edition has all of Apex in it and is free to use. You can also build your custom distribution from the jar files that the build executed with -Prelease produces.

Thomas

--
You received this message because you are subscribed to the Google Groups "apex-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apex-dev+u...@googlegroups.com.
To post to this group, send email to apex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/64423815-e365-474a-add2-2b025915da8b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[binh...@gmail.com]

Thank you Thomas,

I tried the community version but I can't launch PiDemo application because of this error.

ERROR com.datatorrent.stram.StreamingAppMaster: Exiting Application Master

java.lang.NullPointerException

        at org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:551)

        at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:686)

        at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:682)

        at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)

        at org.apache.hadoop.fs.FileContext.create(FileContext.java:682)

        at com.datatorrent.common.util.FSStorageAgent.save(FSStorageAgent.java:92)

        at com.datatorrent.stram.plan.physical.PhysicalPlan.initCheckpoint(PhysicalPlan.java:944)

        at com.datatorrent.stram.plan.physical.PhysicalPlan.<init>(PhysicalPlan.java:363)

        at com.datatorrent.stram.StreamingContainerManager.<init>(StreamingContainerManager.java:330)

        at com.datatorrent.stram.StreamingContainerManager.getInstance(StreamingContainerManager.java:2828)

        at com.datatorrent.stram.StreamingAppMasterService.serviceInit(StreamingAppMasterService.java:516)

        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

        at com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:98)

Do you have any idea what maybe wrong? I am running community version 3.0.0

[Thomas Weise]

I just installed the community edition on CDH 5.2 and the pi demo works.

Did you complete the installation wizzard? Did you import the pi demo or did you upload your own app package?

What's your environment (Hadoop distro, JDK, OS)?

Thomas

[Binh Nguyen Van]

Hi Thomas,

I am testing it with docker using this image https://hub.docker.com/r/sequenceiq/hadoop-docker/

This image uses CentOS with JDK 7u71. The installation wizard works fine and I completed it.

And then I go to the developer section in the console UI to upload and launch demo.

-Binh

[David Yan]

Hi Binh,

I just installed the community edition on Ubuntu 14.04 that is running Apache Hadoop 2.7.1 single node and the Pi Demo works.  

The exception you're getting seems to be HDFS-related.  When you click on the "Monitor" tab in the UI console, does it show any warnings or errors under the "Issues" heading?

David

[Binh Van Nguyen]

Hi David,

No, it does not show any warning or error at all.

Here is the screenshot

​

I think the application can write to HDFS already since the ls command shows

​

FYI: I manually created /user/dtadmin and /user/dtadmin/datatorrent and then I changed the ownership

of those directories to dtadmin:dtadmin. Not sure if that could be the cause of issue.

​

-Binh

To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/8da82c52-a623-4fc9-a1b0-3be248f7d22c%40googlegroups.com.

[David Yan]

Thanks Binh.  My guess is that it has something to do with the hadoop setup in the docker image.  

I'll try set up the docker image myself either later today or tomorrow and see if I'm able to reproduce the problem.

David

[David Yan]

Hi Binh,

I have set up the docker with hadoop-2.7.0 image that you referred to and I am getting this NPE on the namenode side upon an mkdir attempt that should result in AccessControlException instead of NPE (see below), and I don't see this with my hadoop-2.7.1 installation.  This suggests there's something wrong with HDFS in the docker image.

Can you try out the sandbox version in the mean time?  We will research this NPE problem in the namenode in the docker image.

Thanks,

David

java.lang.NullPointerException

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:247)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:227)

        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)

        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1698)

        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1682)

        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:1651)

        at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setPermission(FSDirAttrOp.java:61)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1653)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:693)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:453)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

[Binh Nguyen Van]

Hi David,

Do you get this error when you try to configure it for the first time?

If that the case then I think you need to manually create directory

and change the owner to dtadmin (I also had to do that).

-Binh

[Tushar Gosavi]

Hi Binh,

Following steps worked for me with docker image.

1) docker pull sequenceiq/hadoop-docker

2) docker run -i -v /home/tushar:/home/tushar -t sequenceiq/hadoop-docker:latest -bash

3) Create directory and change permissions.

/usr/local/hadoop/bin/hdfs dfs -mkdir /user/dtadmin

/usr/local/hadoop/bin/hdfs dfs -mkdir /user/dtadmin/datatorrent/

/usr/local/hadoop/bin/hdfs dfs -chown dtadmin:dtadmin /user/dtadmin/datatorrent/

/usr/local/hadoop/bin/hdfs dfs -chown dtadmin:dtadmin /user/dtadmin

4) Download and install datatorrent rts community release.

5) Configure through ui.

6) Restart the dtgateway manually

dtgateway restart

7) Import and run the application.

If the application fails again, then check resource manager log, you can find the log location at

/usr/local/hadoop/logs/yarn--resourcemanager*.log

Also check for any errors into application log directory located at /usr/local/hadoop/logs/userlogs/<appid>.

-Tushar.

--
You received this message because you are subscribed to the Google Groups "apex-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apex-dev+u...@googlegroups.com.
To post to this group, send email to apex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/dc6a6f18-1fce-4056-8a50-52b3126f19ff%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

“I'd have blown my top, because I want to beat this damn thing,
 as long as I've gone this far. I can't just leave it after I've found
 out so much about it. I have to keep going to find out ultimately
what is the matter with it in the end."
                Richard P. Feynman

[David Yan]

Hi Binh,

Actually, it happens on 2.7.1 as well, while 2.6.0 works fine.  There may be something in Apex that is incompatible with the HDFS implementation for 2.7.x that causes the NPE.

I have opened a ticket for that and someone will work on it.

https://malhar.atlassian.net/browse/APEX-45

In the mean time, please try Hadoop 2.6.0.

Thanks!

David

On Tue, Aug 11, 2015 at 11:46 AM, Binh Nguyen Van <binh...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "apex-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apex-dev+u...@googlegroups.com.
To post to this group, send email to apex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/dc6a6f18-1fce-4056-8a50-52b3126f19ff%40googlegroups.com.

[Binh Nguyen Van]

Hi Tushar,

I tried your way but still no luck, I still got the same NPE error.

Thanks

-Binh

To unsubscribe from this group and stop receiving emails from it, send an email to apex-dev+unsubscribe@googlegroups.com.

To post to this group, send email to apex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/dc6a6f18-1fce-4056-8a50-52b3126f19ff%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Binh Nguyen Van]

Hi David,

I tried with Hadoop 2.6.0 and it is working!

Thanks

-Binh

To unsubscribe from this group and stop receiving emails from it, send an email to apex-dev+unsubscribe@googlegroups.com.

To post to this group, send email to apex...@googlegroups.com.

[David Yan]

That's great!  You can watch the aforementioned ticket if you're interested in the progress of fixing this issue with Hadoop 2.7.x.

David

[Thomas Weise]

I'm able to install RTS 3.0.0 and run the pi demo on top of the Apache 2.7.1 binaries. This could be a problem with the docker setup.

Thomas