dt-site.xml configuration

[Guilherme Hott]

Hi guys, what should I put on the `dt.dfsRootDirectory` of the `dt-site.xml`? Should be the same of the `dfs.name.dir` on the `../hadoop/conf/hdfs-site.xml`?

[Sanjay Pujare]

Guilherme

This is explained in https://github.com/apache/apex-core/blob/master/docs/security.md . Pls see the section titled "Application Root Directory under HDFS"

On Thu, Jul 20, 2017 at 7:00 AM, Guilherme Hott <guilhe...@gmail.com> wrote:

Hi guys, what should I put on the `dt.dfsRootDirectory` of the `dt-site.xml`? Should be the same of the `dfs.name.dir` on the `../hadoop/conf/hdfs-site.xml`?

--
You received this message because you are subscribed to the Google Groups "DataTorrent Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dt-users+unsubscribe@googlegroups.com.
To post to this group, send email to dt-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dt-users/b7baddf9-009d-4b71-9e1e-49f7f3e4296f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Guilherme Hott]

Thank you. I read and understand that it doesn't matter which directory is since that exists. So, could I define any directory that will work? Because I tested defining `/home/hadoop/datatorrent` and worked and later I changed for `/opt/datatorrent/current` and still working. But even whit that sometimes I had the error FileNotFoundException on the checkpoints.

--

Guilherme Hott

Software Engineer

Skype: guilhermehott

@guilhermehott

https://www.linkedin.com/in/guilhermehott

[Sanjay Pujare]

Yes, it should not matter what directory you use as long as it is consistent for all your invocations i.e. when you restart a failed application it should have the same dfs root dir as the original invocation.

For your FileNotFoundException on checkpoints I remember the path was something like /mnt/namenode/apps/... and I am not sure where it came from. Can you enable debug logging for your apps so we will see more debug messages in the log?

To view this discussion on the web visit https://groups.google.com/d/msgid/dt-users/CALQs22_1CBm%2BqfpLZd9gX0A1tNoqfqqt4pvAYZhttGEbbR8OHg%40mail.gmail.com.

[Guilherme Hott]

Here the logs.

On Thursday, July 20, 2017 at 2:33:12 AM UTC-7, Sanjay Pujare wrote:

Yes, it should not matter what directory you use as long as it is consistent for all your invocations i.e. when you restart a failed application it should have the same dfs root dir as the original invocation.

For your FileNotFoundException on checkpoints I remember the path was something like /mnt/namenode/apps/... and I am not sure where it came from. Can you enable debug logging for your apps so we will see more debug messages in the log?

On Thu, Jul 20, 2017 at 8:25 AM, Guilherme Hott <guilhe...@gmail.com> wrote:

Thank you. I read and understand that it doesn't matter which directory is since that exists. So, could I define any directory that will work? Because I tested defining `/home/hadoop/datatorrent` and worked and later I changed for `/opt/datatorrent/current` and still working. But even whit that sometimes I had the error FileNotFoundException on the checkpoints.

On Wed, Jul 19, 2017 at 7:21 PM, Sanjay Pujare <san...@datatorrent.com> wrote:

Guilherme

This is explained in https://github.com/apache/apex-core/blob/master/docs/security.md . Pls see the section titled "Application Root Directory under HDFS"

On Thu, Jul 20, 2017 at 7:00 AM, Guilherme Hott <guilhe...@gmail.com> wrote:

Hi guys, what should I put on the `dt.dfsRootDirectory` of the `dt-site.xml`? Should be the same of the `dfs.name.dir` on the `../hadoop/conf/hdfs-site.xml`?

--
You received this message because you are subscribed to the Google Groups "DataTorrent Users Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dt-users+u...@googlegroups.com.

To post to this group, send email to dt-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dt-users/b7baddf9-009d-4b71-9e1e-49f7f3e4296f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Guilherme Hott

Software Engineer

Skype: guilhermehott

@guilhermehott

https://www.linkedin.com/in/guilhermehott

--
You received this message because you are subscribed to the Google Groups "DataTorrent Users Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dt-users+u...@googlegroups.com.

[Sanjay Pujare]

Is this the FileNotFoundException issue or are we looking at a different problem? It seems all your container launches are failing with some basic problem as below. Did you change something such as adding some extra options that haven't been tried before? Also do you have dt.log that you can share?

2017-07-20 19:39:03,245 [main] INFO  stram.LaunchContainerRunnable run - Launching on node: ip-172-30-3-77.ec2.internal:8041 command: $JAVA_HOME/bin/java -agentlib:jdwp=transport=dt_so

cket,server=y,suspend=n  -Xmx1031798784  -Ddt.attr.APPLICATION_PATH=hdfs://ip-172-30-3-222.ec2.internal:8020/mnt/namenode/apps/application_1500509791454_0010 -Djava.io.tmpdir=$PWD/tmp  -Ddt.cid=container_1500509791454_0010_01_000026 -Dhadoop.root.logger=DEBUG,RFA -Dhadoop.log.dir=<LOG_DIR> -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO,com.clout.*:DEBUG com.datatorrent.stram.engine.StreamingContainer 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr

2017-07-20 19:39:03,245 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #4] INFO  impl.NMClientAsyncImpl run - Processing Event EventType: START_CONTAINER for Container container_1500509791454_0010_01_000026

2017-07-20 19:39:03,246 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #4] INFO  impl.ContainerManagementProtocolProxy newProxy - Opening proxy : ip-172-30-3-77.ec2.internal:8041

2017-07-20 19:39:05,254 [main] INFO  stram.StreamingAppMasterService execute - Completed containerId=container_1500509791454_0010_01_000026, state=COMPLETE, exitStatus=1, diagnostics=Exception from container-launch.

Container id: container_1500509791454_0010_01_000026

Exit code: 1

Stack trace: ExitCodeException exitCode=1:

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)

        at org.apache.hadoop.util.Shell.run(Shell.java:479)

        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)

        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:748)

Container exited with a non-zero exit code 1

To unsubscribe from this group and stop receiving emails from it, send an email to dt-users+unsubscribe@googlegroups.com.

To post to this group, send email to dt-u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dt-users/ae9a731a-7300-4bed-bbef-e5aafbe40d7f%40googlegroups.com.

[Guilherme Hott]

You right, I forgot to mention that. This error is different and is occurring before the application starts. That why I can't show you the checkpoint error and I didn't change the config and I have others application Running and some don't even start and the configuration of this application are similar. They are input jdbc and saving into hbase.

I will send you the dt.log

[Guilherme Hott]

Here the dtgateway.log and the FileNotFoundException