FileNotFoundException on the checkpoints MORE DETAILS

[Guilherme Hott]

OK, let's we try again. Before I was running in a Amazon EMR, now I setup a new server on Google Cloud Platform DataProc but the error stills.

I am launching an application and after the container start I have a ContainerError. The error is that:

File does not exist: /user/ghott3/datatorrent/apps/application_1500931817765_0015/checkpoints/3/ffffffffffffffff
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)

I checked the directory on the hdfs and doesn't exist. 

So, what could be this error if I can run this same application locally?

[Sanjay Pujare]

As far as I know you are the first one trying Apex or DT RTS on Google Cloud Platform DataProc. It will be good to get this working/qualified on that platform.

You get "file doe not exist" for /user/ghott3/datatorrent/apps/application_1500931817765_0015/checkpoints/3/ffffffffffffffff

but what about for parent directories such as /user/ghott3/datatorrent/apps/application_1500931817765_0015/checkpoints/3  or /user/ghott3/datatorrent/apps/application_1500931817765_0015/checkpoints or /user/ghott3/datatorrent/apps/application_1500931817765_0015 to  see which node is missing?

Also in a previous email I had asked to enable debug logging as per http://docs.datatorrent.com/configuration/#application-logging . Was this an app restart scenario (launch with -originalAppId option)? 

--
You received this message because you are subscribed to the Google Groups "DataTorrent Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dt-users+unsubscribe@googlegroups.com.
To post to this group, send email to dt-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dt-users/4ba8068a-e9c3-407d-993f-b65d46b2a297%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Guilherme Hott]

I don't have any directory into /user/ghott3/datatorrent/apps/application_1500931817765_0015

just the .jar files

 And this is the directory of the PiDemo application, which contains the checkpoint directory

And until now is being easy run on GCP DataProc, I just had to set the log4j dependency as provided because was conflicting.

Thanks

On Monday, July 24, 2017 at 7:32:15 PM UTC-7, Sanjay Pujare wrote:

As far as I know you are the first one trying Apex or DT RTS on Google Cloud Platform DataProc. It will be good to get this working/qualified on that platform.

You get "file doe not exist" for /user/ghott3/datatorrent/apps/application_1500931817765_0015/checkpoints/3/ffffffffffffffff

but what about for parent directories such as /user/ghott3/datatorrent/apps/application_1500931817765_0015/checkpoints/3  or /user/ghott3/datatorrent/apps/application_1500931817765_0015/checkpoints or /user/ghott3/datatorrent/apps/application_1500931817765_0015 to  see which node is missing?

Also in a previous email I had asked to enable debug logging as per http://docs.datatorrent.com/configuration/#application-logging . Was this an app restart scenario (launch with -originalAppId option)? 

On Tue, Jul 25, 2017 at 6:45 AM, Guilherme Hott <guilhe...@gmail.com> wrote:

OK, let's we try again. Before I was running in a Amazon EMR, now I setup a new server on Google Cloud Platform DataProc but the error stills.

I am launching an application and after the container start I have a ContainerError. The error is that:

File does not exist: /user/ghott3/datatorrent/apps/application_1500931817765_0015/checkpoints/3/ffffffffffffffff
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)

I checked the directory on the hdfs and doesn't exist. 

So, what could be this error if I can run this same application locally?

--
You received this message because you are subscribed to the Google Groups "DataTorrent Users Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dt-users+u...@googlegroups.com.

[Guilherme Hott]

Look the dtgateway.log file

[Guilherme Hott]

Sorry guys, after many trials and errors I found the issue: the dependencies of apache drill. I set the scope to "provided" and worked.

<dependency>

    <groupId>org.apache.drill.exec</groupId>

    <artifactId>drill-jdbc</artifactId>

    <version>1.10.0</version>

    <scope>provided</scope>

</dependency>

So, now I have a Datatorrent environment running on GCP DataProc (https://cloud.google.com/dataproc/) and working better than Amazon EMR in my humble opinion and knowledge. Was easy to setup the instance and I just had to set some dependencies with provided scope (drill-jdbc and log4j-over-slf4j). This instance has 8 vCPUs, 30 GB memory, but you can setup a bunch of different cores and memory. The only thing about the machine is the hbase is not on the startup services, so you have to add it.

Thank Sanjay Pujare for the patience, I appreciate.