ValidateAndSplit fails to run on EMR yarn

398 views

Skip to first unread message

nel...@leantaas.com

unread,

Jan 31, 2018, 9:18:25 PM1/31/18

to Kylo Community

Hello,

I am running Ingest job on EMR cluster using yarn cluster mode.

On really small file which runs on local I am getting this exception.

2018-02-01 02:08:03,385 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 02:08:03 INFO yarn.Client:

2018-02-01 02:08:03,385 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] client token: N/A

2018-02-01 02:08:03,385 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] diagnostics: Shutdown hook called before final status was reported.

2018-02-01 02:08:03,385 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster host: 11.0.16.83

2018-02-01 02:08:03,385 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster RPC port: 0

2018-02-01 02:08:03,385 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] queue: default

2018-02-01 02:08:03,385 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] start time: 1517450813248

2018-02-01 02:08:03,385 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] final status: FAILED

2018-02-01 02:08:03,385 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] tracking URL: http://ip-11-0-16-57.dev.claraanalytics.local:20888/proxy/application_1516226182355_0320/

2018-02-01 02:08:03,385 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] user: nifi

2018-02-01 02:08:03,393 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] Exception in thread "main" org.apache.spark.SparkException: Application application_1516226182355_0320 finished with failed status

2018-02-01 02:08:03,393 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at org.apache.spark.deploy.yarn.Client.run(Client.scala:1104)

2018-02-01 02:08:03,393 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1150)

2018-02-01 02:08:03,393 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at org.apache.spark.deploy.yarn.Client.main(Client.scala)

2018-02-01 02:08:03,393 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

2018-02-01 02:08:03,393 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

2018-02-01 02:08:03,393 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2018-02-01 02:08:03,394 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at java.lang.reflect.Method.invoke(Method.java:498)

2018-02-01 02:08:03,394 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)

2018-02-01 02:08:03,394 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

2018-02-01 02:08:03,397 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 02:08:03 INFO util.ShutdownHookManager: Shutdown hook called

2018-02-01 02:08:03,398 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 02:08:03 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-e5e1d207-3210-4d7f-b4f8-707a1fa51cb6

Profiler job on yarn cluster is working correctly.

Could you please help to troubleshoot this?

Thank you,

Nella

Jagrut Sharma

unread,

Jan 31, 2018, 11:05:04 PM1/31/18

to Kylo Community

Nella - In yarn-cluster mode, the logs will be available on the cluster nodes and you can retrieve them via the tracking URL. That will give some indication on what the issue is. One of the common issues can be insufficient memory.

Thanks.

Jagrut

nel...@claraanalytics.com

unread,

Feb 1, 2018, 1:42:44 AM2/1/18

to Kylo Community

Jagrut,

Thank you for response! I am looking at yarn logs and cant find any clues for failure.

In case it is memory - which parameters would you recommend to adjust? More memory to executors? What would be your recommendation?

This job is running successfully on local with very limited memory.

Thanks

Nella

Jagrut Sharma

unread,

Feb 1, 2018, 4:35:06 AM2/1/18

to Kylo Community

Nella - You can refer to this troubleshooting doc [1] that talks about configuring Kylo Spark processor specifically for yarn-cluster mode. There are also some tips on parameter turning for yarn and spark.

[1] http://kylo.readthedocs.io/en/latest/tips-tricks/TroubleshootingandTips.html#tuning-the-executesparkjob-processor

Thanks.

Jagrut

nel...@claraanalytics.com

unread,

Feb 1, 2018, 11:21:43 AM2/1/18

to Kylo Community

Jagrut,

Thank you!!

I have seen this guide and have configured the processor accordingly. It is still failing with undefined error.

I would appreciate any other help or guidance.

The log does not say much.

Thanks,

Nella

Greg Hart

unread,

Feb 1, 2018, 12:40:23 PM2/1/18

to Kylo Community

Hi Nella,

Could you post your entire nifi-app.log file and the driver logs available from the YARN scheduler? I'm looking for the exit status and the lines before that.

nel...@claraanalytics.com

unread,

Feb 1, 2018, 3:16:10 PM2/1/18

to Kylo Community

Hello Greg,

This is the park of the nifi-app.log corresponding to ValidateAndSplit job.

I am trying to unterstand how to get driver log from EMR... I have found logs in this directory:

/var/log/hadoop-yarn directory

yarn-yarn-proxyserver-ip-11-0-16-57.log

yarn-yarn-resourcemanager-ip-11-0-16-57.log

yarn-yarn-timelineserver-ip-11-0-16-57.log.2018-02-01-19

Could you please help to understand what is the best way to get driver log?

Thank you!

Nella

2018-02-01 18:25:21,054 INFO [Timer-Driven Process Thread-7] c.t.nifi.v2.common.FeedProcessor Resolving id b66c44fb-6168-4556-8d16-1bd9e9a04aa7 for feed care_west/scores_from_app

2018-02-01 18:25:21,094 INFO [Timer-Driven Process Thread-8] c.t.nifi.v2.thrift.RefreshableDataSource connection obtained by RefreshableDatasource

2018-02-01 18:25:21,094 WARN [Timer-Driven Process Thread-8] c.t.nifi.v2.thrift.RefreshableDataSource The Statement.setQueryTimeout() method is not supported for the JDBC URL: jdbc:hive2://localhost:10000/default

2018-02-01 18:25:21,094 INFO [Timer-Driven Process Thread-8] c.t.nifi.v2.thrift.RefreshableDataSource perform validation query in RefreshableDatasource.executeWithTimeout()

2018-02-01 18:25:21,110 INFO [Timer-Driven Process Thread-8] c.t.nifi.v2.thrift.RefreshableDataSource validation query returned from RefreshableDatasource.executeWithTimeout() in 15.57 ms

2018-02-01 18:25:21,110 INFO [Timer-Driven Process Thread-8] c.t.nifi.v2.thrift.RefreshableDataSource Cleaning up the current connection using a background thread.

2018-02-01 18:25:21,166 INFO [Timer-Driven Process Thread-2] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] YARN deploy mode set to: cluster

2018-02-01 18:25:21,166 INFO [Timer-Driven Process Thread-2] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] Adding to class path '/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar'

2018-02-01 18:25:21,175 INFO [Timer-Driven Process Thread-2] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] Waiting for Spark job to complete

2018-02-01 18:25:21,691 INFO [FeedStatisticsManager-SendStats-0] c.t.n.p.jms.ProvenanceEventJmsWriter SENDING Batch Events to JMS ProvenanceEventRecordDTOHolder{events=17}

2018-02-01 18:25:22,724 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2018-02-01 18:25:24,596 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:24 INFO impl.TimelineClientImpl: Timeline service address: http://ip-11-0-16-57.dev.claraanalytics.local:8188/ws/v1/timeline/

2018-02-01 18:25:24,613 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:24 INFO client.RMProxy: Connecting to ResourceManager at ip-11-0-16-57.dev.claraanalytics.local/11.0.16.57:8032

2018-02-01 18:25:24,881 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:24 INFO yarn.Client: Requesting a new application from cluster with 5 NodeManagers

2018-02-01 18:25:24,952 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:24 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (5760 MB per container)

2018-02-01 18:25:24,952 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:24 INFO yarn.Client: Will allocate AM container, with 2432 MB memory including 384 MB overhead

2018-02-01 18:25:24,952 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:24 INFO yarn.Client: Setting up the launch environment for our AM container

2018-02-01 18:25:24,970 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:24 INFO yarn.Client: Preparing resources for our AM container

2018-02-01 18:25:26,403 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:26 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.

2018-02-01 18:25:27,027 INFO [Provenance Maintenance Thread-2] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 6071

2018-02-01 18:25:27,027 INFO [Provenance Maintenance Thread-2] c.t.n.p.r.KyloPersistentProvenanceEventRepository Created 16 KyloRecordWriterDelegate objects starting with: 6071

2018-02-01 18:25:27,049 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.lucene.SimpleIndexManager Index Writer for /opt/nifi/data/provenance_repository/index-1517021395000 has been returned to Index Manager and is no longer in use. Closing Index Writer

2018-02-01 18:25:27,050 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (22 records) into single Provenance Log File /opt/nifi/data/provenance_repository/6049.prov in 26 milliseconds

2018-02-01 18:25:27,050 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 10760 records. In the past 5 minutes, 47 events have been written to the Provenance Repository, totaling 256.71 KB

2018-02-01 18:25:28,236 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:28 INFO yarn.Client: Uploading resource file:/tmp/spark-9640bcba-8581-4d88-9ee8-90bee15e7562/__spark_libs__2341188370010500973.zip -> hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/user/nifi/.sparkStaging/application_1516226182355_0326/__spark_libs__2341188370010500973.zip

2018-02-01 18:25:31,199 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:31 INFO yarn.Client: Uploading resource file:/opt/nifi/current/lib/app/kylo-spark-validate-cleanse-jar-with-dependencies.jar -> hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/user/nifi/.sparkStaging/application_1516226182355_0326/kylo-spark-validate-cleanse-jar-with-dependencies.jar

2018-02-01 18:25:31,408 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:31 INFO yarn.Client: Uploading resource file:/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -> hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/user/nifi/.sparkStaging/application_1516226182355_0326/hive-hcatalog-core.jar

2018-02-01 18:25:31,437 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:31 INFO yarn.Client: Uploading resource file:/usr/hdp/current/spark-client/jars/datanucleus-api-jdo-3.2.6.jar -> hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/user/nifi/.sparkStaging/application_1516226182355_0326/datanucleus-api-jdo-3.2.6.jar

2018-02-01 18:25:31,461 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:31 INFO yarn.Client: Uploading resource file:/usr/hdp/current/spark-client/jars/datanucleus-core-3.2.10.jar -> hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/user/nifi/.sparkStaging/application_1516226182355_0326/datanucleus-core-3.2.10.jar

2018-02-01 18:25:31,912 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:31 INFO yarn.Client: Uploading resource file:/usr/hdp/current/spark-client/jars/datanucleus-rdbms-3.2.9.jar -> hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/user/nifi/.sparkStaging/application_1516226182355_0326/datanucleus-rdbms-3.2.9.jar

2018-02-01 18:25:32,352 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:32 INFO yarn.Client: Uploading resource file:/tmp/kylo-nifi/spark/care_west/scores_from_app/1517509521027/scores_from_app_field_policy.json -> hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/user/nifi/.sparkStaging/application_1516226182355_0326/scores_from_app_field_policy.json

2018-02-01 18:25:32,373 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:32 INFO yarn.Client: Uploading resource file:/etc/hadoop/conf/hive-site.xml -> hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/user/nifi/.sparkStaging/application_1516226182355_0326/hive-site.xml

2018-02-01 18:25:32,410 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:32 INFO yarn.Client: Uploading resource file:/tmp/spark-9640bcba-8581-4d88-9ee8-90bee15e7562/__spark_conf__4692910422266686112.zip -> hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/user/nifi/.sparkStaging/application_1516226182355_0326/__spark_conf__.zip

2018-02-01 18:25:32,459 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:32 INFO spark.SecurityManager: Changing view acls to: nifi

2018-02-01 18:25:32,460 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:32 INFO spark.SecurityManager: Changing modify acls to: nifi

2018-02-01 18:25:32,461 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:32 INFO spark.SecurityManager: Changing view acls groups to:

2018-02-01 18:25:32,462 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:32 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nifi); groups with view permissions: Set(); users with modify permissions: Set(nifi); groups with modify permissions: Set()

2018-02-01 18:25:32,481 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:32 INFO yarn.Client: Submitting application application_1516226182355_0326 to ResourceManager

2018-02-01 18:25:32,511 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:32 INFO impl.YarnClientImpl: Submitted application application_1516226182355_0326

2018-02-01 18:25:33,515 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:33 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:25:33,519 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:33 INFO yarn.Client:

2018-02-01 18:25:33,519 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] client token: N/A

2018-02-01 18:25:33,519 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] diagnostics: N/A

2018-02-01 18:25:33,519 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster host: N/A

2018-02-01 18:25:33,519 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster RPC port: -1

2018-02-01 18:25:33,519 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] queue: default

2018-02-01 18:25:33,519 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] start time: 1517509532497

2018-02-01 18:25:33,519 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] final status: UNDEFINED

2018-02-01 18:25:33,519 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] tracking URL: http://ip-11-0-16-57.dev.claraanalytics.local:20888/proxy/application_1516226182355_0326/

2018-02-01 18:25:33,519 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] user: nifi

2018-02-01 18:25:34,520 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:34 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:25:35,522 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:35 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:25:36,523 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:36 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:25:37,525 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:37 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:25:38,527 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:38 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:25:39,528 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:39 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:25:40,530 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:40 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:25:41,532 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:41 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:41,532 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:41 INFO yarn.Client:

2018-02-01 18:25:41,532 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] client token: N/A

2018-02-01 18:25:41,532 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] diagnostics: N/A

2018-02-01 18:25:41,532 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster host: 11.0.16.196

2018-02-01 18:25:41,532 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster RPC port: 0

2018-02-01 18:25:41,532 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] queue: default

2018-02-01 18:25:41,532 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] start time: 1517509532497

2018-02-01 18:25:41,532 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] final status: UNDEFINED

2018-02-01 18:25:41,532 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] tracking URL: http://ip-11-0-16-57.dev.claraanalytics.local:20888/proxy/application_1516226182355_0326/

2018-02-01 18:25:41,532 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] user: nifi

2018-02-01 18:25:42,534 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:42 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:43,535 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:43 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:44,537 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:44 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:45,539 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:45 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:46,540 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:46 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:47,292 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5724131e checkpointed with 2204 Records and 0 Swap Files in 13 milliseconds (Stop-the-world time = 3 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID 7246

2018-02-01 18:25:47,544 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:47 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:48,545 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:48 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:49,547 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:49 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:50,549 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:50 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:51,551 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:51 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:52,552 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:52 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:53,554 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:53 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:54,555 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:54 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:55,557 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:55 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:56,559 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:56 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:57,560 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:57 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:58,562 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:58 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:25:59,564 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:25:59 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:00,565 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:00 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:01,567 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:01 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:02,569 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:02 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:03,571 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:03 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:04,572 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:04 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:05,574 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:05 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:06,576 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:06 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:07,577 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:07 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:08,579 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:08 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:09,581 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:09 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:10,582 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:10 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:11,584 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:11 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:12,586 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:12 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:13,588 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:13 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:14,590 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:14 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:15,592 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:15 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:16,593 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:16 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:17,595 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:17 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:18,597 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:18 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:19,599 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:19 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:20,600 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:20 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:21,602 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:21 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:22,603 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:22 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:23,605 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:23 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:24,607 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:24 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:25,608 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:25 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:26,610 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:26 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:27,613 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:27 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:28,615 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:28 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:29,617 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:29 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:30,618 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:30 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:31,620 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:31 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:32,621 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:32 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:33,623 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:33 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:34,625 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:34 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:35,627 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:35 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:36,628 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:36 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:37,630 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:37 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:38,632 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:38 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:39,633 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:39 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:40,635 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:40 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:41,636 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:41 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:42,639 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:42 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:26:42,639 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:42 INFO yarn.Client:

2018-02-01 18:26:42,639 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] client token: N/A

2018-02-01 18:26:42,639 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] diagnostics: N/A

2018-02-01 18:26:42,639 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster host: N/A

2018-02-01 18:26:42,639 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster RPC port: -1

2018-02-01 18:26:42,639 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] queue: default

2018-02-01 18:26:42,639 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] start time: 1517509532497

2018-02-01 18:26:42,639 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] final status: UNDEFINED

2018-02-01 18:26:42,639 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] tracking URL: http://ip-11-0-16-57.dev.claraanalytics.local:20888/proxy/application_1516226182355_0326/

2018-02-01 18:26:42,639 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] user: nifi

2018-02-01 18:26:43,640 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:43 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:26:44,642 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:44 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:26:45,643 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:45 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:26:46,646 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:46 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:26:47,647 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:47 INFO yarn.Client: Application report for application_1516226182355_0326 (state: ACCEPTED)

2018-02-01 18:26:48,649 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:48 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:48,649 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:48 INFO yarn.Client:

2018-02-01 18:26:48,649 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] client token: N/A

2018-02-01 18:26:48,649 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] diagnostics: N/A

2018-02-01 18:26:48,649 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster host: 11.0.16.83

2018-02-01 18:26:48,649 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster RPC port: 0

2018-02-01 18:26:48,649 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] queue: default

2018-02-01 18:26:48,649 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] start time: 1517509532497

2018-02-01 18:26:48,649 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] final status: UNDEFINED

2018-02-01 18:26:48,649 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] tracking URL: http://ip-11-0-16-57.dev.claraanalytics.local:20888/proxy/application_1516226182355_0326/

2018-02-01 18:26:48,649 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] user: nifi

2018-02-01 18:26:49,651 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:49 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:50,652 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:50 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:51,654 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:51 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:52,656 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:52 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:53,657 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:53 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:54,659 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:54 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:55,660 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:55 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:56,662 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:56 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:57,664 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:57 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:58,665 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:58 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:26:59,667 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:26:59 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:00,669 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:00 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:01,671 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:01 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:02,672 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:02 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:03,674 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:03 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:04,675 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:04 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:05,677 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:05 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:06,678 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:06 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:07,680 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:07 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:08,681 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:08 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:09,683 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:09 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:10,685 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:10 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:11,687 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:11 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:12,688 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:12 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:13,326 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository

2018-02-01 18:27:13,394 INFO [pool-10-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@25b27891 checkpointed with 1 Records and 0 Swap Files in 67 milliseconds (Stop-the-world time = 39 milliseconds, Clear Edit Logs time = 23 millis), max Transaction ID 5884

2018-02-01 18:27:13,394 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 1 records in 67 milliseconds

2018-02-01 18:27:13,690 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:13 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:13,829 INFO [FileSystemRepository Workers Thread-1] o.a.n.c.repository.FileSystemRepository Successfully destroyed 1 Resource Claims for Container default in 0 millis

2018-02-01 18:27:14,691 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:14 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:15,693 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:15 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:16,694 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:16 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:17,696 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:17 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:18,698 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:18 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:19,699 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:19 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:20,701 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:20 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:21,703 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:21 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:22,705 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:22 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:23,707 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:23 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:24,709 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:24 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:25,710 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:25 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:26,712 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:26 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:27,713 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:27 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:28,715 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:28 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:29,716 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:29 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:30,718 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:30 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:31,719 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:31 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:32,721 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:32 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:33,722 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:33 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:34,724 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:34 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:35,725 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:35 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:36,727 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:36 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:37,728 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:37 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:38,730 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:38 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:39,732 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:39 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:40,733 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:40 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:41,735 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:41 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:42,736 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:42 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:43,737 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:43 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:44,739 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:44 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:45,741 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:45 INFO yarn.Client: Application report for application_1516226182355_0326 (state: RUNNING)

2018-02-01 18:27:46,745 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:46 INFO yarn.Client: Application report for application_1516226182355_0326 (state: FINISHED)

2018-02-01 18:27:46,746 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:46 INFO yarn.Client:

2018-02-01 18:27:46,746 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] client token: N/A

2018-02-01 18:27:46,746 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] diagnostics: Shutdown hook called before final status was reported.

2018-02-01 18:27:46,746 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster host: 11.0.16.83

2018-02-01 18:27:46,746 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ApplicationMaster RPC port: 0

2018-02-01 18:27:46,746 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] queue: default

2018-02-01 18:27:46,746 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] start time: 1517509532497

2018-02-01 18:27:46,746 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] final status: FAILED

2018-02-01 18:27:46,746 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] tracking URL: http://ip-11-0-16-57.dev.claraanalytics.local:20888/proxy/application_1516226182355_0326/

2018-02-01 18:27:46,746 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] user: nifi

2018-02-01 18:27:46,750 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] Exception in thread "main" org.apache.spark.SparkException: Application application_1516226182355_0326 finished with failed status

2018-02-01 18:27:46,750 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at org.apache.spark.deploy.yarn.Client.run(Client.scala:1104)

2018-02-01 18:27:46,750 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1150)

2018-02-01 18:27:46,750 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at org.apache.spark.deploy.yarn.Client.main(Client.scala)

2018-02-01 18:27:46,750 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

2018-02-01 18:27:46,750 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

2018-02-01 18:27:46,750 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2018-02-01 18:27:46,750 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at java.lang.reflect.Method.invoke(Method.java:498)

2018-02-01 18:27:46,750 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)

2018-02-01 18:27:46,751 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)

2018-02-01 18:27:46,751 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

2018-02-01 18:27:46,755 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:46 INFO util.ShutdownHookManager: Shutdown hook called

2018-02-01 18:27:46,756 INFO [stream error] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] 18/02/01 18:27:46 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-9640bcba-8581-4d88-9ee8-90bee15e7562

2018-02-01 18:27:46,820 ERROR [Timer-Driven Process Thread-2] c.t.nifi.v2.spark.ExecuteSparkJob ExecuteSparkJob[id=9493d49b-fe13-35bd-7880-659f01bc541c] ExecuteSparkJob for Validate And Split Records and flowfile: StandardFlowFileRecord[uuid=976349b5-c076-4c50-8642-7650c6d804c0,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1517509521018-351, container=default, section=351], offset=0, length=278591],offset=0,name=carewest_2017Q3_scores.csv,size=278591] completed with failed status 1

2018-02-01 18:27:47,034 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 6074

2018-02-01 18:27:47,034 INFO [Provenance Maintenance Thread-3] c.t.n.p.r.KyloPersistentProvenanceEventRepository Created 16 KyloRecordWriterDelegate objects starting with: 6074

2018-02-01 18:27:47,053 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.lucene.SimpleIndexManager Index Writer for /opt/nifi/data/provenance_repository/index-1517021395000 has been returned to Index Manager and is no longer in use. Closing Index Writer

2018-02-01 18:27:47,054 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (3 records) into single Provenance Log File /opt/nifi/data/provenance_repository/6071.prov in 22 milliseconds

2018-02-01 18:27:47,055 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 5979 records. In the past 5 minutes, 25 events have been written to the Provenance Repository, totaling 139.43 KB

2018-02-01 18:27:47,305 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5724131e checkpointed with 2204 Records and 0 Swap Files in 13 milliseconds (Stop-the-world time = 3 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID 7246

2018-02-01 18:27:48,690 INFO [FeedStatisticsManager-SendStats-1] c.t.n.p.jms.ProvenanceEventJmsWriter SENDING Batch Events to JMS ProvenanceEventRecordDTOHolder{events=3}

2018-02-01 18:29:13,394 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository

2018-02-01 18:29:13,455 INFO [pool-10-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@25b27891 checkpointed with 0 Records and 0 Swap Files in 60 milliseconds (Stop-the-world time = 37 milliseconds, Clear Edit Logs time = 20 millis), max Transaction ID 5887

2018-02-01 18:29:13,455 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 60 milliseconds

nel...@claraanalytics.com

unread,

Feb 1, 2018, 3:19:49 PM2/1/18

to Kylo Community

This is the end of the file... Seems like it has been cut off

Jagrut Sharma

unread,

Feb 1, 2018, 3:27:41 PM2/1/18

to Kylo Community

Nella - Try this:

yarn logs -applicationId application_1516226182355_0326

Thanks.

Jagrut

nel...@claraanalytics.com

unread,

Feb 1, 2018, 3:40:16 PM2/1/18

to Kylo Community

Jagrut,

This is not working on EMR since log aggregation has not been enabled.

Thank you,

Nella

Jagrut Sharma

unread,

Feb 1, 2018, 4:07:37 PM2/1/18

to Kylo Community

Nella - You may want to refer the EMR documentation for getting the logs. Here is one result that came up on SO [1]

[1] https://stackoverflow.com/questions/30494905/where-are-the-spark-logs-on-emr

Thanks.

Jagrut

Greg Hart

unread,

Feb 1, 2018, 4:37:06 PM2/1/18

to Kylo Community

Hi Nella,

I found the error message:

Shutdown hook called before final status was reported.

I think you're running into this issue:

https://kylo-io.atlassian.net/browse/KYLO-1460

Kylo currently doesn't support Spark 2.2 yet. You can try downgrading to an earlier version of Spark, or delete the following line and recompile the app:

https://github.com/Teradata/kylo/blob/v0.8.4.1/integrations/spark/spark-validate-cleanse/spark-validate-cleanse-app/src/main/java/com/thinkbiganalytics/spark/datavalidator/Validator.java#L51

nel...@claraanalytics.com

unread,

Feb 1, 2018, 5:00:25 PM2/1/18

to Kylo Community

I have found the file on the slave...

Seems like the job is running and exiting due to Shutdown hook called before final status was reported.

Any clues here what could be going wrong?

18/02/01 19:13:06 INFO metadata.Hive: Renaming src: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/.hive-staging_hive_2018-02-01_19-13-01_512_851811547252953553-1/-ext-10000/part-00043-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, dest: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/processing_dttm=1517512242655/part-00043-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, Status:true

18/02/01 19:13:06 INFO metadata.Hive: Renaming src: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/.hive-staging_hive_2018-02-01_19-13-01_512_851811547252953553-1/-ext-10000/part-00044-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, dest: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/processing_dttm=1517512242655/part-00044-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, Status:true

18/02/01 19:13:06 INFO metadata.Hive: Renaming src: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/.hive-staging_hive_2018-02-01_19-13-01_512_851811547252953553-1/-ext-10000/part-00045-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, dest: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/processing_dttm=1517512242655/part-00045-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, Status:true

18/02/01 19:13:06 INFO metadata.Hive: Renaming src: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/.hive-staging_hive_2018-02-01_19-13-01_512_851811547252953553-1/-ext-10000/part-00046-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, dest: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/processing_dttm=1517512242655/part-00046-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, Status:true

18/02/01 19:13:06 INFO metadata.Hive: Renaming src: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/.hive-staging_hive_2018-02-01_19-13-01_512_851811547252953553-1/-ext-10000/part-00047-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, dest: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/processing_dttm=1517512242655/part-00047-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, Status:true

18/02/01 19:13:06 INFO metadata.Hive: Renaming src: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/.hive-staging_hive_2018-02-01_19-13-01_512_851811547252953553-1/-ext-10000/part-00048-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, dest: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/processing_dttm=1517512242655/part-00048-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, Status:true

18/02/01 19:13:06 INFO metadata.Hive: Renaming src: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/.hive-staging_hive_2018-02-01_19-13-01_512_851811547252953553-1/-ext-10000/part-00049-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, dest: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/processing_dttm=1517512242655/part-00049-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, Status:true

18/02/01 19:13:06 INFO metadata.Hive: Renaming src: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/.hive-staging_hive_2018-02-01_19-13-01_512_851811547252953553-1/-ext-10000/part-00050-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, dest: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/processing_dttm=1517512242655/part-00050-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, Status:true

18/02/01 19:13:06 INFO metadata.Hive: Renaming src: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/.hive-staging_hive_2018-02-01_19-13-01_512_851811547252953553-1/-ext-10000/part-00051-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, dest: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/processing_dttm=1517512242655/part-00051-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, Status:true

18/02/01 19:13:06 INFO metadata.Hive: Renaming src: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/.hive-staging_hive_2018-02-01_19-13-01_512_851811547252953553-1/-ext-10000/part-00052-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, dest: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/processing_dttm=1517512242655/part-00052-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, Status:true

18/02/01 19:13:06 INFO metadata.Hive: Renaming src: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/.hive-staging_hive_2018-02-01_19-13-01_512_851811547252953553-1/-ext-10000/part-00053-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, dest: hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/model.db/care_west/scores_from_app/profile/processing_dttm=1517512242655/part-00053-8ad4e5b3-e582-45de-8d30-78a75546d454-c000, Status:true

18/02/01 19:13:06 INFO execution.SparkSqlParser: Parsing command: `care_west`.`scores_from_app_profile`

18/02/01 19:13:06 INFO parser.CatalystSqlParser: Parsing command: string

18/02/01 19:13:06 INFO rdd.MapPartitionsRDD: Removing RDD 8 from persistence list

18/02/01 19:13:06 INFO storage.BlockManager: Removing RDD 8

18/02/01 19:13:06 INFO spark.SparkContext: Invoking stop() from shutdown hook

18/02/01 19:13:06 INFO server.AbstractConnector: Stopped Spark@45b948d4{HTTP/1.1,[http/1.1]}{0.0.0.0:0}

18/02/01 19:13:06 INFO ui.SparkUI: Stopped Spark web UI at http://11.0.16.83:38571

18/02/01 19:13:06 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).

18/02/01 19:13:06 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors

18/02/01 19:13:06 INFO yarn.YarnAllocator: Canceling requests for 5 executor container(s) to have a new desired total 0 executors.

18/02/01 19:13:06 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down

18/02/01 19:13:06 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices

(serviceOption=None,

services=List(),

started=false)

18/02/01 19:13:06 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

18/02/01 19:13:06 INFO memory.MemoryStore: MemoryStore cleared

18/02/01 19:13:06 INFO storage.BlockManager: BlockManager stopped

18/02/01 19:13:06 INFO storage.BlockManagerMaster: BlockManagerMaster stopped

18/02/01 19:13:06 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

18/02/01 19:13:06 INFO spark.SparkContext: Successfully stopped SparkContext

18/02/01 19:13:06 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 16, (reason: Shutdown hook called before final status was reported.)

18/02/01 19:13:06 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Shutdown hook called before final status was reported.)

18/02/01 19:13:06 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.

18/02/01 19:13:06 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://ip-11-0-16-57.dev.claraanalytics.local:8020/user/nifi/.sparkStaging/application_1516226182355_0327

18/02/01 19:13:06 INFO util.ShutdownHookManager: Shutdown hook called

18/02/01 19:13:06 INFO util.ShutdownHookManager: Deleting directory /mnt1/yarn/usercache/nifi/appcache/application_1516226182355_0327/spark-1325ac78-413b-4955-90d8-7178acac01ff

18/02/01 19:13:06 INFO util.ShutdownHookManager: Deleting directory /mnt/yarn/usercache/nifi/appcache/application_1516226182355_0327/spark-aecc3ac8-10a0-4309-a2ef-c22b8f108f86

nel...@claraanalytics.com

unread,

Feb 1, 2018, 5:10:50 PM2/1/18

to Kylo Community

Thank you!!!

nel...@claraanalytics.com

unread,

Feb 2, 2018, 3:32:00 PM2/2/18

to Kylo Community

Thank you!!

I have recompiled and it is working now!!!

Could you please advise what needs to be changed in SparkShellApp so that we could run VisualQuery on the cluster? Seems like it is failing with the same error.

Gregory Hart

unread,

Feb 2, 2018, 6:21:11 PM2/2/18

to Kylo Community

Hi Nella,

Could you attach your /var/log/kylo-spark-shell/kylo-spark-shell.log file?

nel...@claraanalytics.com

unread,

Feb 7, 2018, 2:36:45 AM2/7/18

to Kylo Community

I am creating configuration to run in managed mode.

In Visual Query I am getting message

An error occurred while initializing the Spark Shell.

2018-02-07 07:25:46 INFO localhost-startStop-1:Utils:54 - Successfully started service 'sparkDriver' on port 35256.

2018-02-07 07:25:46 INFO localhost-startStop-1:SparkEnv:54 - Registering MapOutputTracker

2018-02-07 07:25:46 INFO localhost-startStop-1:SparkEnv:54 - Registering BlockManagerMaster

2018-02-07 07:25:46 INFO localhost-startStop-1:BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information

2018-02-07 07:25:46 INFO localhost-startStop-1:BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up

2018-02-07 07:25:46 INFO localhost-startStop-1:DiskBlockManager:54 - Created local directory at /tmp/blockmgr-99d406fa-32b6-4b90-85db-af1e1ef5eb41

2018-02-07 07:25:46 INFO localhost-startStop-1:MemoryStore:54 - MemoryStore started with capacity 2004.6 MB

2018-02-07 07:25:46 INFO localhost-startStop-1:SparkEnv:54 - Registering OutputCommitCoordinator

2018-02-07 07:25:47 INFO localhost-startStop-1:log:192 - Logging initialized @29046ms

2018-02-07 07:25:47 INFO localhost-startStop-1:Server:345 - jetty-9.3.z-SNAPSHOT

2018-02-07 07:25:47 INFO localhost-startStop-1:Server:403 - Started @29180ms

2018-02-07 07:25:47 INFO localhost-startStop-1:AbstractConnector:270 - Started ServerConnector@575610a5{HTTP/1.1,[http/1.1]}{0.0.0.0:8451}

2018-02-07 07:25:47 INFO localhost-startStop-1:Utils:54 - Successfully started service 'SparkUI' on port 8451.

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@241c3a1e{/jobs,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6af0b501{/jobs/json,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@37ec4c2c{/jobs/job,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6fe50002{/jobs/job/json,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@ba1abaf{/stages,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2de76978{/stages/json,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7c10da47{/stages/stage,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7b8ccc5f{/stages/stage/json,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3de46d99{/stages/pool,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e3fd7c{/stages/pool/json,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7491e39a{/storage,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77295ad7{/storage/json,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@58130562{/storage/rdd,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4956ecfc{/storage/rdd/json,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7385fe9{/environment,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2a4a3192{/environment/json,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2b9fb04a{/executors,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2cec7c14{/executors/json,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7f54b232{/executors/threadDump,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6f975f3e{/executors/threadDump/json,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66117af4{/static,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@483893aa{/,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@42003aa3{/api,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@8849812{/jobs/job/kill,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7cd66057{/stages/stage/kill,null,AVAILABLE,@Spark}

2018-02-07 07:25:47 INFO localhost-startStop-1:SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://11.0.16.69:8451

2018-02-07 07:25:47 INFO localhost-startStop-1:SparkContext:54 - Added JAR file:/opt/kylo/kylo-services/lib/app/kylo-spark-shell-client-v2-0.8.4.1.jar at spark://11.0.16.69:35256/jars/kylo-spark-shell-client-v2-0.8.4.1.jar with timestamp 1517988347408

2018-02-07 07:25:47 INFO localhost-startStop-1:Executor:54 - Starting executor ID driver on host localhost

2018-02-07 07:25:47 INFO localhost-startStop-1:Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34073.

2018-02-07 07:25:47 INFO localhost-startStop-1:NettyBlockTransferService:54 - Server created on 11.0.16.69:34073

2018-02-07 07:25:47 INFO localhost-startStop-1:BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy

2018-02-07 07:25:47 INFO localhost-startStop-1:BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, 11.0.16.69, 34073, None)

2018-02-07 07:25:47 INFO dispatcher-event-loop-2:BlockManagerMasterEndpoint:54 - Registering block manager 11.0.16.69:34073 with 2004.6 MB RAM, BlockManagerId(driver, 11.0.16.69, 34073, None)

2018-02-07 07:25:47 INFO localhost-startStop-1:BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 11.0.16.69, 34073, None)

2018-02-07 07:25:47 INFO localhost-startStop-1:BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, 11.0.16.69, 34073, None)

2018-02-07 07:25:47 INFO localhost-startStop-1:ContextHandler:781 - Started o.s.j.s.ServletContextHandler@51129671{/metrics/json,null,AVAILABLE,@Spark}

2018-02-07 07:25:48 INFO localhost-startStop-1:FilterRegistrationBean:258 - Mapping filter: 'characterEncodingFilter' to: [/*]

2018-02-07 07:25:48 INFO localhost-startStop-1:FilterRegistrationBean:258 - Mapping filter: 'requestContextFilter' to: [/*]

2018-02-07 07:25:48 INFO localhost-startStop-1:ServletRegistrationBean:189 - Mapping servlet: 'org.glassfish.jersey.server.ResourceConfig' to [/*]

2018-02-07 07:25:49 INFO main:AnnotationMBeanExporter:431 - Registering beans for JMX exposure on startup

2018-02-07 07:25:49 INFO main:Http11NioProtocol:180 - Initializing ProtocolHandler ["http-nio-8450"]

2018-02-07 07:25:49 INFO main:Http11NioProtocol:180 - Starting ProtocolHandler ["http-nio-8450"]

2018-02-07 07:25:49 INFO main:NioSelectorPool:180 - Using a shared selector for servlet write/read

2018-02-07 07:25:49 INFO main:TomcatEmbeddedServletContainer:162 - Tomcat started on port(s): 8450 (http)

2018-02-07 07:25:49 INFO main:RemoteClientRunner:137 - Remote server address not set. Skipping registration.

2018-02-07 07:25:49 INFO main:SparkShellApp:57 - Started SparkShellApp in 29.748 seconds (JVM running for 31.936)

Greg Hart

unread,

Feb 8, 2018, 2:19:23 PM2/8/18

to Kylo Community

Hi Nella,

Here's some additional instructions to help you configuring Kylo Spark Shell:

http://kylo.readthedocs.io/en/latest/security/KyloUserImpersonation.html

In managed mode the logs will be written to /var/log/kylo-services/kylo-services.log instead of /var/log/kylo-spark-shell/kylo-spark-shell.log. See step 4 for more information and for changing the location.

If you don't need user impersonation then you can set spark.shell.proxyUser=false and ignore the Kerberos settings in step 3. You also can ignore step 5.

nel...@claraanalytics.com

unread,

Feb 9, 2018, 2:58:15 AM2/9/18

to Kylo Community

Greg,

Thank you!!

I have configured managed mode without user impersonation and repointed log.

Now I can see following in the log:

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - Exception in thread "main" java.lang.LinkageError: ClassCastException: attempting to castjar:file:/opt/kylo/kylo-services/lib/app/kylo-spark-shell-client-v2-0.8.4.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class to jar:file:/opt/spark/spark-2.1.1-bin-hadoop2.7/jars/javax.ws.rs-api-2.0.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:146)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:120)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at javax.ws.rs.core.MediaType.valueOf(MediaType.java:179)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:65)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at com.sun.jersey.api.client.Client.init(Client.java:342)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at com.sun.jersey.api.client.Client.access$000(Client.java:118)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at com.sun.jersey.api.client.Client$1.f(Client.java:191)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at com.sun.jersey.api.client.Client$1.f(Client.java:187)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at com.sun.jersey.api.client.Client.<init>(Client.java:187)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at com.sun.jersey.api.client.Client.<init>(Client.java:170)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(TimelineClientImpl.java:280)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:169)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:152)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1226)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.spark.deploy.yarn.Client.main(Client.scala)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at java.lang.reflect.Method.invoke(Method.java:498)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)

2018-02-09 07:53:33 INFO launcher-proc-2:SparkShellApp:61 - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Greg Hart

unread,

Feb 13, 2018, 2:23:07 PM2/13/18

to Kylo Community

Hi Nella,

I'll need some more information to reproduce this issue. Does this also happen when you use a similar configuration in the Kylo sandbox? Does this also happen with Spark 2.0 or Spark 1.6?

You can try fixing this if you don't mind setting up a development environment. You would need to edit the https://github.com/Teradata/kylo/blob/master/integrations/spark/spark-shell-client/spark-shell-client-v2/pom.xml file and exclude the javax.ws.rs-api artifact in the maven-shade-plugin. There may be additional jars that need to be excluded or incompatibilities between the artifacts used by Kylo and the artifacts used by Spark.

nel...@claraanalytics.com

unread,

Feb 16, 2018, 6:05:05 PM2/16/18

to Kylo Community

Greg,

Thank you!!

I have been able to figure out how to resolve this issue.

I changed yarn-site.xml

yarn-timeline-service = false

This resolved the inconsistency.

Thank you!

Greg Hart

unread,

Feb 20, 2018, 12:22:51 PM2/20/18

to Kylo Community

Thanks for posting the solution!

Reply all

Reply to author

Forward

0 new messages