NIFI out of memory error

[Jasim Waheed]

I am currently facing issue in starting up NIFI service. In the nifi-app,log file, I noticed that its out of memory. So, I changed the original memory heap (-Xmx512m) in /opt/nifi/current/conf/bootstap.conf file to -Xmx4096m. But I  am still unable to run the nifi service.

Attached are the log file of nifi:

nifi-app.log:

2017-12-19 17:08:37,133 ERROR [ActiveMQ InactivityMonitor ReadCheckTimer] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[ActiveMQ InactivityMonitor ReadCheckTimer,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,134 ERROR [Provenance Maintenance Thread-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Provenance Maintenance Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,134 ERROR [logback-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[logback-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,133 ERROR [FileSystemRepository Workers Thread-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[FileSystemRepository Workers Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,135 ERROR [pool-15-thread-1] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[pool-15-thread-1,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,133 ERROR [Flow Service Tasks Thread-1] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Flow Service Tasks Thread-1,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [Provenance Maintenance Thread-2] org.apache.nifi.NiFi
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [logback-2] org.apache.nifi.NiFi
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [FileSystemRepository Workers Thread-2] org.apache.nifi.NiFi
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [Flow Service Tasks Thread-1] org.apache.nifi.NiFi
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [Event-Driven Process Thread-1] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,483 ERROR [FileSystemRepository Workers Thread-4] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [pool-15-thread-1] org.apache.nifi.NiFi
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,497 ERROR [ActiveMQ InactivityMonitor ReadCheckTimer] org.apache.nifi.NiFi
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:40,509 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5351d0ef checkpointed with 2358 Records and 0 Swap Files in 114969 milliseconds (Stop-the-world time = 2361 milliseconds, Clear Edit Logs time = 936 millis), max Transaction ID 7622
2017-12-19 17:10:41,433 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5351d0ef checkpointed with 2358 Records and 0 Swap Files in 922 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 7622

[Greg Hart]

Hi Jasim,

This looks like an issue with NiFi and not with Kylo. You can try asking on the Apache NiFi mailing list or contacting Think Big Analytics for paid support options.

Please note that if you're on the Kylo sandbox then it only supports files that are a few megabytes in size. You'll need to install Kylo into an actual Hadoop cluster if you need to process files that are gigabytes in size.

If this error is related to one of Kylo's sample templates then please provide more information on how to reproduce this error, what Kylo version you're using, and please attach your entire nifi-app.log file.

[Jagrut Sharma]

Jasim - Check the memory for the Kylo sandbox - it needs at least 8 GB.

Thanks.

--

Jagrut

[Jasim Waheed]

Hi Greg, Jagrut,

I was able to resolve the issue. Reason for OOM was that I was using Split Processor for a large flow file and it was splitting into millions of small flow files.

To resolve this, I deleted the repos and  set nifi.flowcontroller.autoResumeState=false in nifi.properties. Restarted nifi and it worked.

Thank you. :)