Kylo Feeds giving NiFi not Up Error

[Bimal Mehta]

I had a small issue with my PC where it shut down automatically while a few of my feeds were running.

After the restart I am not able to access the NiFi Url.(localhost:8079)

The Kylo UI opens but none of the feeds.

Below is the latest Kylo Services log. I have also attached the complete log.

2017-10-21 18:17:49 ERROR http-nio-8420-exec-1:BaseExceptionMapper:46 - toResponse() caught Exception: com.thinkbiganalytics.nifi.rest.client.NifiClientRuntimeException, java.lang.IllegalStateException
com.thinkbiganalytics.nifi.rest.client.NifiClientRuntimeException: java.lang.IllegalStateException
 at com.thinkbiganalytics.nifi.rest.client.NifiRestClientExceptionTranslator.translateException(NifiRestClientExceptionTranslator.java:58)
 at com.thinkbiganalytics.nifi.rest.config.NifiRestClientAroundAspect.NifiRestClientAroundAspect(NifiRestClientAroundAspect.java:50)
 at sun.reflect.GeneratedMethodAccessor391.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
 at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
 at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
 at com.thinkbiganalytics.nifi.rest.client.LegacyNifiRestClient$$EnhancerBySpringCGLIB$$e134a44d.getFeedFlows(<generated>)
 at com.thinkbiganalytics.feedmgr.nifi.cache.NifiFlowCacheImpl.rebuildAll(NifiFlowCacheImpl.java:329)
 at com.thinkbiganalytics.feedmgr.rest.controller.NifiProvenanceRestController.resetCache(NifiProvenanceRestController.java:144)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
 at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
 at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
 at org.glassfish.jersey.process.internal.RequestScop

[Bimal Mehta]

I checked port mapping in all property files as mentioned in other posts. All of them are fine.

I also tried the below:

http://localhost:8400/proxy/v1/metadata/nifi-provenance/nifi-flow-cache/reset-cache

I get the following error

{"status":"error","message":"java.lang.IllegalStateException","developerMessage":null,"url":"/api/v1/metadata/nifi-provenance/nifi-flow-cache/reset-cache","validationError":false,"properties":{},"validationErrors":[]}

[Scott Reisdorf]

Can you make sure NiFi is up and running?
The logs indicate a "Connection Exception"

[Bimal Mehta]

I already restarted NiFi twice using sudo service NiFi restart.
It shows running in CLI, but when I try to access it using the browser it does not open .

[Gregory Hart]

Hi Bimal,

This sounds like an issue with NiFi and not with Kylo. In the Kylo sandbox you can find the NiFi logs at /var/log/nifi/nifi-app.log. If the errors indicate a problem with NiFi then you'll need to content the NiFi mailing list or contact Think Big Analytics for a support contract.

[Bimal Mehta]

The NiFi log is not giving any errors that's why it is weird.
Night shows running when checked from command prompt. It won't work when trying from browser. The only place where I am getting this error is in Kylo. Not sure what went wrong

[Gregory Hart]

Hi Bimal,

Could you try the command below within the Kylo sandbox to verify if the issue is with Kylo or with NiFi?

curl http://localhost:8079/

[Bimal Mehta]

That's what I said. The URL won't open when tried from browser. At the same time I can't see any errors in NiFi log too.
And NiFi shows up and running too when checked in shell

[Bimal Mehta]

When checked in shell it says:

Command Apache NiFi is currently running, listening to Bootstrap on Port 33954

[Gregory Hart]

Hi Bimal,

Sorry, I can't be of more help. If you find an issue with Kylo then I'd be happy to help.

[Bimal Mehta]

I got this error in the NiFi log after restarting everything. Not sure if its a NiFi or Kylo issue?

2017-10-23 17:30:07,973 ERROR [main] c.t.n.p.r.KyloProvenanceEventRepositoryUtil Error loading backup

[Bimal Mehta]

You are right. Yea it seems NiFi is the cause.

It created a bootsrap log. I see a bunch of errors there which starts NiFi and gives OutofMemory error and than shuts down the services.

I will check with NiFi community. If you have some thoughts on what I can do it will be helpful.

Thank you for the help so far! Appreciate it.

2017-10-23 17:32:51,614 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:32:51,631 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "FeedStatisticsManager-GatherStats-0"
2017-10-23 17:32:59,912 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:32:59,912 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "FileSystemRepository Workers Thread-4"
2017-10-23 17:33:11,457 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:11,457 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ InactivityMonitor ReadCheckTimer"
2017-10-23 17:33:13,110 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:13,110 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Provenance Maintenance Thread-2"
2017-10-23 17:33:21,289 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:21,289 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "logback-2"
2017-10-23 17:33:27,967 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:27,967 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Provenance Maintenance Thread-1"
2017-10-23 17:33:34,337 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:34,337 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Provenance Maintenance Thread-3"
2017-10-23 17:33:40,620 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:40,620 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ InactivityMonitor WriteCheckTimer"
2017-10-23 17:33:43,801 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:43,801 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616@33140"
2017-10-23 17:33:50,137 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:50,137 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Event-Driven Process Thread-5"
2017-10-23 17:33:51,785 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:51,785 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "FileSystemRepository Workers Thread-2"
2017-10-23 17:33:53,380 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:53,381 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Write-Ahead Local State Provider Maintenance"
2017-10-23 17:33:55,045 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:55,046 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-15-thread-1"
2017-10-23 17:33:59,963 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:33:59,963 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Listen to Bootstrap"
2017-10-23 17:34:06,550 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:34:06,551 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Event-Driven Process Thread-2"
2017-10-23 17:34:08,259 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:34:08,260 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-Driven Process Thread-1"
2017-10-23 17:34:41,410 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:34:41,410 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Flow Service Tasks Thread-2"
2017-10-23 17:34:53,196 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:34:53,197 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DefaultMessageListenerContainer-1"
2017-10-23 17:35:01,989 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:35:01,989 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Cleanup Archive for default"
2017-10-23 17:35:05,776 ERROR [NiFi logging handler] org.apache.nifi.StdErr
2017-10-23 17:35:05,777 ERROR [NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"
2017-10-23 17:35:08,075 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi
2017-10-23 17:47:27,350 INFO [main] o.a.n.b.NotificationServiceManager Successfully loaded the following 0 services: []
2017-10-23 17:47:27,357 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STARTED
2017-10-23 17:47:27,357 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_STOPPED
2017-10-23 17:47:27,357 INFO [main] org.apache.nifi.bootstrap.RunNiFi Registered no Notification Services for Notification Type NIFI_DIED
2017-10-23 17:47:27,486 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is not running

[Bimal Mehta]

Thank you guys, I was able to increase the memory and restart NiFi.

As soon as I started it some 40k jobs that were in queue got triggered.

How do I cancel them all in one go?

[Gregory Hart]

Hi Bimal,

Glad you were able to resolve your NiFi issue!

You'll need to go into NiFi and empty the queues containing these flow files. Kylo currently doesn't have a way to automate this.

[Bimal Mehta]

Hi,

I deleted the queues and the deactivated NiFi feeds.

But now I am unable to import templates in Kylo.

I get the following error in Kylo Log

2017-10-24 01:24:25 ERROR main:NifiFlowCacheImpl:405 - Error attempting to build cache.  The system will attempt to retry 9 more times.  Next attempt to rebuild in 5 seconds.  The error was: javax.jcr.ItemNotFoundException: ff80bc37505d64ef71cd1b-0e9c-471a-8969-58616bcd94db.

I tried to reset the cache using 

http://localhost:8400/proxy/v1/metadata/nifi-provenance/nifi-flow-cache/reset-cache

but rest does not happen and I get below error

Whitelabel Error Page

This application has no explicit mapping for /error, so you are seeing this as a fallback.

Tue Oct 24 01:28:34 UTC 2017

There was an unexpected error (type=Internal Server Error, status=500).

Internal Server Error

Also attached is the detailed failure log of Kylo Services. None of the standard templates appear now when creating a new feed.

[Gregory Hart]

Hi Bimal,

It looks like part of the kylo database may have been corrupted when your computer restarted. Specifically, it's trying to load a template and is missing information. You can try editing the templates in the Kylo UI and saving them to see if that fixes the error. Otherwise it might be easier to export your feeds and start with a fresh sandbox.