Integration of Kylo 0.9.0 with Existing Nifi Cluster (SSL enabled)

248 views
Skip to first unread message

nelson.aras...@verizon.com

unread,
Apr 23, 2018, 6:44:36 PM4/23/18
to Kylo Community
Hi,

We are trying to integrate Kylo with existing Nifi cluster (SSL enabled) and we are receiving this message in the logs. Seems like Nifi is not getting connected with Kylo. All the steps in manual deployment guide were followed during this set up. 

2018-04-23 13:27:15 INFO  pool-9-thread-1:DefaultServicesApplicationStartup:108 - Running startup listener PreconditionPolicyRuleCache
2018-04-23 13:27:15 INFO  pool-9-thread-3:DefaultServicesApplicationStartup:108 - Running startup listener FeedFailureServiceStatusStartupListener
2018-04-23 13:27:15 INFO  pool-9-thread-6:DefaultServicesApplicationStartup:108 - Running startup listener StandardizationStartupListener
2018-04-23 13:27:15 INFO  pool-9-thread-5:DefaultServicesApplicationStartup:108 - Running startup listener ValidationStartupListener
2018-04-23 13:27:15 INFO  pool-9-thread-2:DefaultServicesApplicationStartup:108 - Running startup listener NifiFlowCacheImpl
2018-04-23 13:27:15 INFO  pool-9-thread-4:DefaultServicesApplicationStartup:108 - Running startup listener ClusterServiceStartupListener
2018-04-23 13:27:15 INFO  pool-9-thread-1:ReflectionPolicyAnnotationDiscoverer:48 - Found 2 classes annotated with interface com.thinkbiganalytics.policy.precondition.PreconditionPolicy
2018-04-23 13:27:15 INFO  pool-9-thread-2:NifiFlowCacheImpl:231 - Check and Initialize NiFi Flow Cache. Kylo up to date:true, NiFi Connected:false, Cache needs loading:true
2018-04-23 13:27:15 INFO  pool-9-thread-5:ReflectionPolicyAnnotationDiscoverer:48 - Found 13 classes annotated with interface com.thinkbiganalytics.policy.validation.Validator
2018-04-23 13:27:15 INFO  pool-9-thread-6:ReflectionPolicyAnnotationDiscoverer:48 - Found 11 classes annotated with interface com.thinkbiganalytics.policy.standardization.Standardizer
2018-04-23 13:27:16 INFO  DefaultMessageListenerContainer-1:ProvenanceEventReceiver:182 - About to process batch: 3e968434-1986-4e19-9cd6-aca851f5e3fe,  62 events from the thinkbig.feed-manager queue
2018-04-23 13:27:16 INFO  main:KyloServerApplication:57 - Started KyloServerApplication in 88.587 seconds (JVM running for 91.945)
2018-04-23 13:27:16 INFO  DefaultMessageListenerContainer-1:ProvenanceEventReceiver:212 - NiFi is not up yet. Sending batch 3e968434-1986-4e19-9cd6-aca851f5e3fe back to JMS for later dequeue
2018-04-23 13:27:16 WARN  DefaultMessageListenerContainer-1:DefaultMessageListenerContainer:936 - Execution of JMS message listener failed, and no ErrorHandler has been set.
org.springframework.jms.listener.adapter.ListenerExecutionFailedException: Listener method 'public void com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.ProvenanceEventReceiver.receiveEvents(com.thinkbiganalytics.nifi.provenance.model.ProvenanceEventRecordDTOHolder)' threw exception; nested exception is com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.JmsProcessingException: Unable to process events.  NiFi is either not up, or there is an error trying to populate the Kylo NiFi Flow Cache.
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:98)
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:66)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:721)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:681)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:315)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:253)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1158)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1150)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1047)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.JmsProcessingException: Unable to process events.  NiFi is either not up, or there is an error trying to populate the Kylo NiFi Flow Cache.
        at com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.ProvenanceEventReceiver.receiveEvents(ProvenanceEventReceiver.java:213)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:115)
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:90)
        ... 10 more


Binh Nguyen Van

unread,
Apr 23, 2018, 7:09:06 PM4/23/18
to Kylo Community
What version of NiFi are you using?
If you are using NiFi 1.5 and using load balancer then you may hit this issue https://issues.apache.org/jira/browse/NIFI-4761

-Binh

Ruslans Uralovs

unread,
Apr 24, 2018, 5:49:09 AM4/24/18
to Kylo Community
The problem is that Kylo cannot connect to Nifi.
See following doc on connecting to Nifi with SSL: http://kylo.readthedocs.io/en/latest/security/ConfigureNiFiWithSSL.html
Specifically see point 10 which describes which changes need to be made to kylo-services application.properties configuration file.

Ramprabhu Premkumar

unread,
Apr 24, 2018, 11:08:38 AM4/24/18
to Kylo Community
Yes, we are using Nifi 1.5, however we are not using load balancer.

Ramprabhu Premkumar

unread,
Apr 24, 2018, 11:09:30 AM4/24/18
to Kylo Community
The steps mention in the link (point 10) has been followed.

Greg Hart

unread,
Apr 24, 2018, 12:47:02 PM4/24/18
to Kylo Community
Hi Ramprabhu,

1) Could you share the nifi.rest.* properties in your /opt/kylo/kylo-services/conf/application.properties file?

2) Please go to the Templates page in Kylo and click on any template. Then please share your /var/log/kylo-services/kylo-services.log file.

3) Please verify that the truststore and keystore files you're using are readable by the kylo user.

nelson.aras...@verizon.com

unread,
Apr 24, 2018, 1:11:27 PM4/24/18
to Kylo Community
Hi Greg ,

Please find the details below .  

1) Could you share the nifi.rest.* properties in your /opt/kylo/kylo-services/conf/application.properties file?

nifi.rest.host=test1
nifi.rest.port=8888

###
# NiFi Https configuration below
#
# Follow directions here: http://kylo.readthedocs.io/en/v0.8.2/how-to-guides/ConfigureNiFiWithSSL.html to setup certificates and properties in NiFi
#
### The port should match the port found in the /opt/nifi/current/conf/nifi.properties (nifi.web.https.port)
#nifi.rest.port=9443
nifi.rest.https=true
nifi.rest.useConnectionPooling=false
nifi.rest.truststorePath=/apps/opt/hdp/kylo/nifi/keystore/truststore.ts
#####the truststore password below needs to match that found in the nifi.properties file (nifi.security.truststorePasswd)
nifi.rest.truststorePassword=NiFiKS
nifi.rest.truststoreType=JKS
nifi.rest.keystorePath=/apps/opt/hdp/kylo/nifi/keystore/test1.pkcs12
###value found in the .password file /opt/nifi/data/ssl/CN=kylo_OU=NIFI.password
nifi.rest.keystorePassword=NiFiKS
nifi.rest.keystoreType=PKCS12

2) Please go to the Templates page in Kylo and click on any template. Then please share your /var/log/kylo-services/kylo-services.log file. - Attached log file and screenshot 

3) Please verify that the truststore and keystore files you're using are readable by the kylo user. - Yes . it has 644 access . 



template_error.PNG
kylo_services_log.txt

Greg Hart

unread,
Apr 24, 2018, 2:32:24 PM4/24/18
to Kylo Community
Hi Nelson,

After step 10 on the NiFi SSL document are instructions for verifying that the client certificate is configured properly in NiFi. Could you try the "Importing the Client Cert on the Mac" and "Accessing NiFi under SSL" sections to verify that NiFi is configured correctly?

Ramprabhu Premkumar

unread,
Apr 26, 2018, 11:24:38 AM4/26/18
to Kylo Community
Hi Greg,

Thank you, this helped solve the problem. We had access issue when Kylo was trying to connect to Nifi (after the authentication takes place). Access was granted in Nifi and it worked.

However, we still have issues in importing the reusable templates from Kylo Templates page. We are trying to import data_ingest.zip template and it is not getting registered with Kylo. One thing to be noted here is that the Kylo already had these templates registered and we have configured it work with new instance of NiFi (version 1.5). I am attaching the screenshot and logs from Kylo Services. 

Ramprabhu Premkumar
kylo_template_import.PNG
kylo_services.txt

Ramprabhu Premkumar

unread,
Apr 26, 2018, 11:49:26 AM4/26/18
to Kylo Community
Greg,

Wanted to add more info on this issue. 

We tried to reset the cache and we are not successful as there is no flow cache created yet from NiFi.

We had a lot of feeds created in Kylo by integrating with another instance of NiFi and we are now integrating with another instance of NiFi and having this issue.

More from the logs when trying to import the template.


2018-04-26 11:15:48 INFO  DefaultMessageListenerContainer-1:ProvenanceEventReceiver:182 - About to process batch: 7ab03a12-2721-41f4-97b1-9617e5a654ee,  72 events from the thinkbig.feed-manager queue
2018-04-26 11:15:48 INFO  DefaultMessageListenerContainer-1:ProvenanceEventReceiver:212 - NiFi is not up yet. Sending batch 7ab03a12-2721-41f4-97b1-9617e5a654ee back to JMS for later dequeue
2018-04-26 11:15:48 WARN  DefaultMessageListenerContainer-1:DefaultMessageListenerContainer:936 - Execution of JMS message listener failed, and no ErrorHandler has been set.
org.springframework.jms.listener.adapter.ListenerExecutionFailedException: Listener method 'public void com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.ProvenanceEventReceiver.receiveEvents(com.thinkbiganalytics.nifi.provenance.model.ProvenanceEventRecordDTOHolder)' threw exception; nested exception is com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.JmsProcessingException: Unable to process events.  NiFi is either not up, or there is an error trying to populate the Kylo NiFi Flow Cache.
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:98)
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:66)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:721)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:681)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:315)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:253)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1158)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1150)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1047)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.JmsProcessingException: Unable to process events.  NiFi is either not up, or there is an error trying to populate the Kylo NiFi Flow Cache.
        at com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.ProvenanceEventReceiver.receiveEvents(ProvenanceEventReceiver.java:213)
        at sun.reflect.GeneratedMethodAccessor431.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:115)
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:90)
        ... 10 more
2018-04-26 11:15:49 ERROR http-nio-8963-exec-3:ImportTemplateArchive:337 - Error importing reusable template from archive data_ingest.zip.
2018-04-26 11:15:49 DEBUG http-nio-8963-exec-9:JwtRememberMeServices:118 - Remember-me cookie detected
2018-04-26 11:15:49 DEBUG http-nio-8963-exec-9:JwtRememberMeServices:133 - Remember-me cookie accepted
2018-04-26 11:15:49 DEBUG http-nio-8963-exec-2:JwtRememberMeServices:118 - Remember-me cookie detected
2018-04-26 11:15:49 DEBUG http-nio-8963-exec-2:JwtRememberMeServices:133 - Remember-me cookie accepted
2018-04-26 11:15:49 INFO  http-nio-8963-exec-3:AbstractImportTemplateRoutine:157 - Attempting to import Nifi Template: Data Ingest for file data_ingest.zip
2018-04-26 11:15:49 INFO  DefaultMessageListenerContainer-1:NifiStatsJmsReceiver:281 - NiFi is not up yet.  Sending back to JMS for later dequeue
2018-04-26 11:15:49 WARN  DefaultMessageListenerContainer-1:DefaultMessageListenerContainer:936 - Execution of JMS message listener failed, and no ErrorHandler has been set.
org.springframework.jms.listener.adapter.ListenerExecutionFailedException: Listener method 'public void com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.NifiStatsJmsReceiver.receiveTopic(com.thinkbiganalytics.nifi.provenance.model.stats.AggregatedFeedProcessorStatisticsHolder)' threw exception; nested exception is com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.JmsProcessingException: Unable to process Statistics Events.  NiFi is either not up, or there is an error trying to populate the Kylo NiFi Flow Cache.
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:98)
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:66)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:721)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:681)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:315)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:253)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1158)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1150)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1047)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.JmsProcessingException: Unable to process Statistics Events.  NiFi is either not up, or there is an error trying to populate the Kylo NiFi Flow Cache.
        at com.thinkbiganalytics.metadata.jobrepo.nifi.provenance.NifiStatsJmsReceiver.receiveTopic(NifiStatsJmsReceiver.java:282)
        at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:115)
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:90)
        ... 10 more

Ramprabhu Premkumar

Ruslans Uralovs

unread,
Apr 26, 2018, 12:13:52 PM4/26/18
to Kylo Community
In the previous post you mention that issue was resolved after you granted access in Nifi. What did you change? Is it possible that you granted access only to a subset of Nifi API?

Ramprabhu Premkumar

unread,
Apr 26, 2018, 12:56:01 PM4/26/18
to Kylo Community
We have granted the permissions in Nifi. During the import of template from Kylo, processor group is created in NiFi and when it registers the template with Kylo it is failing and hence removing the processor group created in NiFi.


Greg Hart

unread,
May 1, 2018, 12:42:36 PM5/1/18
to Kylo Community
Hi Ramprubhu,

Could you open the following URL in your browser using the Kylo certificate you created for NiFi? You may need to change the host and port to your NiFi instance.

Please attach the response here. Under processGroupFlow.flow.processGroups, every component.name entry should be non-null.

Ramprabhu Premkumar

unread,
May 1, 2018, 1:27:53 PM5/1/18
to Kylo Community
Hi Greg,

Please find the response attached.

Ramprabhu Premkumar
NiFi_Resp.txt

Greg Hart

unread,
May 1, 2018, 1:56:08 PM5/1/18
to Kylo Community
Hi Ramprubhu,

The response you sent doesn't match what I'd expect for NiFi v1.5. Could you try again with this URL?

Ramprabhu Premkumar

unread,
May 1, 2018, 4:43:22 PM5/1/18
to Kylo Community
Greg,

Please find attached the response.

Ramprabhu Premkumar
NiFi_Flow_Resp.txt

Greg Hart

unread,
May 1, 2018, 6:26:48 PM5/1/18
to Kylo Community
Hi Ramprabhu,

It looks like Kylo doesn't have access to some of the top-level process groups and this is causing problems. I've opened a JIRA to track this issue:

As a work-around, could you grant read access to all top-level process groups? Alternatively, you can setup a new, blank NiFi instance specifically for Kylo. You can also contact Think Big Analytics for paid support if you would like a patch to fix the issue.

Ramprabhu Premkumar

unread,
May 3, 2018, 12:16:43 PM5/3/18
to Kylo Community
Hi Greg,

We have granted view and modify access to all process groups in NiFi and still have the same issue. 

As per your recommendation we will try integrating with a new blank NiFi instance specifically for Kylo.

Thanks
Ramprabhu Premkumar

Bharat Solanki

unread,
Sep 9, 2019, 2:04:31 AM9/9/19
to Kylo Community
Hi Ram,

I need small help. I am aslo facing the same issue. could you please tell me the what kind of access you have given on nifi side?

Thanks
Bharat
Reply all
Reply to author
Forward
0 new messages