Issues with Nifi / ActiveMQ integration

[Mathieu Marie]

Hi,

I am trying to execute a feed by providing a sample file from the UI on my newly installed Kylo.
The data_ingest feed never appears in the 'Profile' tab as running, nor failed, nor anything.
I have no exceptions in the kylo-services log.

In the NiFi log I do have errors and exceptions :

      2017-08-04 18:54:22,961 ERROR [StandardProcessScheduler Thread-6] c.t.n.v.c.s.SpringContextLoaderService SpringContextLoaderService[id=ad7c86df-015d-1000-0ef1-124faf7d0cc2] Failed to load spring configurations: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sendJmsMessage': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.springframework.jms.core.JmsMessagingTemplate com.thinkbiganalytics.activemq.SendJmsMessage.jmsMessagingTemplate; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jmsMessagingTemplate' defined in com.thinkbiganalytics.activemq.config.ActiveMqConfig: Unsatisfied dependency expressed through constructor argument with index 0 of type [javax.jms.ConnectionFactory]: Error creating bean with name 'connectionFactory' defined in com.thinkbiganalytics.activemq.config.ActiveMqConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.jms.ConnectionFactory]: Factory method 'connectionFactory' threw exception; nested exception is java.lang.NullPointerException; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connectionFactory' defined in com.thinkbiganalytics.activemq.config.ActiveMqConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.jms.ConnectionFactory]: Factory method 'connectionFactory' threw exception; nested exception is java.lang.NullPointerException

      2017-08-04 18:54:22,962 ERROR [StandardProcessScheduler Thread-6] o.a.n.c.s.StandardControllerServiceNode Failed to invoke @OnEnabled method of SpringContextLoaderService[id=ad7c86df-015d-1000-0ef1-124faf7d0cc2] due to org.apache.nifi.reporting.InitializationException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sendJmsMessage': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.springframework.jms.core.JmsMessagingTemplate com.thinkbiganalytics.activemq.SendJmsMessage.jmsMessagingTemplate; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jmsMessagingTemplate' defined in com.thinkbiganalytics.activemq.config.ActiveMqConfig: Unsatisfied dependency expressed through constructor argument with index 0 of type [javax.jms.ConnectionFactory]: Error creating bean with name 'connectionFactory' defined in com.thinkbiganalytics.activemq.config.ActiveMqConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.jms.ConnectionFactory]: Factory method 'connectionFactory' threw exception; nested exception is java.lang.NullPointerException; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connectionFactory' defined in com.thinkbiganalytics.activemq.config.ActiveMqConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.jms.ConnectionFactory]: Factory method 'connectionFactory' threw exception; nested exception is java.lang.NullPointerException


Both seems related to activemq, and by going to the ActiveMQ console at http://localhost:8161/admin/topics.jsp, I can see that messages are not dequeued.

Any clue of what is going wrong?
Regards,
Mathieu

[Greg Hart]

Hi Mathieu,

NiFi uses the jms.activemq.broker.url property from /opt/nifi/ext-config/config.properties to determine the ActiveMQ URL. Could you verify this property is set correctly? The default value is tcp://localhost:61616.

[Mathieu Marie]

Hi Greg,

Yes it does :

$> grep tcp /opt/nifi/ext-config/config.properties
jms.activemq.broker.url=tcp://localhost:61616 

$> ll /opt/nifi/ext-config/config.properties
-rwxr-xr-x. 1 nifi nifi 1977 13 juil. 00:36 /opt/nifi/ext-config/config.properties

Regards,

Mathieu

[Greg Hart]

Hi Mathieu,

Can you attach a screenshot of your ActiveMQ Queues page? 

[Mathieu Marie]

Hi Greg,

Sure, I attached a screenshot.
Many thanks for your support.

Regards,
Mathieu

[Mathieu Marie]

Hi Greg,

Here is another screenshot with the activeMQ queues page this time.

Regards,
Mathieu

[Mathieu Marie]

Hi all,

I've been able to fix many issue son my setup today, but I still have that one issue that produces that error message in a loop :

SpringContextLoaderService[id=ad7c86df-015d-1000-0ef1-124faf7d0cc2] Failed to load spring configurations: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sendJmsMessage': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.springframework.jms.core.JmsMessagingTemplate com.thinkbiganalytics.activemq.SendJmsMessage.jmsMessagingTemplate; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jmsMessagingTemplate' defined in com.thinkbiganalytics.activemq.config.ActiveMqConfig: Unsatisfied dependency expressed through constructor argument with index 0 of type [javax.jms.ConnectionFactory]: Error creating bean with name 'connectionFactory' defined in com.thinkbiganalytics.activemq.config.ActiveMqConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.jms.ConnectionFactory]: Factory method 'connectionFactory' threw exception; nested exception is java.lang.NullPointerException; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'connectionFactory' defined in com.thinkbiganalytics.activemq.config.ActiveMqConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.jms.ConnectionFactory]: Factory method 'connectionFactory' threw exception; nested exception is java.lang.NullPointerException

From the error message, it seems NiFi is not able to find javax.jms.ConnectionFactory?
Do I need to copy a jar somewhere or configure a path for it?

I realized that my SpringContextLoaderService has no value set for "Configuration Classes". Should I set it to something? I've not found anything yet in the doc that talked about it.

Thanks for any help on that topic.
Regards,
Mathieu

[Greg Hart]

Hi Mathieu,

Could you verify that you have the following lines at the end of your /opt/nifi/current/conf/bootstrap.conf file?

# Set kylo nifi configuration file directory path

java.arg.15=-Dkylo.nifi.configPath=/opt/nifi/ext-config

And that the /opt/nifi/ext-config directory and the following permissions?

drwxr-xr-x 2 nifi users 31 Aug  7 17:59 /opt/nifi/ext-config
-rwxr-xr-x 1 nifi users 1977 Jul 17 20:52 /opt/nifi/ext-config/config.properties

[Mathieu Marie]

Hi Greg,

I do not have the java.arg.15 property as you stated :

Mine is :

    # Sets the provider of SecureRandom to /dev/urandom to prevent blocking on VMs
    java.arg.15=-Djava.security.egd=file:/dev/urandom

I just tried with :

    java.arg.16=-Dkylo.nifi.configPath=/opt/nifi/ext-config

And that was the issue.
All my remaining issues disappeared (I wasn't able to delete feeds either...).

Many thanks for the quick resolution.

Regards,
Mathieu





[sidoine kakeuh fosso]

hello Mathieu, Greg,

i'm trying to make working my kylo 0.8.3 in my hortonworks sandbox on azure.

I'm having the same active MQ screen shoot like you (see attachment).

how did you solve it please?

on another hand, here is my ext-config content and permission, and also my bootstrap content on the line pointed by greg:

[root@sandbox ext-config]# ll

total 4

-rwxr-xr-x 1 nifi nifi 2014 Sep  8 10:41 config.properties

# Set kylo nifi configuration file directory path

java.arg.15=-Dkylo.nifi.configPath=/opt/nifi/ext-config

in my kylo-services.log, i'm having below error:

2017-09-11 08:43:50 ERROR DefaultMessageListenerContainer-1:DefaultMessageListenerContainer:941 - Could not refresh JMS Connection for destination 'thinkbig.feed-manager' - retrying using FixedBackOff{interval=5000, currentAttempts=956, maxAttempts=unlimited}. Cause: Error while attempting to add new Connection to the pool; nested exception is javax.jms.JMSException: Could not connect to broker URL: tcp://localhost:61616. Reason: java.net.ConnectException: Connection refused (Connection refused)

thanks for your help.

[b...@adazza.com]

https://groups.google.com/forum/#!topic/kylo-community/rJPJSyof88U

I think this might be an example of the bug we found in the above thread. 

The bean wiring error is b/c your don't have the correct spring profile set for the ext config in nifi. 

https://github.com/Teradata/kylo/blob/master/integrations/nifi/nifi-nar-bundles/nifi-provenance-repo-bundle/nifi-provenance-repo/src/main/resources/config.properties#L20

[Ruslans Uralovs]

sidoine kakeuh fosso,

Connection refused usually means that either service is not running or your are connecting to wrong host/port.

Can you check whether ActiveMQ service is up, e.g. service activemq status

And also check whether you are connecting to correct host port, from the error log we can see you are connecting to localhost. Is ActiveMQ running on the same host with Nifi?

[sidoine kakeuh fosso]

hello, 

thanks for your quick response.

from my understanding, the bug has been fixed in 0.8.3.x release.

I thinks that my ext-config in nifi is up to data, see below:

[root@sandbox ext-config]# cat config.properties 

spring.profiles.active=jms-activemq

jms.activemq.broker.url=tcp://localhost:61616

## Back up location to write the Feed stats data if NiFi goes down

kylo.provenance.cache.location=/opt/nifi/feed-event-statistics.gz

## The maximum number of starting flow files per feed during the given run interval to send to ops manager

kylo.provenance.max.starting.events=5

kylo.provenance.event.count.throttle.threshold=15

## Throttle timefame used to check the rolling window to determine if rapid fire is occurring

kylo.provenance.event.throttle.threshold.time.millis=1000

## run interval to gather stats and send to ops manager

kylo.provenance.run.interval.millis=3000

## Child flow files produced by events  matching the EventType and processor class will not be processed

kylo.provenance.orphan.child.flowfile.processors={"CLONE":["ConvertCSVToAvro"]}

thanks for help.

[sidoine kakeuh fosso]

hello Ruslans,

my activeMQ is running, see below:

[root@sandbox /]# service activemq status

INFO: Loading '/etc/default/activemq'

INFO: Using java '/opt/java/current/bin/java'

ActiveMQ is running (pid '63638')

i've installed kylo in a single node hortonworks sandbox, so nifi and active mq are installed on the same node.

below is a grep on port 61616 of the sandbox host:

[root@sandbox /]# netstat -an|grep 61616

tcp        0      0 127.0.0.1:50624             127.0.0.1:61616             ESTABLISHED 

tcp        0      0 127.0.0.1:50548             127.0.0.1:61616             ESTABLISHED 

tcp        0      0 127.0.0.1:50760             127.0.0.1:61616             ESTABLISHED 

tcp        0      0 :::61616                    :::*                        LISTEN      

tcp        0      0 ::ffff:127.0.0.1:61616      ::ffff:127.0.0.1:50352      ESTABLISHED 

tcp        0      0 ::ffff:127.0.0.1:61616      ::ffff:127.0.0.1:50548      ESTABLISHED 

tcp        0      0 ::ffff:127.0.0.1:61616      ::ffff:127.0.0.1:50624      ESTABLISHED 

tcp        0      0 ::ffff:127.0.0.1:50352      ::ffff:127.0.0.1:61616      ESTABLISHED 

tcp        0      0 ::ffff:127.0.0.1:61616      ::ffff:127.0.0.1:50760      ESTABLISHED 

thanks for help.

sidoine.

[Ruslans Uralovs]

sidoine kakeuh fosso 

Can you restart kylo-services and attach your kylo-services.log please

Thanks

[sidoine kakeuh fosso]

hello,

yes sure, you have it in attachment.

i'm also wordering about an observation in my bootstrap.conf:

i have java.arg.15 on two distinct line, is that ok? or the last one should be java.arg.16? (see below):

# Sets the provider of SecureRandom to /dev/urandom to prevent blocking on VMs

java.arg.15=-Djava.security.egd=file:/dev/urandom

# Set kylo nifi configuration file directory path

java.arg.15=-Dkylo.nifi.configPath=/opt/nifi/ext-config

[root@sandbox conf]# cat bootstrap.conf 

#

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#   http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

#

# Java command to use when running NiFi

java=java

# Username to use when running NiFi. This value will be ignored on Windows.

run.as=

# Configure where NiFi's lib and conf directories live

lib.dir=./lib

conf.dir=./conf

# How long to wait after telling NiFi to shutdown before explicitly killing the Process

graceful.shutdown.seconds=20

# Disable JSR 199 so that we can use JSP's without running a JDK

java.arg.1=-Dorg.apache.jasper.compiler.disablejsr199=true

# JVM memory settings

java.arg.2=-Xms512m

java.arg.3=-Xmx512m

# Enable Remote Debugging

#java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000

java.arg.4=-Djava.net.preferIPv4Stack=true

# allowRestrictedHeaders is required for Cluster/Node communications to work properly

java.arg.5=-Dsun.net.http.allowRestrictedHeaders=true

java.arg.6=-Djava.protocol.handler.pkgs=sun.net.www.protocol

# The G1GC is still considered experimental but has proven to be very advantageous in providing great

# performance without significant "stop-the-world" delays.

java.arg.13=-XX:+UseG1GC

#Set headless mode by default

java.arg.14=-Djava.awt.headless=true

# Master key in hexadecimal format for encrypted sensitive configuration values

nifi.bootstrap.sensitive.key=

# Sets the provider of SecureRandom to /dev/urandom to prevent blocking on VMs

java.arg.15=-Djava.security.egd=file:/dev/urandom

###

# Notification Services for notifying interested parties when NiFi is stopped, started, dies

###

# XML File that contains the definitions of the notification services

notification.services.file=./conf/bootstrap-notification-services.xml

# In the case that we are unable to send a notification for an event, how many times should we retry?

notification.max.attempts=5

# Comma-separated list of identifiers that are present in the notification.services.file; which services should be used to notify when NiFi is started?

#nifi.start.notification.services=email-notification

# Comma-separated list of identifiers that are present in the notification.services.file; which services should be used to notify when NiFi is stopped?

#nifi.stop.notification.services=email-notification

# Comma-separated list of identifiers that are present in the notification.services.file; which services should be used to notify when NiFi dies?

#nifi.dead.notification.services=email-notification

# Set kylo nifi configuration file directory path

java.arg.15=-Dkylo.nifi.configPath=/opt/nifi/ext-config

thanks.

[Ruslans Uralovs]

sidoine kakeuh fosso 

Unless you notice that Kylo doesn't display information in Ops Manager correctly you can disregard JMS "Session is closed" errors. Here they most likely mean that JMS connection was broken temporarily.

And yes, you certainly need to increment java argument each time you add a new one.

[sidoine kakeuh fosso]

i've updated the java argument, but i'm still facing the same problem.

find in attachment the both nifi and kylo-services.log files for help.

thanks a lot.

sidoine.

[Ruslans Uralovs]

The problems are actually totally different.

I suggest you start a new thread describing your problems