Error importing reusable template

[nel...@leantaas.com]

I have created reusable template and importing it to Kylo.

I am getting this error

An error occurred attempting to access: /api/v2/feedmgr/admin/import-template. com.thinkbiganalytics.nifi.feedmgr.TemplateCreationException: javax.ws.rs.ClientErrorException: HTTP 409 Conflict

In kylo-services.log

Reusable flow, attempt to mark the connection ports as running.
2018-01-10 08:51:09 ERROR http-nio-8420-exec-10:BaseExceptionMapper:46 - toResponse() caught Exception: com.thinkbiganalytics.nifi.rest.client.NifiClientRuntimeException, com.thinkbiganalytics.nifi.feedmgr.TemplateCreationException: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
com.thinkbiganalytics.nifi.rest.client.NifiClientRuntimeException: com.thinkbiganalytics.nifi.feedmgr.TemplateCreationException: javax.ws.rs.ClientErrorException: HTTP 409 Conflict
at com.thinkbiganalytics.nifi.rest.client.NifiRestClientExceptionTranslator.translateException(NifiRestClientExceptionTranslator.java:58)
at com.thinkbiganalytics.nifi.rest.config.NifiRestClientAroundAspect.NifiRestClientAroundAspect(NifiRestClientAroundAspect.java:50)
at sun.reflect.GeneratedMethodAccessor740.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:620)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:609)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.thinkbiganalytics.nifi.rest.client.LegacyNifiRestClient$$EnhancerBySpringCGLIB$$958252c5.createNewTemplateInstance(<generated>)
at com.thinkbiganalytics.feedmgr.service.template.ExportImportTemplateService.importNifiTemplate(ExportImportTemplateService.java:927)
at com.thinkbiganalytics.feedmgr.service.template.ExportImportTemplateService.importTemplate(ExportImportTemplateService.java:595)
at com.thinkbiganalytics.feedmgr.rest.controller.AdminControllerV2.uploadTemplate(AdminControllerV2.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)

This started happening after I have removed reusable template from nifi. Now I cant import any reusable template.

Could you please help to troubleshoot?

Thank you,
Nella

[Ruslans Uralovs]

Hi Nella

This conflict is usually because some controller services in Nifi are in invalid state. The fix would be to enable controller services in Nifi and import the template again.

[nel...@leantaas.com]

Ruslans,

I have ensured that all services are in valid state and reimported the teamplate.

I got the same error.

Any idea what else I could try?

Thank you,
Nella

[Jagrut Sharma]

Nella - Upload a template in NiFi with the same name that you deleted. 

Then, try uploading your updated reusable template via Kylo and select 'overwrite existing' option.

Thanks.

--

Jagrut

[nel...@leantaas.com]

Thank you!!

This have not worked either :(

[Ruslans Uralovs]

Can you attach the nifi log for when you are importing the template please

[nel...@leantaas.com]

Ruslans,

This is what gets logged in nifi after I start the import. It imports correctry but not enables the template in nifi.

The template actually works. In Steps it fails to recognize names of nifi processors.

2018-01-12 02:09:50,536 INFO [NiFi Web Server-3794] o.a.nifi.groups.StandardProcessGroup Template[id=c9d9efa5-c866-4058-ad20-00d7967511ab] removed from flow
2018-01-12 02:09:52,138 INFO [NiFi Web Server-3789] o.a.nifi.groups.StandardProcessGroup Template[id=f2c93b7b-abab-43c9-a57c-0217af006593] added to StandardProcessGroup[identifier=4d1e5cff-0160-1000-f27b-1705b0736236]
2018-01-12 02:09:54,268 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T055347+0000_flow.xml.gz to reduce storage usage. currentSize=524093413
2018-01-12 02:09:54,268 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T055350+0000_flow.xml.gz to reduce storage usage. currentSize=523963699
2018-01-12 02:09:54,281 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5c3e35b5 // Another save pending = true
2018-01-12 02:09:54,317 INFO [NiFi Web Server-3793] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling LocalPort[id=670c2644-fbb0-34e9-c94e-756d8a000a50] to run
2018-01-12 02:09:55,842 INFO [NiFi Web Server-3793] o.a.nifi.groups.StandardProcessGroup Connection[Source ID=e7f2976e-0160-1000-6338-cb33af1013b0,Dest ID=670c2644-fbb0-34e9-c94e-756d8a000a50] removed from flow
2018-01-12 02:09:55,898 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@4789d924 checkpointed with 3481 Records and 0 Swap Files in 19 milliseconds (Stop-the-world time = 10 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID 12138
2018-01-12 02:09:56,191 INFO [NiFi Web Server-3782] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled LocalPort[id=670c2644-fbb0-34e9-c94e-756d8a000a50] to run with 1 threads
2018-01-12 02:09:57,050 INFO [NiFi Web Server-3790] o.a.n.c.s.StandardControllerServiceProvider Created Controller Service of type com.thinkbiganalytics.nifi.v2.core.metadata.MetadataProviderSelectorService with identifier 1dfa1aa4-dddb-3d6b-576c-0f4478f390cb
2018-01-12 02:09:57,051 INFO [NiFi Web Server-3790] o.a.nifi.groups.StandardProcessGroup MetadataProviderSelectorService[id=1dfa1aa4-dddb-3d6b-576c-0f4478f390cb] added to StandardProcessGroup[identifier=e8217c09-0160-1000-13cf-c91c49824dab]
2018-01-12 02:09:57,112 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T055351+0000_flow.xml.gz to reduce storage usage. currentSize=524048514
2018-01-12 02:09:57,125 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5c3e35b5 // Another save pending = true
2018-01-12 02:09:57,545 INFO [NiFi Web Server-3792] o.a.nifi.groups.StandardProcessGroup MetadataProviderSelectorService[id=1dfa1aa4-dddb-3d6b-576c-0f4478f390cb] removed from StandardProcessGroup[identifier=e8217c09-0160-1000-13cf-c91c49824dab]
2018-01-12 02:09:57,549 INFO [NiFi Web Server-3793] o.a.n.c.s.StandardControllerServiceProvider Created Controller Service of type com.thinkbiganalytics.nifi.v2.core.metadata.MetadataProviderSelectorService with identifier e8217f2b-0160-1000-5a6a-4d7958580934
2018-01-12 02:09:57,549 INFO [NiFi Web Server-3793] o.a.nifi.groups.StandardProcessGroup MetadataProviderSelectorService[id=e8217f2b-0160-1000-5a6a-4d7958580934] added to StandardProcessGroup[identifier=4d1e5cff-0160-1000-f27b-1705b0736236]
2018-01-12 02:10:00,116 INFO [NiFi Web Server-3794] o.a.nifi.groups.StandardProcessGroup MetadataProviderSelectorService[id=e8217f2b-0160-1000-5a6a-4d7958580934] removed from StandardProcessGroup[identifier=4d1e5cff-0160-1000-f27b-1705b0736236]
2018-01-12 02:10:00,160 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T064955+0000_flow.xml.gz to reduce storage usage. currentSize=524134547
2018-01-12 02:10:00,161 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T065013+0000_flow.xml.gz to reduce storage usage. currentSize=524004759
2018-01-12 02:10:00,182 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5c3e35b5 // Another save pending = true
2018-01-12 02:10:04,422 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T065015+0000_flow.xml.gz to reduce storage usage. currentSize=524091109
2018-01-12 02:10:04,422 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T065037+0000_flow.xml.gz to reduce storage usage. currentSize=523961153
2018-01-12 02:10:04,432 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5c3e35b5 // Another save pending = true
2018-01-12 02:10:07,384 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T065040+0000_flow.xml.gz to reduce storage usage. currentSize=524047322
2018-01-12 02:10:07,394 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5c3e35b5 // Another save pending = true
2018-01-12 02:10:10,289 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T065044+0000_flow.xml.gz to reduce storage usage. currentSize=524133412
2018-01-12 02:10:10,289 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T065056+0000_flow.xml.gz to reduce storage usage. currentSize=524003373
2018-01-12 02:10:10,299 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5c3e35b5 // Another save pending = true
2018-01-12 02:10:10,834 INFO [NiFi Web Server-3790] o.a.n.c.s.StandardProcessScheduler Starting ExecuteSparkJob[id=796fd000-5dfc-3b3b-822c-fdc7cb6ebd61]
2018-01-12 02:10:10,834 INFO [NiFi Web Server-3790] o.a.n.c.s.StandardProcessScheduler Starting ExecuteSparkJob[id=adcc48f1-c7a1-3e5b-4686-777c71773cda]
2018-01-12 02:10:10,834 INFO [NiFi Web Server-3790] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled LocalPort[id=670c2644-fbb0-34e9-1eac-47fc23da48e8] to run with 1 threads
2018-01-12 02:10:10,835 INFO [StandardProcessScheduler Thread-6] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled ExecuteSparkJob[id=adcc48f1-c7a1-3e5b-4686-777c71773cda] to run with 1 threads
2018-01-12 02:10:10,835 INFO [StandardProcessScheduler Thread-5] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled ExecuteSparkJob[id=796fd000-5dfc-3b3b-822c-fdc7cb6ebd61] to run with 1 threads
2018-01-12 02:10:14,387 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T065202+0000_flow.xml.gz to reduce storage usage. currentSize=524089378
2018-01-12 02:10:14,387 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T065217+0000_flow.xml.gz to reduce storage usage. currentSize=523959179
2018-01-12 02:10:14,395 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5c3e35b5 // Another save pending = true
2018-01-12 02:10:16,983 INFO [Flow Service Tasks Thread-1] o.a.n.p.FlowConfigurationArchiveManager Removing old archive file /opt/nifi/data/conf/archive/20180103T065219+0000_flow.xml.gz to reduce storage usage. currentSize=524044983
2018-01-12 02:10:16,990 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5c3e35b5 // Another save pending = false
2018-01-12 02:10:29,158 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2018-01-12 02:10:29,232 INFO [pool-10-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@735e5a69 checkpointed with 3 Records and 0 Swap Files in 73 milliseconds (Stop-the-world time = 46 milliseconds, Clear Edit Logs time = 22 millis), max Transaction ID 94171
2018-01-12 02:10:29,232 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 3 records in 73 milliseconds

[Ruslans Uralovs]

Hey Nella, 

Nothing out of the ordinary in the log.

What do you mean "it doesn't enable the template in nifi"?

And "in steps it fails to recognise the namses of nifi processors"? Do you mean that once the feed has ran, in Kylo Ops Manager you can see the job, but in the Steps tab it does not show processor names?

[alpesh ghori]

Hi Guys,

I know this is old thread but facing the similar issue so writing it here.

Basically, we are using Apache NiFi - Version 1.5.0.3.1.0.0-564 and have kylo 0.10.0. I am able to import "Publish Kafka Provenance" as template out of all the default templates in Kylo which came with the installation of Kylo. And also able to import the templates that we have created in NIFI.

But, while importing the Data_Ingest.zip or any other template; it gives the below error.

Error importing reusable template from archive data_ingest.zip. com.thinkbiganalytics.nifi.feedmgr.TemplateCreationException:Unable to create the template for the Id of [37937807-7462-37a6-b89b-e631ccac703f].

Unable to create Template instance for templateId: 37937807-7462-37a6-b89b-e631ccac703f under Process Group cdf35426-0167-1000-ffff-ffffce456ce8. com.thinkbiganalytics.nifi.v2.hdfs.CreateHDFSFolder is not known to this NiFi instance.

And when I try to upload the reusable template from kylo, I am getting the below error.

com.thinkbiganalytics.nifi.feedmgr.TemplateCreationException: Unable to create the template for the Id of [ceda83b7-8faa-3c67-93b8-7be54ec243ea].
Unable to create Template instance for templateId: ceda83b7-8faa-3c67-93b8-7be54ec243ea under Process Group cdf511a4-0167-1000-ffff-fffff1630d01. com.thinkbiganalytics.nifi.v2.spark.ExecuteSparkJob is not known to this NiFi instance.

So, i unzipped the data_ingest.zip and opened the reusable xml template and realize it requires the com.thinkbiganalytics.nifi.v2.spark.ExecuteSparkJob class from the kylo-spark-job-profiler-jar-with-dependencies.jar file.

The jar file does exist in the required location mentioned in the applicaiton.properties of the kylo-services. But the ExecuteSparkJob  class file DOES NOT exist in the kylo-spark-job-profiler-jar-with-dependencies.jar file.

So, do we need to create the separate jar for this or we are doing something wrong?

Note: As mentioned in the XML reusable template, the job also requires the kylo-nifi-spark-v1.2-nar file in the nifi home location which also exists in the mentioned location.

Please let me know if you have come across the same issue and have the resolution for this?

[Scott Reisdorf]

ExecuteSparkJob is part of the kylo-nifi-spark-#VERSION#.nar (i.e. kylo-nifi-spark-v1.2-nar-0.10.0.nar), not part of the kylo-spark-job-profiler-jar-with-dependencies.jar.

You need to make sure you have the NiFi classpath setup correctly after you configure/install Kylo.

Can you ensure the kylo required "nar" files are in the NiFi classpath?

•  In the shell list the class path for nifi: ls -l /opt/nifi/current/lib/kylo*
  
• Example output should reveal the symlinks going to the correct kylo nar files:

  [root@sandbox kylo]# ls -l /opt/nifi/current/lib/kylo*

  lrwxrwxrwx 1 nifi users 53 Dec  8 06:39 /opt/nifi/current/lib/kylo-nifi-core-nar.nar -> /opt/nifi/data/lib/kylo-nifi-core-v1.2-nar-0.10.0.nar

  lrwxrwxrwx 1 nifi users 61 Dec  8 06:39 /opt/nifi/current/lib/kylo-nifi-core-service-nar.nar -> /opt/nifi/data/lib/kylo-nifi-core-service-v1.2-nar-0.10.0.nar

  lrwxrwxrwx 1 nifi users 60 Dec  8 06:39 /opt/nifi/current/lib/kylo-nifi-elasticsearch-nar.nar -> /opt/nifi/data/lib/kylo-nifi-elasticsearch-v1-nar-0.10.0.nar

  lrwxrwxrwx 1 nifi users 55 Dec  8 06:39 /opt/nifi/current/lib/kylo-nifi-hadoop-nar.nar -> /opt/nifi/data/lib/kylo-nifi-hadoop-v1.2-nar-0.10.0.nar

  lrwxrwxrwx 1 nifi users 63 Dec  8 06:39 /opt/nifi/current/lib/kylo-nifi-hadoop-service-nar.nar -> /opt/nifi/data/lib/kylo-nifi-hadoop-service-v1.2-nar-0.10.0.nar

  lrwxrwxrwx 1 nifi users 64 Dec  8 06:39 /opt/nifi/current/lib/kylo-nifi-provenance-repo-nar.nar -> /opt/nifi/data/lib/kylo-nifi-provenance-repo-v1.2-nar-0.10.0.nar

  lrwxrwxrwx 1 nifi users 54 Dec  8 06:39 /opt/nifi/current/lib/kylo-nifi-spark-nar.nar -> /opt/nifi/data/lib/kylo-nifi-spark-v1.2-nar-0.10.0.nar

  lrwxrwxrwx 1 nifi users 62 Dec  8 06:39 /opt/nifi/current/lib/kylo-nifi-spark-service-nar.nar -> /opt/nifi/data/lib/kylo-nifi-spark-service-v1.2-nar-0.10.0.nar

  lrwxrwxrwx 1 nifi users 66 Dec  8 06:39 /opt/nifi/current/lib/kylo-nifi-standard-services-nar.nar -> /opt/nifi/data/lib/kylo-nifi-standard-services-v1.2-nar-0.10.0.nar

  lrwxrwxrwx 1 nifi users 57 Dec  8 06:39 /opt/nifi/current/lib/kylo-nifi-teradata-tdch-nar.nar -> /opt/nifi/data/lib/kylo-nifi-teradata-tdch-nar-0.10.0.nar

  
  
• If you dont have the files there or the symlinks are disconnected you can run the following shell script to update the nars and symlinks:  /opt/kylo/setup/nifi/update-nars-jars.sh