Apache Nifi integration...
169 views
Skip to first unread message
Pat Trainor
May 23, 2016, 3:25:02 AM5/23/16
to Aurelius
I have reasons to use Nifi for a larger architecture, but a big part is titan graph.
Since there are no Titan connectors, what are the ways to get data IN to titan via the processors/connectors available in Apache Nifi?
Nifi can run SCQL commands on my Cassandra backend, but that doesn't help. I need to insert into Titan the way I do with Java.
Now it is true that I can call a java program, but it has to load on a new thread each and every time it is called, which will be a mess. Ideally, such a java program could be running 24/7 in a loop waiting for Nifi input of a flowfile to fill in the ?s and create the nodes... I'm still trying to figure out how to do that...
Or is it best to make a daemon/web server and call that with Nifi?
In one flow example, I load Stanford NLP, which takes over 30 seconds to load... But a string parsing/tokenizing takes a split second. I wrote that in C#, loading the jar files from Stanford... Still can't figure out how to let Nifi at it, though-especially since my programs will be passing data back to Nifi after they are done...So the flowfile gets transformed & routed at each processor...
Has anyone tried to do any of this? Is there an obvious solution in front of me?
No doubt the NSA was doing this... :)
TIA!
Since there are no Titan connectors, what are the ways to get data IN to titan via the processors/connectors available in Apache Nifi?
Nifi can run SCQL commands on my Cassandra backend, but that doesn't help. I need to insert into Titan the way I do with Java.
Now it is true that I can call a java program, but it has to load on a new thread each and every time it is called, which will be a mess. Ideally, such a java program could be running 24/7 in a loop waiting for Nifi input of a flowfile to fill in the ?s and create the nodes... I'm still trying to figure out how to do that...
Or is it best to make a daemon/web server and call that with Nifi?
In one flow example, I load Stanford NLP, which takes over 30 seconds to load... But a string parsing/tokenizing takes a split second. I wrote that in C#, loading the jar files from Stanford... Still can't figure out how to let Nifi at it, though-especially since my programs will be passing data back to Nifi after they are done...So the flowfile gets transformed & routed at each processor...
Has anyone tried to do any of this? Is there an obvious solution in front of me?
No doubt the NSA was doing this... :)
TIA!
Stephen Mallette
May 23, 2016, 1:11:13 PM5/23/16
to Aurelius
I don't have any Nifi experience so I don't have much to offer. Interesting you should ask this as Matt Burress mentioned Nifi on the TinkerPop mailing list within hours of your email:
Not sure if you are working together on something, but Matt has been using TinkerPop for a while now and might have some insight if you reach out to him. I'm sure if other Titan users who are more Nifi knowledgeable can offer something here they will.
--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/b9960286-2676-43aa-87da-1918eceb32d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages