List loaded plugins
230 views
Skip to first unread message
Pasquale Iodice
Feb 16, 2015, 5:02:13 AM2/16/15
to lum...@googlegroups.com
Hi all,
I launched this command: mvn clean package -DskipTests
-Dsource.skip=true from lumify root directory and now I have this results.
[INFO] Reactor Summary:
[INFO]
[INFO] Lumify Root: Build Tools .......................... SUCCESS [ 2.086 s]
[INFO] Lumify ............................................ SUCCESS [ 1.290 s]
[INFO] Lumify: Web ....................................... SUCCESS [ 0.094 s]
[INFO] Lumify: Web: Client API ........................... SUCCESS [ 5.722 s]
[INFO] Lumify: Core ...................................... SUCCESS [ 0.147 s]
[INFO] Lumify: Core ...................................... SUCCESS [ 24.965 s]
[INFO] Lumify: Core: Plugins ............................. SUCCESS [ 0.075 s]
[INFO] Lumify: Core: Plugin: Model: BigTable ............. SUCCESS [ 18.138 s]
[INFO] Lumify: Core: Plugin: Model: RabbitMQ ............. SUCCESS [ 1.380 s]
[INFO] Lumify: Core: Plugin: Model: Secure Graph ......... SUCCESS [ 23.088 s]
[INFO] Lumify: Web: Base ................................. SUCCESS [ 4.021 s]
[INFO] Lumify: Core: Plugin: Model: SQL .................. SUCCESS [ 5.013 s]
[INFO] Lumify: Core: Plugin: New User Workspace Copy ..... SUCCESS [ 0.574 s]
[INFO] Lumify: Core: Plugin: New User Workspace Sharer ... SUCCESS [ 0.443 s]
[INFO] Lumify: Web: Server ............................... SUCCESS [ 0.035 s]
[INFO] Lumify: Web: Server: Base ......................... SUCCESS [ 0.452 s]
[INFO] Lumify: Web: Server: Jetty ........................ SUCCESS [ 0.678 s]
[INFO] Lumify: Tools ..................................... SUCCESS [ 0.035 s]
[INFO] Lumify: Tools: Format ............................. SUCCESS [ 19.578 s]
[INFO] Lumify: Core: Test ................................ SUCCESS [ 4.440 s]
[INFO] Lumify: YARN Base ................................. SUCCESS [ 0.282 s]
[INFO] Lumify: Datasets .................................. SUCCESS [ 0.037 s]
[INFO] Lumify: Datasets: Friendster ...................... SUCCESS [ 19.867 s]
[INFO] Lumify: Datasets: Wikipedia ....................... SUCCESS [ 0.287 s]
[INFO] Lumify: Datasets: Wikipedia: Core ................. SUCCESS [ 0.666 s]
[INFO] Lumify: Datasets: Wikipedia: MapReduce ............ SUCCESS [ 21.258 s]
[INFO] Lumify: Datasets: Flight Aware .................... SUCCESS [ 10.993 s]
[INFO] Lumify: Datasets: Twitter ......................... SUCCESS [ 0.052 s]
[INFO] Lumify: Datasets: Twitter: Common ................. SUCCESS [ 0.421 s]
[INFO] Lumify: Datasets: Twitter: Graph Property Worker .. SUCCESS [ 0.460 s]
[INFO] Lumify: Datasets: The World Factbook .............. SUCCESS [ 18.322 s]
[INFO] Lumify: Datasets: GDELT ........................... SUCCESS [ 0.387 s]
[INFO] Lumify: Datasets: GDELT: Parser ................... SUCCESS [ 1.398 s]
[INFO] Lumify: Datasets: GDELT: MapReduce ................ SUCCESS [ 17.973 s]
[INFO] Lumify: Datasets: TheMovieDb ...................... SUCCESS [ 1.130 s]
[INFO] Lumify: Datasets: TheMovieDb Download ............. SUCCESS [ 0.231 s]
[INFO] Lumify: Datasets: TheMovieDb MR ................... SUCCESS [ 18.580 s]
[INFO] Lumify: Datasets: DBpedia ......................... SUCCESS [ 20.064 s]
[INFO] Lumify: Datasets: Palantir ........................ SUCCESS [ 1.403 s]
[INFO] Lumify: Datasets: Palantir: Ontology to OWL ....... SUCCESS [ 6.812 s]
[INFO] Lumify: Datasets: Palantir: Data Import ........... SUCCESS [ 9.278 s]
[INFO] Lumify: Examples .................................. SUCCESS [ 0.041 s]
[INFO] Lumify: Examples: Food Truck Analytic ............. SUCCESS [ 0.439 s]
[INFO] Lumify: GraphPropertyWorker ....................... SUCCESS [ 0.030 s]
[INFO] Lumify: GraphPropertyWorker: Plugins .............. SUCCESS [ 0.015 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: SubRip Parser SUCCESS [ 0.374 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: CCExtractor .. SUCCESS [ 0.456 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: E-Mail Address Extractor SUCCESS [ 0.361 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Known Entity Extractor SUCCESS [ 0.648 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: OpenNLP Dictionary Extractor SUCCESS [ 2.121 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: OpenNLP Maximum Entropy SUCCESS [ 1.030 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Phone Number Extractor SUCCESS [ 0.862 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Sphinx ....... SUCCESS [ 0.414 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: SubRip Transcript SUCCESS [ 0.393 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Tesseract .... SUCCESS [ 2.950 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Tika Mime Type Mapper SUCCESS [ 10.247 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Tika Text Extractor SUCCESS [ 9.598 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Youtube Transcript SUCCESS [ 1.950 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Zip Code Extractor SUCCESS [ 0.432 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Zip Code Resolver SUCCESS [ 0.747 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Search Reindex SUCCESS [ 1.324 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Translate Language Detection SUCCESS [ 1.829 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Translator Bing SUCCESS [ 0.444 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: OpenCV Object Detector SUCCESS [ 6.762 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Clavin ....... SUCCESS [ 2.126 s]
[INFO] Lumify: GraphPropertyWorker: Base ................. SUCCESS [ 0.471 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Drew Noakes Image Metadata Extractor SUCCESS [ 0.989 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Java Code .... SUCCESS [ 0.746 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Mime Type Ontology Mapper SUCCESS [ 0.238 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: CSV .......... SUCCESS [ 0.554 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: RDF .......... SUCCESS [ 1.590 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Video MP4 Encoder SUCCESS [ 0.483 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Video WebM Encoder SUCCESS [ 0.455 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Video Audio Extract SUCCESS [ 0.402 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Audio MP4 Encoder SUCCESS [ 0.381 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Audio OGG Encoder SUCCESS [ 0.422 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Video Poster Frame SUCCESS [ 0.398 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Video Frame Extract SUCCESS [ 0.462 s]
[INFO] Lumify: GraphPropertyWorker: Plugin: Video Metadata SUCCESS [ 0.443 s]
[INFO] Lumify: GraphPropertyWorker: YARN ................. SUCCESS [ 10.110 s]
[INFO] Lumify: Tools: Version Inspector .................. SUCCESS [ 6.470 s]
[INFO] Lumify: Tools: Import ............................. SUCCESS [ 0.779 s]
[INFO] Lumify: Tools: RDF Import ......................... SUCCESS [ 0.346 s]
[INFO] Lumify: Tools: User Admin ......................... SUCCESS [ 0.374 s]
[INFO] Lumify: Tools: Requeue ............................ SUCCESS [ 20.339 s]
[INFO] Lumify: Tools: CLI ................................ SUCCESS [ 10.605 s]
[INFO] Lumify: Tools: Re-index MapReduce ................. SUCCESS [ 19.953 s]
[INFO] Lumify: Tools: Backup/Restore ..................... SUCCESS [ 5.502 s]
[INFO] Lumify: Tools: Assign Image MapReduce ............. SUCCESS [ 19.784 s]
[INFO] Lumify: Tools: Long Running Process YARN .......... SUCCESS [ 10.104 s]
[INFO] Lumify: Web: Server: Tomcat ....................... SUCCESS [ 1.400 s]
[INFO] Lumify: Web: War .................................. SUCCESS [01:40 min]
[INFO] Lumify: Web: Plugins .............................. SUCCESS [ 0.021 s]
[INFO] Lumify: Web: Plugin: Social Authentication ........ SUCCESS [ 0.625 s]
[INFO] Lumify: Web: Plugin: Username-only Authentication . SUCCESS [ 0.349 s]
[INFO] Lumify: Web: Plugin: Username/Password Authentication SUCCESS [ 0.510 s]
[INFO] Lumify: Web: Plugin: X509 Authentication .......... SUCCESS [ 0.334 s]
[INFO] Lumify: Web: Plugin: Dev Tools .................... SUCCESS [ 0.511 s]
[INFO] Lumify: Web: Plugin: Geo-coder: Bing .............. SUCCESS [ 0.364 s]
[INFO] Lumify: Web: Plugin: Google Analytics ............. SUCCESS [ 0.317 s]
[INFO] Lumify: Web: Plugin: Import/Export Workspaces ..... SUCCESS [ 0.477 s]
[INFO] Lumify: Web: Plugin: OpenNLP Dictionary Extractor . SUCCESS [ 0.843 s]
[INFO] Lumify: Web: Plugin: Terms of Use ................. SUCCESS [ 0.425 s]
[INFO] Lumify: Web: Plugin: Analyst's Notebook Export .... SUCCESS [ 19.347 s]
[INFO] Lumify: Web: Plugin: Change Password .............. SUCCESS [ 1.347 s]
[INFO] Lumify: Web: Plugin: Change Email ................. SUCCESS [ 0.336 s]
[INFO] Lumify: Web: Client API: Code Generator ........... SUCCESS [ 1.228 s]
[INFO] Lumify: Dev ....................................... SUCCESS [ 0.034 s]
[INFO] Lumify: Dev: Jetty Server ......................... SUCCESS [ 0.233 s]
[INFO] Lumify: Dev: GraphPropertyWorker: Local ........... SUCCESS [ 0.434 s]
[INFO] Lumify: Dev: Tomcat Server ........................ SUCCESS [ 0.130 s]
[INFO] Lumify: Dev: Import ............................... SUCCESS [ 0.225 s]
[INFO] Lumify: Integration Test .......................... SUCCESS [ 0.683 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 09:14 min
[INFO] Finished at: 2015-02-14T04:45:13-08:00
[INFO] Final Memory: 186M/743M
When I start Lumify, should I see all plugins in the " List loaded plugins" ? I attached a picture of what I'm seeing.
Thanks in advance.
Pasquale Iodice
Feb 17, 2015, 9:47:54 AM2/17/15
to lum...@googlegroups.com
I followed YARN Plugin Build Instructions guide https://github.com/lumifyio/lumify/blob/master/docs/build.md and now I can see all built plugin in my List loaded plugins under GRAPH PROPERTY WORKER. My problem is when I import a text file or an image in lumify by drag-drop, I only see (?) icon. I looked hadoop resource manager and it seems there are no application running.
Thanks
Thanks
abhinav mohan
Feb 18, 2015, 1:17:18 AM2/18/15
to lum...@googlegroups.com
Hey,
I am not sure about the Hadoop RM part but do you have the mime-type-ontology-mapper plugin loaded ? Adding that plugin solved the problem of (?) icon for me.
Pasquale Iodice
Feb 18, 2015, 3:18:04 AM2/18/15
to lum...@googlegroups.com
Hey, thanks for your replay!I built all the plugins in graph-property worker/plugins included tika-mime-type. I can see them in the plugin list, but they seem not working. Did I miss some step?
Joe Ferner
Feb 18, 2015, 12:48:24 PM2/18/15
to lum...@googlegroups.com
Graph property workers can run in one of two ways. Via YARN (this is for a production system) or via the web server. By default the web server option is turned on and should run the graph property workers. But graph property workers only run if something is new or something has changed. If you did not re-add your data after adding the graph property workers it will not fix the (?) problem. You can either re-ingest your data or re-queue the items. Re-queue'ing the items is the easiest. If you open the admin panel on the web interface and click "Vertex/Requeue" then click "Requeue" with the property name "http://lumify.io#raw". This will force the graph property workers to re-look at the data.
Pasquale Iodice
Feb 19, 2015, 5:33:46 AM2/19/15
to lum...@googlegroups.com
IThanks Joe. I followed your instruction but I have the same problem. I can see all built plugins in loaded plugins under graph-property-worker section, but if I try to drag and drop some doc as .odt .png ... I only see (?) icons.
Pasquale Iodice
Feb 20, 2015, 10:29:06 AM2/20/15
to lum...@googlegroups.com
This is my log
2015-02-20 15:11:46.716/UTC INFO [core.util.LumifyLoggerFactory] Using ConfigurationLoader: io.lumify.core.config.FileConfigurationLoader
2015-02-20 15:11:46.726/UTC INFO [core.util.LumifyLoggerFactory] Using log4j.xml: /opt/lumify/config/log4j.xml
2015-02-20 15:11:46.726/UTC DEBUG [core.config.FileConfigurationLoader] Attempting to load configuration from directory: /opt/lumify/config
2015-02-20 15:11:46.727/UTC INFO [core.config.FileConfigurationLoader] Loading config file: /opt/lumify/config/lumify.properties
2015-02-20 15:11:46.745/UTC INFO [core.bootstrap.InjectHelper] Loading libs...
2015-02-20 15:11:46.746/UTC DEBUG [core.util.ServiceLoaderUtil] loading services for class META-INF/services/io.lumify.core.bootstrap.lib.LibLoader
2015-02-20 15:11:46.747/UTC DEBUG [core.util.ServiceLoaderUtil] loadFile(jar:file:/tmp/jetty-0.0.0.0-8080-root.war-_-any-6764799641748919875.dir/webapp/WEB-INF/lib/lumify-core-0.4.1-SNAPSHOT.jar!/META-INF/services/io.lumify.core.bootstrap.lib.LibLoader)
2015-02-20 15:11:46.747/UTC INFO [core.util.ServiceLoaderUtil] Loading io.lumify.core.bootstrap.lib.LibDirectoryLoader from jar:file:/tmp/jetty-0.0.0.0-8080-root.war-_-any-6764799641748919875.dir/webapp/WEB-INF/lib/lumify-core-0.4.1-SNAPSHOT.jar!/META-INF/services/io.lumify.core.bootstrap.lib.LibLoader
2015-02-20 15:11:46.749/UTC INFO [core.util.ServiceLoaderUtil] Loading io.lumify.core.bootstrap.lib.HdfsLibCacheLoader from jar:file:/tmp/jetty-0.0.0.0-8080-root.war-_-any-6764799641748919875.dir/webapp/WEB-INF/lib/lumify-core-0.4.1-SNAPSHOT.jar!/META-INF/services/io.lumify.core.bootstrap.lib.LibLoader
"lumify-root-3816.log" 2572L, 254793C
Leo nardo Furio
Feb 20, 2015, 11:03:17 AM2/20/15
to lum...@googlegroups.com
I found it!!!
1) the log folder (that I told you on chat) is wrong, you can see the log from the jetty log folder /opt/jetty-distribution-9.2.7.v20150116/logs/2015_02_20.stderrout.log
2) Some plugins need external file (like clavin or Know Entity extractor) and if you don't load these file you will get an error. So I suggest you to put just this and try again!
lumify-gpw-mime-type-ontology-mapper-0.4.1-SNAPSHOT-jar-with-dependencies.jar
lumify-gpw-opennlp-dictionary-extractor-0.4.1-SNAPSHOT-jar-with-dependencies.jar
lumify-gpw-opennlp-me-extractor-0.4.1-SNAPSHOT-jar-with-dependencies.jar
lumify-gpw-tika-mime-type-0.4.1-SNAPSHOT-jar-with-dependencies.jar
lumify-gpw-tika-text-extractor-0.4.1-SNAPSHOT-jar-with-dependencies.jar
lumify-web-auth-username-only-0.4.1-SNAPSHOT.jar
lumify-web-dev-tools-0.4.1-SNAPSHOT.jar
Regards!
leo
Pasquale Iodice
Feb 21, 2015, 11:31:34 AM2/21/15
to lum...@googlegroups.com
Hi Leo, thanks for your replay.
This is my /opt/jetty-distribution-9.2.7.v20150116/logs/2015_02_20.stderrout.log file if I put all graph-property-worker/plugins in /lumify/libcache
2015-02-21 15:26:57.295:INFO:oejs.Server:main: jetty-9.2.6.v20141205
2015-02-21 15:26:57.385:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:/opt/jetty-distribution-9.2.6.v20141205/webapps/] at interval 1
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
"2015_02_21.stderrout.log" 2L, 219C
And this one if I just put your listed .jar
2015-02-21 16:18:27.711:INFO:oejs.Server:main: jetty-9.2.6.v20141205
2015-02-21 16:18:27.781:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:/opt/jetty-distribution-9.2.6.v20141205/webapps/] at interval 1
2015-02-21 16:18:51.228/UTC WARN [hadoop.util.NativeCodeLoader] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-02-21 16:18:58.490:DBUG:oejws.WebSocketUpgradeFilter:main: Adding [Jetty_Dynamic_WebSocketUpgradeFilter] WebSocketUpgradeFilter[factory=org.eclipse.jetty.websocket.server.WebSocketServerFactory@2885b799,pathmap=PathMappings[size=0]] mapped to /* to Servlet...@o.e.j.w.WebAppContext@369332f4{/,file:/tmp/jetty-0.0.0.0-8080-root.war-_-any-7932266947021888936.dir/webapp/,STARTING}{/root.war}
2015-02-21 16:18:58.552:DBUG:oejwj.DecoderFactory:main: init(org.eclipse.jetty.websocket.jsr356.client.EmptyClientEndpointConfig@3af242bf)
2015-02-21 16:18:58.557:DBUG:oejwj.EncoderFactory:main: init(org.eclipse.jetty.websocket.jsr356.client.EmptyClientEndpointConfig@3af242bf)
2015-02-21 16:18:58.730:DBUG:oejwjsd.WebSocketServerContainerInitializer:main: Found 1 classes
2015-02-21 16:18:58.731:DBUG:oejwjsd.WebSocketServerContainerInitializer:main: Discovered 1 extends Endpoint classes
2015-02-21 16:18:58.731:DBUG:oejwjsd.WebSocketServerContainerInitializer:main: Discovered 0 @ServerEndpoint classes
2015-02-21 16:18:58.732:DBUG:oejwjsd.WebSocketServerContainerInitializer:main: Discovered 0 ServerApplicationConfig classes
2015-02-21 16:18:58.732:DBUG:oejwjsd.WebSocketServerContainerInitializer:main: Deploying 0 ServerEndpointConfig(s)
2015-02-21 16:18:58.732:DBUG:oejwjsd.WebSocketServerContainerInitializer:main: Deploying 0 @ServerEndpoint(s)
2015-02-21 16:18:59.711:INFO:root:main: Initializing AtmosphereFramework
2015-02-21 16:18:59.744:DBUG:oejwc.WebSocketClient:main: Starting org.eclipse.jetty.websocket.client.WebSocketClient@6d658d83
2015-02-21 16:19:01.402:DBUG:oejwc.WebSocketClient:main: Started org.eclipse.jetty.websocket.client.WebSocketClient@6d658d83
Servlet context initialized...
2015-02-21 16:19:08.469:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@369332f4{/,file:/tmp/jetty-0.0.0.0-8080-root.war-_-any-7932266947021888936.dir/webapp/,AVAILABLE}{/root.war}
2015-02-21 16:19:08.855:INFO:oejs.ServerConnector:main: Started ServerConnector@6d249320{HTTP/1.1}{0.0.0.0:8080}
2015-02-21 16:19:08.977:INFO:oejs.ServerConnector:main: Started ServerConnector@7210a2fd{SSL-http/1.1}{0.0.0.0:8443}
2015-02-21 16:19:08.977:INFO:oejs.Server:main: Started @45460ms
~
~
~
~
~
~
~
~
~
~
"2015_02_21.stderrout.log" 20L, 2584C
Result is the same!!! I tried to drag&drop a .pdf .odt .png files and I am seeing (?) icons yet!
Leo nardo Furio
Feb 23, 2015, 7:47:27 AM2/23/15
to lum...@googlegroups.com
Could you paste here the value of hdfsLibcache.sourceDirectory= in lumify.properties file ?
Have you put there your jar plugins files?
Leo
Leo nardo Furio
Feb 23, 2015, 10:38:09 AM2/23/15
to lum...@googlegroups.com
After load of ontology dev restart the jetty....
Pasquale Iodice
Feb 24, 2015, 4:06:09 AM2/24/15
to lum...@googlegroups.com
Hi Leo,
hdfsLibcache.sourceDirectory=hdfs://lumify-dev/lumify/libcache and this is where i put my jar. After building I alwayse run
hadoop fs -put /path/jar_with_dependencies /lumify/libcache
However I added a variable on hadoop-env.sh and yarn-env.sh.
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/nativeexport HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"
and now it works.
Now I'm trying to build all the jar from "lumify-master" directory launching the command mvn clean package -DskipTests
-Dsource.skip=true and I have a build failure.
I try to change the pom.xml as you reported in the other thread.
Thanks!
vincenzo algieri
Jul 11, 2016, 8:14:34 AM7/11/16
to Lumify
Hi Leo,
can you explain me step by step what I shoul do to make it work??
1) So firstly, in what folder I should put the plugin of the property-worker??
2) Have I also to add "something" in the config/lumify.properties file????
3) where is the log file from which I can see what appen when I run server??
Thanks.
can you explain me step by step what I shoul do to make it work??
1) So firstly, in what folder I should put the plugin of the property-worker??
2) Have I also to add "something" in the config/lumify.properties file????
3) where is the log file from which I can see what appen when I run server??
Thanks.
Reply all
Reply to author
Forward
0 new messages