Configuring Warp10 for Py4J python plugin

CD_KND

unread,

Mar 5, 2020, 10:22:48 AM3/5/20

to Warp 10 users

Hi to all,

I'm pretty new to Warp10, trying to wrap my head around configuring it so that I can run python code to upload and fetch data.

I'm running the Warp10 Docker container on Windows 10. I managed to upload/fetch/delete data through the WarpStudio.

I'm now trying to follow Mathias's blog https://blog.senx.io/the-py4j-plugin-for-warp-10/

I downloaded the 'warp10-plugin-py4j-1.0.0-uberjar.jar' file from the download section of https://warpfleet.senx.io/browse/io.warp10/warp10-plugin-py4j

The next instructions are as follow:

Next, declare the Py4J plugin in the Warp 10 configuration file.

warp10.plugin.py4j = io.warp10.plugins.py4j.Py4JWarp10Plugin

Then, if you want to be able to fetch data stored in Warp 10 from Python, set the following property to true:

egress.clients.expose = true

After that, restart your Warp 10™ instance for the plugin to be taken into account.

In the log file <Warp10>/logs/warp10.log, check that the following line appears:

LOADED plugin 'io.warp10.plugins.py4j.Py4JWarp10Plugin'

First of all, I'm not sure what the 'Warp 10 configuration file' is. Should I add the specified plugin declaration line in '80--plugins.conf' ? Or should I create a specific '80-py4j-plugin.conf' file?

I tried both but never managed to see the

LOADED plugin 'io.warp10.plugins.py4j.Py4JWarp10Plugin'

in the Warp10 log after restarting my container.

Thanks for your help in advance!

CD

Jean-Charles Vialatte

unread,

Mar 11, 2020, 12:06:50 PM3/11/20

to Warp 10 users

Hi CD,

The Warp10 configuration was held in a single file in old releases. Now it is splitted in multiple '.conf' files in the 'WARP10_HOME/etc/conf.d' directory.

It doesn't matter in which 'xxxx.conf' file the plugin declaration is done.

It does not seem like you did anything wrong with Warp10 setup.

Can you please check that you put the 'warp10-plugin-py4j-1.0.0-uberjar.jar' in the right docker volume under 'WARP10_HOME/lib' ?

Then can you please paste the last 30 lines of the warp10 log file ?

CD_KND

unread,

Mar 11, 2020, 1:42:02 PM3/11/20

to Warp 10 users

Hi Jean-Charles,

Thank you for your answer.

I confirm my 'warp10-plugin-py4j-1.0.0-uberjar.jar' is in the warp10\lib dir. The only other jar present is 'warp10-plugin-warpstudio-1.0.20.jar' and my WarpStudio works fine.

I created a '80--py4j-plugin.conf', in 'warp10\etc\conf.d' with the following content:

#

# register the Py4J plugin

warp10.plugin.py4j = io.warp10.plugins.py4j.Py4JWarp10Plugin

#

# set to true to allow to find/fetch data through py4j

# (this configuration is to be modified in 10-egress.conf.template)

egress.clients.expose = false

#

# set a secret to use an authentication token (strongly advised)

# py4j.authtoken = @encryption_secret@

#

# release stack limits (to be used with caution)

py4j.stack.nolimits = false

For some reason my Docker seems to take over 3min to load, or at least I cannot access the WarpStudio until that period of time. Not sure if that's an issue or not. Here's what is in the log. I'm not seeing anything about Py4j. Thank you for your help!

___ __ ____________

__ | / /_____ _______________ __< /_ __ \

__ | /| / /_ __ `/_ ___/__ __ \ __ /_ / / /

__ |/ |/ / / /_/ /_ / __ /_/ / _ / / /_/ /

____/|__/ \__,_/ /_/ _ .___/ /_/ \____/

/_/

Revision 2.4.0

2020-03-11 17:39:39.440:INFO:iwsjoejs.Server:jetty-8.y.z-SNAPSHOT

########[ Initialized with 1000 time units per millisecond ]########

2020-03-11 17:39:39.620:INFO:iwsjoejs.AbstractConnector:Started SelectChann...@127.0.0.1:37417

2020-03-11T17:39:39,828 main WARN script.WarpFleetMacroRepository - No validator macro, default macro will reject all URLs.

REPORT secret not set, using '7320959a-1ab0-4fb1-bef0-7a76b5b8327c'.

Loaded 163 GTS in 355.222236 ms

#### standalone.endpoint /0.0.0.0:8080

Property warpstudio.host found, value: ${main.host}

2020-03-11 17:39:43.472:INFO::[WarpStudioPlugin 0.0.0.0:8081]: Logging initialized @4713ms

host: 0.0.0.0

port: 8081

Web Root URI: jar:file:/data/warp10/lib/warp10-plugin-warpstudio-1.0.20.jar!/public_html/

#### warpstudio.endpoint /0.0.0.0:8081

2020-03-11 17:39:44.007:INFO:iwwoejs.Server:[WarpStudioPlugin 0.0.0.0:8081]: io.warp10.warpstudio.jetty-9.3.z-SNAPSHOT

2020-03-11 17:39:44.127:INFO:iwwoejsh.ContextHandler:[WarpStudioPlugin 0.0.0.0:8081]: Started ServletContextHandler@1316dd8d{/,jar:file:/data/warp10/lib/warp10-plugin-warpstudio-1.0.20.jar!/public_html/,AVAILABLE}

2020-03-11 17:39:44.146:INFO:iwwoejs.AbstractConnector:[WarpStudioPlugin 0.0.0.0:8081]: Started ServerConnector@2941ef32{HTTP/1.1,[http/1.1, h2c, h2c-17, h2c-16, h2c-15, h2c-14]}{0.0.0.0:8081}

2020-03-11 17:39:44.150:INFO:iwwoejs.Server:[WarpStudioPlugin 0.0.0.0:8081]: Started @5390ms

io.warp10.warpstudio.org.eclipse.jetty.server.Server@40519bbc - STARTED

+= qtp1726425249{STARTED,8<=8<=200,i=6,q=0} - STARTED

| +- 66 qtp1726425249-66 TIMED_WAITING @ sun.misc.Unsafe.park(Native Method) IDLE

| +- 62 qtp1726425249-62 TIMED_WAITING @ sun.misc.Unsafe.park(Native Method) IDLE

| +- 59 qtp1726425249-59 RUNNABLE @ sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

| +- 65 qtp1726425249-65 TIMED_WAITING @ sun.misc.Unsafe.park(Native Method) IDLE

| +- 60 qtp1726425249-60-acceptor-0@11f5e921-ServerConnector@2941ef32{HTTP/1.1,[http/1.1, h2c, h2c-17, h2c-16, h2c-15, h2c-14]}{0.0.0.0:8081} RUNNABLE @ sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3

| +- 61 qtp1726425249-61 TIMED_WAITING @ sun.misc.Unsafe.park(Native Method) IDLE

| +- 64 qtp1726425249-64 TIMED_WAITING @ sun.misc.Unsafe.park(Native Method) IDLE

| +- 63 qtp1726425249-63 TIMED_WAITING @ sun.misc.Unsafe.park(Native Method) IDLE

+= ServerConnector@2941ef32{HTTP/1.1,[http/1.1, h2c, h2c-17, h2c-16, h2c-15, h2c-14]}{0.0.0.0:8081} - STARTED

| +~ io.warp10.warpstudio.org.eclipse.jetty.server.Server@40519bbc - STARTED

| +~ qtp1726425249{STARTED,8<=8<=200,i=6,q=0} - STARTED

| += io.warp10.warpstudio.org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@76e52bf9 - STARTED

| +- io.warp10.warpstudio.org.eclipse.jetty.io.ArrayByteBufferPool@500dca62

| += HttpConnectionFactory@53143904[HTTP/1.1] - STARTED

| | +- HttpConfiguration@7693972e{32768/8192,8192/8192,https://:0,[]}

| += HTTP2CServerConnectionFactory@6de69d49[h2c, h2c-17, h2c-16, h2c-15, h2c-14] - STARTED

| | +- HttpConfiguration@7693972e{32768/8192,8192/8192,https://:0,[]}

| += io.warp10.warpstudio.org.eclipse.jetty.server.ServerConnector$ServerConnectorManager@44c4c568 - STARTED

| | += io.warp10.warpstudio.org.eclipse.jetty.io.ManagedSelector@8e33f30 id=0 keys=0 selected=0 id=0

| | +- sun.nio.ch.EPollSelectorImpl@52866e6 keys=0

| +- sun.nio.ch.ServerSocketChannelImpl[/0.0.0.0:8081]

| +- qtp1726425249-60-acceptor-0@11f5e921-ServerConnector@2941ef32{HTTP/1.1,[http/1.1, h2c, h2c-17, h2c-16, h2c-15, h2c-14]}{0.0.0.0:8081}

+= ServletContextHandler@1316dd8d{/,jar:file:/data/warp10/lib/warp10-plugin-warpstudio-1.0.20.jar!/public_html/,AVAILABLE} - STARTED

| += io.warp10.warpstudio.org.eclipse.jetty.servlet.ServletHandler@439fac8 - STARTED

| | += default@5c13d641==io.warp10.warpstudio.org.eclipse.jetty.servlet.DefaultServlet,-1,false - STARTED

| | | +- dirAllowed=true

| | +- [/]=>default

| |

| +> No ClassLoader

| +> Handler attributes ServletContextHandler@1316dd8d{/,jar:file:/data/warp10/lib/warp10-plugin-warpstudio-1.0.20.jar!/public_html/,AVAILABLE}

| | +- io.warp10.warpstudio.org.eclipse.jetty.server.Executor=qtp1726425249{STARTED,8<=8<=200,i=6,q=0}

| +> Context attributes ServletContextHandler@1316dd8d{/,jar:file:/data/warp10/lib/warp10-plugin-warpstudio-1.0.20.jar!/public_html/,AVAILABLE}

| | +- io.warp10.warpstudio.org.eclipse.jetty.util.DecoratedObjectFactory=io.warp10.warpstudio.org.eclipse.jetty.util.DecoratedObjectFactory[decorators=1]

| +> Initparams ServletContextHandler@1316dd8d{/,jar:file:/data/warp10/lib/warp10-plugin-warpstudio-1.0.20.jar!/public_html/,AVAILABLE}

|

+> io.warp10.WarpClassLoader@29d89d5d

+- sun.misc.Launcher$AppClassLoader@33909752

Jean-Charles Vialatte

unread,

Mar 12, 2020, 8:12:20 AM3/12/20

to Warp 10 users

It looks like the logging information about plugins that have been loaded has been erroneously removed in last revisions.

To check that the Py4J plugin has been correctly loaded, can you run a container mapping on the same volume and on the host network (for py4j port bindings) with:

docker run -d --network host --volume=<insert-volume-here>:/data warp10io/warp10:latest

Then try to run the examples in the blog post ?

2020-03-11 17:39:39.620:INFO:iwsjoejs.AbstractConnector:Started SelectChannelConnector@127.0.0.1:37417

Jean-Charles Vialatte

unread,

Mar 17, 2020, 9:23:00 AM3/17/20

to Warp 10 users

The issue with some information not being logged by default has been corrected on master. See https://github.com/senx/warp10-platform/pull/695

2020-03-11 17:39:39.620:INFO:iwsjoejs.AbstractConnector:Started SelectChann...@127.0.0.1:37417

--
You received this message because you are subscribed to the Google Groups "Warp 10 users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp10-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp10-users/a169dee7-25d6-406d-bb29-c76b611ea425%40googlegroups.com.

Reply all

Reply to author

Forward