concurrent callbacks from java to python

[Aditya Singh]

Hi All,

I am new to this community and py4j so please excuse me if I my question is too silly to ask. Just wanted to know, if it is possible to have multiple callbacks from java to python?

Details:-

I am starting communication from python side and using Muti threaded model of py4j. Now let's say I start a gateway server on java side and multiple python process try to connect to it and then java tries to do a call back to the respective python process, is it possible. I did a small test and think that all callbacks will go to the single python process.

Please let me know your thoughts on this.

Regards,

Aditya

[Aditya Singh]

Can someone please help answer this question? In case this is not yet available, can we add this as a feature request, I am more than willing to work on this feature and add this to py4j.

Regards,

Aditya

--
You received this message because you are subscribed to the Google Groups "Py4J Support and Comments" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4j+uns...@py4j.org.
To view this discussion on the web visit https://groups.google.com/a/py4j.org/d/msgid/py4j/de8a1880-e8f4-410f-ad24-5a7fe78bb996n%40py4j.org.

[Jonah Graham]

Yes that should be possible using the pinned threading model - https://www.py4j.org/advanced_topics.html#using-single-threading-model-pinned-thread

(Note the link is currently unsecure, see https://github.com/bartdag/py4j/issues/434)

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com

To view this discussion on the web visit https://groups.google.com/a/py4j.org/d/msgid/py4j/CAO%3DD1RycnuvzVyL6zV29-CTjwVO-D%2BeQQ1zRHzUp12taiQ%3DGTQ%40mail.gmail.com.

[Marian Petrik]

you can't connect to a single gateway using more than one Python process. Py4j uses sockets and they are exclusive. you need to start multiple gateways on the Java side on different ports and connect to them respectively from different Python processes.

Here is an example in the documentation https://gist.github.com/bartdag/0a6043a62e12f0e53ae296c112786ac7

Marian

st 7. 7. 2021 o 20:54 Jonah Graham <jo...@kichwacoders.com> napísal(a):

To view this discussion on the web visit https://groups.google.com/a/py4j.org/d/msgid/py4j/CAPmGMviPKHTc6F4Ti8%2BiMqqDEa4L2OmAwoGYbLFP5kAboJ5Yiw%40mail.gmail.com.

[Aditya Singh]

Hi guys, 

Firstly thank you for the replies.

Now, I agree with both of you partially and that is because of a test that I did. I used py4j multithreaded model and started one java gateway server. Now there were the two test scenarios:-

• Multiple python processes without callback - All the processes worked fine and all were able to access values from the java side.
• Multiple python processes with callback - all the processes got connected and were able to access values from the java side. Now I added a python object from each python processes to the java side in a hashmap(variable type being the interface python class implements) from all the processes which went fine. However when I tried to call functions on these python objects all the calls went to the same python process, whose call back server's port was configured in the call back client attribute. The way I figured this is, I made all the python processes log their output to a text file. The output log had their pid.

Ask - Can we add a feature in py4j to add multiple callback facilities also to py4j, since we already support multiple python process connections to a single gateway server. I checked the gateway server source code which has a call back client attribute, can we change it to a list of call back clients?

Regards,

Aditya

To view this discussion on the web visit https://groups.google.com/a/py4j.org/d/msgid/py4j/CAE_6%3D66231rpXMg%2BzNREV7wjx_M2Otmvsq4RQLnpwW-dNj2t3g%40mail.gmail.com.

[Jonah Graham]

>  Ask - Can we add a feature in py4j to add multiple callback facilities also to py4j

Assuming the "we" is you and your team, I don't see why not assuming you can overcome the technical challenges, write tests, etc. Then the issue is convincing maintainers to adopt such a change. See https://github.com/bartdag/py4j/issues/426

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com

To view this discussion on the web visit https://groups.google.com/a/py4j.org/d/msgid/py4j/CAO%3DD1RwjZQB35zpQaqd8CVG2L_xtSHH%3DtgD18HDpBz%2BsZ1GLMg%40mail.gmail.com.

[Aditya Singh]

Actually its just me as of now. Will try to solve it, if solved then will connect back. Thanks for the link.

To view this discussion on the web visit https://groups.google.com/a/py4j.org/d/msgid/py4j/CAPmGMvgYAtET536k7u7%3Di96GHCwxfw%2BvRc3s5PL84VDuny9yYA%40mail.gmail.com.

[Jonah Graham]

Sounds good!

BTW - A number of years ago we needed a new threading model and we paid the Py4J maintainer to add the feature (pinned threading model). That is another option you can consider, although I don't have any idea how much time the maintainers have.

Jonah

~~~
Jonah Graham
Kichwa Coders
www.kichwacoders.com

To view this discussion on the web visit https://groups.google.com/a/py4j.org/d/msgid/py4j/CAO%3DD1RzGdsZKbNfqKc1Sdo-6Qsh_nVt2DmTeQrxiP2meGhm1OA%40mail.gmail.com.