PyCall and callbacks (paho-mqtt)

[Kaj Wiik]

I am trying to call the paho-mqtt library (https://pypi.python.org/pypi/paho-mqtt), here's the Python example code:

import paho.mqtt.client as mqtt

# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
    print("Connected with result code "+str(rc))

    # Subscribing in on_connect() means that if we lose the connection and
    # reconnect then subscriptions will be renewed.
    client.subscribe("$SYS/#")

# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
    print(msg.topic+" "+str(msg.payload))

client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message

client.connect("iot.eclipse.org", 1883, 60)

# Blocking call that processes network traffic, dispatches callbacks and
# handles reconnecting.
# Other loop*() functions are available that give a threaded interface and a
# manual interface.
client.loop_forever()

And here's my naïve translation to Julia:

using PyCall

@pyimport paho.mqtt.client as mqtt

function on_connect(client, userdata, flags, rc)
    println("Connected with result code "*string(rc))
    client[:subscribe]("$SYS/#")
end

function on_message(client, userdata, msg)
    println(msg[:topic]*" "*string(msg[:payload]))
end

client = mqtt.Client()
client[:on_connect] = on_connect
client[:on_message] = on_message

client[:connect]("iot.eclipse.org", 1883, 60)
client[:loop_forever]()

I get the following errors:

ERROR: LoadError: PyError (:PyObject_Call) <type 'exceptions.AttributeError'>
AttributeError("'PyCall.jl_Function' object has no attribute 'func_code'",)
  File "/home/kjwiik/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1261, in loop_forever
    rc = self.loop(timeout, max_packets)
  File "/home/kjwiik/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 811, in loop
    rc = self.loop_read(max_packets)
  File "/home/kjwiik/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1073, in loop_read
    rc = self._packet_read()
  File "/home/kjwiik/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1475, in _packet_read
    rc = self._packet_handle()
  File "/home/kjwiik/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1949, in _packet_handle
    return self._handle_connack()
  File "/home/kjwiik/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2001, in _handle_connack
    argcount = self.on_connect.func_code.co_argcount

 [inlined code] from /home/kjwiik/.julia/v0.4/PyCall/src/exception.jl:81
 in pycall at /home/kjwiik/.julia/v0.4/PyCall/src/PyCall.jl:356
 in call at /home/kjwiik/.julia/v0.4/PyCall/src/PyCall.jl:372
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
while loading /home/kjwiik/proj/telescopes/software/mqttclient.jl, in expression starting on line 19

It seems that I haven't been able to pass the callbacks correctly, how it should be done? The Python code works OK, of course.

Thanks,
Kaj

[Steven G. Johnson]

ERROR: LoadError: PyError (:PyObject_Call) <type 'exceptions.AttributeError'>
AttributeError("'PyCall.jl_Function' object has no attribute 'func_code'",)

The foo.func_code attribute of a Python function foo returns a code object representing the compiled Python bytecode for foo.   A Julia function is not going to have this attribute, because it has no Python bytecode.

It's really surprising to me that the paho.mqtt.client module only works for callbacks with func_code attributes, which means that it only works for pure Python functions.   Maybe you can inquire with the Paho authors to find out why (and whether they can lift this limitation in a future version).

[Kaj Wiik]

On Thursday, April 28, 2016 at 1:11:03 AM UTC+3, Steven G. Johnson wrote:

ERROR: LoadError: PyError (:PyObject_Call) <type 'exceptions.AttributeError'>
AttributeError("'PyCall.jl_Function' object has no attribute 'func_code'",)

The foo.func_code attribute of a Python function foo returns a code object representing the compiled Python bytecode for foo.   A Julia function is not going to have this attribute, because it has no Python bytecode.

OK, I see, thanks. I found this from the module source (client.py):
 

            if sys.version_info[0] < 3:
                argcount = self.on_connect.func_code.co_argcount
            else:
                argcount = self.on_connect.__code__.co_argcount

I thought this is the limitation and installed Python3, unfortunately then I got an error that __code__ is missing:

ERROR: LoadError: PyError (:PyObject_Call) <class 'AttributeError'>
AttributeError("'PyCall.jl_Function' object has no attribute '__code__'",)

Well, func_code seems to be __code__ in v3, so still the same problem... Maybe I have to contact the authors, anyone have another idea how to use MQTT from Julia (of course mosquitto_pub and _sub should work)?

Thanks,
Kaj

[Steven G. Johnson]

On Wednesday, April 27, 2016 at 7:06:46 PM UTC-4, Kaj Wiik wrote:

OK, I see, thanks. I found this from the module source (client.py):

            if sys.version_info[0] < 3:
                argcount = self.on_connect.func_code.co_argcount
            else:
                argcount = self.on_connect.__code__.co_argcount

It might be possible to provide this.   The PyCall function wrapper could add a func_code (or __code__ in Py3) attribute that returns an object with at least some of the attributes of the Python func_code, e.g. co_argcount (the number of arguments), co_filename (the file where it is defined), co_firstlineno (the line number), co_name (the name of the function).

An inherent difficulty is that Julia functions can have multiple methods with different argument counts (and defined in different files etcetera), but I guess we could look at methods(f) to see if all of the methods have the same number of arguments, or return the maximum number of arguments or something.

(PyCall PRs welcome, of course.)

[Cedric St-Jean]

Maybe this will work?

client.on_connect = PyCall.jlfun2pyfun(on_connect)
client.on_message = PyCall.jlfun2pyfun(on_message)

jlfun2pyfun simply inserts an extra level of indirection through a native Python lambda, which does have a func_code.

[Steven G. Johnson]

On Wednesday, April 27, 2016 at 8:29:02 PM UTC-4, Cedric St-Jean wrote:

Maybe this will work?

client.on_connect = PyCall.jlfun2pyfun(on_connect)
client.on_message = PyCall.jlfun2pyfun(on_message)

jlfun2pyfun simply inserts an extra level of indirection through a native Python lambda, which does have a func_code.

It does, but the lambda has a func_code.co_argcount of 0 (because it is a varargs function that just passes through all its arguments to the underlying Julia function), which is probably not what the library wants here.