WAMP and serialization of Python class

536 views
Skip to first unread message

Florian Conrady

unread,
Dec 19, 2014, 7:41:51 AM12/19/14
to autob...@googlegroups.com
Hi,

I tried to create a simple example where I register and call a procedure that has an instance of a class as argument (see code pasted at bottom). It fails with the following message:

./client3.py 
session ready
procedure registered
could not call procedure: Unable to serialize WAMP application payload (<__main__.SomeClass instance at 0x1f86878> is not JSON serializable)

Should serialization of this simple object work out of the box or do I need to do more? What is the right approach for this? (I started specializing JsonSerializer, but that resulted in further problems.)

Thanks
Florian


#!/usr/bin/python

from autobahn.twisted.wamp import ApplicationSession
from twisted.internet.defer import inlineCallbacks
from autobahn.twisted.util import sleep

class SomeClass:
   def __init__(self,name):
      self.name = name

class MyComponent(ApplicationSession):

   @inlineCallbacks
   def onJoin(self, details):
      print("session ready")

      def printName(obj):
         print(obj.name)

      try:
         yield self.register(printName, u'com.myapp.printname')
         print("procedure registered")
      except Exception as e:
         print("could not register procedure: {0}".format(e))

      sleep(1)

      try:
         someObj = SomeClass("Oberstet")
         yield self.call(u'com.myapp.printname', someObj)
         print("procedure called")
      except Exception as e:
         print("could not call procedure: {0}".format(e))


from autobahn.twisted.wamp import ApplicationRunner

runner = ApplicationRunner(url = "ws://localhost:8080/ws1", realm = "realm1")
runner.run(MyComponent)














Clayton Daley

unread,
Dec 20, 2014, 11:58:22 AM12/20/14
to autob...@googlegroups.com
I'm facing the same issue and am a little further in the research process.  It's my understanding that this is the expected behavior.  To send classes over the WebSockets, you need a different serializer like JsonPickle.  See as reference:
Unfortunately, I haven't found an example of how to actually implement it so I'm optimistic that someone can provide this to us both (even better if it were available out-of-the-box).

Florian Conrady

unread,
Dec 21, 2014, 7:50:12 PM12/21/14
to autob...@googlegroups.com
Hi Clayton,

I just finished writing code that seems to solve the issue for me and I did use jsonpickle for it. (I hadn't yet read your reply that suggested jsonpickle as well!) I wrapped the register and call methods of the WAMP application session. In the register method shown below I first "serialize" the method to be registered, i.e. I transform the method with Python object parameter into a function with a JSON string as parameter. Then I register this "serialized" version of the method with WAMP. In the call method I first convert the object parameter into a JSON string and then pass it to the WAMP call method. I pasted part of the code below to explain the gist of it. (Let me know if you need more details.)

Thanks
Florian

    @inlineCallbacks
    def register(self, instance, method):
        uri = self.getUri(instance, method)
        def serialize(func):
            def wrapper(jsonString):
                # Deserialize Json string to Python object.
                obj = jsonpickle.decode(jsonString)
                return func(obj)
            return wrapper
        print("Trying to register %s." % uri)
        yield self.getSession().register(serialize(method), uri)
        print("Registered %s." % uri)

    @inlineCallbacks
    def call(self, methodUri, parameter, callback):
        jsonString = jsonpickle.encode(parameter)
        result = yield self.getSession().call(methodUri, jsonString)
        callback(result)

Florian Conrady

unread,
Dec 22, 2014, 12:38:31 PM12/22/14
to autob...@googlegroups.com
In addition to wrapping the Autobahn register and call methods I also tried specializing the serialize / unserialize method in autobahn/wamp/serializer.py. However, in that case, my code fails. The connection is dropped for some reason.

$ ./client4.py 
WAMP-over-WebSocket transport lost: wasClean = False, code = 1006, reason = 'connection was closed uncleanly (I failed the WebSocket connection by dropping the TCP connection)'
Traceback (most recent call last):
  File "/home/florian/Downloads/autobahn-0.9.4-2/autobahn/wamp/websocket.py", line 75, in onClose
    self._session.onClose(wasClean)
AttributeError: WampWebSocketClientProtocol instance has no attribute '_session'

I pasted the complete code of the script below. When I modify the JsonObjectSerializer in the Autobahn source code itself, I do not get this error message. So there must be something wrong in the way I am specializing it. Could someone give me a hint at what should be done differently? Note that in the script I have not yet changed anything about the serialization. At this point JsonPickleSerializer should behave identically to JsonSerializer.

Thanks
Florian

#!/usr/bin/python

############################################################
# Specializing serialization ###############################
############################################################

from autobahn.wamp.serializer import *
from autobahn.wamp.interfaces import IObjectSerializer, ISerializer

class JsonPickleObjectSerializer:

    BINARY = False

    def __init__(self, batched = False):
        """
        Ctor.

        :param batched: Flag that controls whether serializer operates in batched mode.
        :type batched: bool
        """
        self._batched = batched

    def serialize(self, obj):
        """
        Implements :func:`autobahn.wamp.interfaces.IObjectSerializer.serialize`
        """
        _dumps = lambda obj: json.dumps(obj, separators = (',',':'), ensure_ascii = False)
        s = _dumps(obj)
        if isinstance(s, six.text_type):
            s = s.encode('utf8')
        if self._batched:
            return s + b'\30'
        else:
            return s

    def unserialize(self, payload):
        """
        Implements :func:`autobahn.wamp.interfaces.IObjectSerializer.unserialize`
        """
        if self._batched:
            chunks = payload.split(b'\30')[:-1]
        else:
            chunks = [payload]
        if len(chunks) == 0:
            raise Exception("batch format error")
        _loads = json.loads
        return [_loads(data.decode('utf8')) for data in chunks]

IObjectSerializer.register(JsonPickleObjectSerializer)

class JsonPickleSerializer(Serializer):

   SERIALIZER_ID = "jsonpickle"
   MIME_TYPE = "application/json"

   def __init__(self, batched = False):
      """
      Ctor.

      :param batched: Flag to control whether to put this serialized into batched mode.
      :type batched: bool
      """
      Serializer.__init__(self, JsonPickleObjectSerializer(batched = batched))
      if batched:
         self.SERIALIZER_ID = "jsonpickle.batched"

ISerializer.register(JsonPickleSerializer)

############################################################
# Application component ####################################
############################################################

from autobahn.twisted.wamp import ApplicationSession
from twisted.internet.defer import inlineCallbacks
from autobahn.twisted.util import sleep

class SomeClass:
   def __init__(self,name):
      self.name = name

class MyComponent(ApplicationSession):

   @inlineCallbacks
   def onJoin(self, details):
      print("session ready")

      # Method with string parameter.
      def printName(name):
         print(name)

      try:
         yield self.register(printName, u'com.myapp.printname')
         print("procedure registered")
      except Exception as e:
         print("could not register procedure: {0}".format(e))

      sleep(1)

      try:
         someName = "Tobias"
         yield self.call(u'com.myapp.printname', someName)
         print("procedure called")
      except Exception as e:
         print("could not call procedure: {0}".format(e))

      # Method with object parameter.
      def printObjectName(obj):
         print(obj.name)

      try:
         yield self.register(printObjectName, u'com.myapp.printobjectname')
         print("procedure registered")
      except Exception as e:
         print("could not register procedure: {0}".format(e))

      sleep(1)

      try:
         someObj = SomeClass("Oberstein")
         yield self.call(u'com.myapp.printobjectname', someObj)
         print("procedure called")
      except Exception as e:
         print("could not call procedure: {0}".format(e))

############################################################
# Running client ###########################################
############################################################

from autobahn.twisted.wamp import ApplicationSessionFactory
from autobahn.twisted.websocket import WampWebSocketClientFactory
from autobahn.twisted.choosereactor import install_reactor
from twisted.internet.endpoints import clientFromString
from twisted.internet.defer import inlineCallbacks
from autobahn.wamp import types

config = types.ComponentConfig(realm = "realm1")
sessionFactory = ApplicationSessionFactory(config)
sessionFactory.session = MyComponent
sessionFactory.sessionInstance = None
serializers = []
serializers.append(JsonPickleSerializer()) #JsonSerializer()) JsonPickleSerializer())
transportFactory = WampWebSocketClientFactory(sessionFactory, "ws://localhost:8080/ws1", serializers = serializers, debug = False, debug_wamp = True)
reactor = install_reactor()
client = clientFromString(reactor, "tcp:localhost:8080")
client.connect(transportFactory)

reactor.run()

Tobias Oberstein

unread,
Dec 30, 2014, 8:25:33 AM12/30/14
to autob...@googlegroups.com
Hi Florian,

WAMP is language agnostic. That means, given a WAMP procedure, _any_ WAMP client (in any language) is supposed to be able to call the procedure.

That being said, it should still be possible to do if you insist;)

Don't follow the overrriding "call, register, .." approach .. this leads to insanity. Your approach with JsonPickleObjectSerializer is the right one.

With your code, change this

SERIALIZER_ID = "jsonpickle"

and leave it at "json".

a) the output of jsonpickle is still JSON (not MsgPack or anything)
b) no WAMP router will understand how to treat "jsonpickle" serialization
c) the fact that the JSON is "special" (e.g. contains Python classnames) is irrelevant to a WAMP router

Of course to make it work, you'll then need to use jsonpickle.encode etc in your code below.

Cheers,
/Tobias

Clayton Daley

unread,
Dec 30, 2014, 5:14:07 PM12/30/14
to autob...@googlegroups.com
Tobias,

I understand how this approach aligns with platform independence.  At the same time,
  • If I want to use Autobahn in a single-language environment, it should be easy to drop in a serializer that minimizes complexity. Better for the community if I'm using autobahn for everything I write... rather than switching to something that's simpler for a single-language implementation.
  • For Python (at least), it might be better to use a "safe" serializer like serpent (https://pypi.python.org/pypi/serpent).  Incidentally, this already includes support for Python, Java and C#/.NET.
  • When it comes to some packages and types, it'd be nice to have a library of (community contributed) plugins that serialize (and de-serialize) specific objects (or packages) in one or more languages.  For example, Python's timedelta is unsupported by JsonSerialize but is generated by MySQLdb and supported by PyMongo.  Certainly, anything supported widely in Mongo clients could be duplicated in Autobahn.  
Especially on this last point, I'm not clear if this is easy (but not documented) or difficult.  Certainly would be nice to know (and have some pointers to best practices) so users like Florian and I can implement the capabilities correctly and easily share with other users who may be interested.

Clayton

Tobias Oberstein

unread,
Dec 31, 2014, 6:35:49 AM12/31/14
to autob...@googlegroups.com
Hi Clayton,

> I understand how this approach aligns with platform independence. At
> the same time,
>
> * If I want to use Autobahn in a single-language environment, it
> should be easy to drop in a serializer that minimizes complexity.
> Better for the community if I'm using autobahn for everything I
> write... rather than switching to something that's simpler for a
> single-language implementation.

If you want language specific, transparent type marshalling (like Python
serpent) or stuff like transparent object remoting (like CORBA or Java
RMI), then WAMP probably isn't for you.

WAMP was deliberately designed _not_ following these approaches. We like
to keep stuff simple and flexible.

> * For Python (at least), it might be better to use a "safe" serializer
> like serpent (https://pypi.python.org/pypi/serpent). Incidentally,
> this already includes support for Python, Java and C#/.NET.

This seems to use a proprietory, Python specific serialization format.

This cannot be used with WAMP, since WAMP currently only specifies JSON
and MsgPack on the wire.

We won't add language specific serializers to WAMP as this does not
serve the WAMP ecosystem, but fragments it.

> * When it comes to some packages and types, it'd be nice to have a
> library of (community contributed) plugins that serialize (and
> de-serialize) specific objects (or packages) in one or more
> languages. For example, Python's timedelta is unsupported by
> JsonSerialize but is generated by MySQLdb and supported by PyMongo.
> Certainly, anything supported widely in Mongo clients could be
> duplicated in Autobahn.
>
> Especially on this last point, I'm not clear if this is easy (but not
> documented) or difficult. Certainly would be nice to know (and have
> some pointers to best practices) so users like Florian and I can
> implement the capabilities correctly and easily share with other users
> who may be interested.

Sure. You can create a custom JSON serializer that is capable of
serializing Python timedelta. It will need to have means to
differentiate normal strings from timedelta strings for deserialization.

You would start creating custom versions of

https://github.com/tavendo/AutobahnPython/blob/master/autobahn/autobahn/wamp/serializer.py#L206

and plug that into the respective WAMP transport.

Cheers,
/Tobias
> Should serialization of this simple object work out of the box or do I need to do more? What is the right approach for this? (I started specializingJsonSerializer, but that resulted in further problems.)
>
>
> Thanks
>
> Florian
>
>
>
> #!/usr/bin/python
>
> from autobahn.twisted.wamp import ApplicationSession
> from twisted.internet.defer import inlineCallbacks
> from autobahn.twisted.util import sleep
>
> class SomeClass:
> def __init__(self,name):
> self.name <http://self.name> = name
>
> class MyComponent(ApplicationSession):
>
> @inlineCallbacks
> def onJoin(self, details):
> print("session ready")
>
> def printName(obj):
> print(obj.name <http://obj.name>)
>
> try:
> yield self.register(printName, u'com.myapp.printname')
> print("procedure registered")
> except Exception as e:
> print("could not register procedure: {0}".format(e))
>
> sleep(1)
>
> try:
> someObj = SomeClass("Oberstet")
> yield self.call(u'com.myapp.printname', someObj)
> print("procedure called")
> except Exception as e:
> print("could not call procedure: {0}".format(e))
>
>
> from autobahn.twisted.wamp import ApplicationRunner
>
> runner = ApplicationRunner(url = "ws://localhost:8080/ws1", realm = "realm1")
> runner.run(MyComponent)
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/cc026982-b503-4003-a907-a57c1a4e6484%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/cc026982-b503-4003-a907-a57c1a4e6484%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

Clayton Daley

unread,
Dec 31, 2014, 5:29:53 PM12/31/14
to autob...@googlegroups.com
Obviously, this is your thing... but I always assume that the amount of support (financial and volunteer, as well as commercial revenue) for a project is roughly proportional to the number of users.
  • Telling the large fraction of users -- who are building a single language system -- to go somewhere else for ease of use seems counter-productive. I figure it actually invites fragmentation by encouraging the development/ enhancement of competitive platforms.
  • Your JsonPickle response seems to suggest that the only barrier is a serializer... so the rest of the platform (and any future improvements) would be shared by the entire community.
> Sure. You can create a custom JSON serializer that is capable of serializing Python timedelta. It will need to have means to differentiate normal strings from timedelta strings [empahsis added] for deserialization.

Agreed.  Perhaps a generic (at least standard to WAMP) solution to this problem would be warranted.  My impression is the following (but you know your architecture better to criticize):
  • I believe Florian and I are really concerned about serializing args and kwargs.
  • The current architecture only serializes once... on the complete message.  Changing the message-level serializer works, but it's massive overkill.  Every receiver must use that substitute serializer... which means that the serializer must exist on every supported platform.
  • It would be better to implement a set of (optional) serializer for args/kwargs. Recurse through dicts and lists. Skip JSON native types.  Add some metadata identifying the serializer (None for native, string identifying other serialized).
Clayton Daley


To post to this group, send email to autob...@googlegroups.com

--
You received this message because you are subscribed to a topic in the Google Groups "Autobahn" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/autobahnws/6Pk0c8JiSn0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to autobahnws+unsubscribe@googlegroups.com.
To post to this group, send email to autob...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/54A3DF91.1060804%40gmail.com.

Tobias Oberstein

unread,
Jan 3, 2015, 7:53:31 AM1/3/15
to autob...@googlegroups.com
Hi Clayton,

happy new year to you also!

I reply not inline, because I first need to understand better: it might
be we are not on the same track yet.

==

We should differentiate the following in the discussion:

1. the WAMP serialization format at the wire-level
2. the serializers in WAMP client libraries (and routers)

The WAMP spec currently defines two serialization formats for 1):

* JSON (with an added/optional convention for binary data)
* MsgPack

==

jsonpickle (https://jsonpickle.github.io/) is a serializer library for
Python that produces _standard_ JSON.

However, the JSON produces will follow certain conventions to add Python
language specific information:

$ python
Python 2.7.9 (default, Dec 10 2014, 12:24:55) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import jsonpickle
>>> class Foo:
... pass
...
>>> f = Foo()
>>> jsonpickle.encode(f)
'{"py/object": "__main__.Foo"}'


Any WAMP router will happily process messages which have stuff
serialized using this serializer in args or kwargs.

Any WAMP client library (supporting JSON) will be able to process such
messages .. it just won't interpret the "magic attributes" like
"py/object" specially.

The question of how to hook jsonpickle into AutobahnPython is totally
specific to AutobahnPython, and this should be possible today.

If you want that, and can't get it working, come back, we'll get it done.

Note: I still don't think it's wise to restrict your choices (language)
by using above, but if you are doing a Python-only app it'll work, and
it should be a developer choice to do so.

==

Serpent (https://github.com/irmen/Serpent) is both a serialization
format _and_ a serialization library (for Java, .NET, and Python
https://pypi.python.org/pypi/serpent)

The serialization format doesn't seem to be documented, but it is
definitely not JSON, but something "custom":

$ python
Python 2.7.9 (default, Dec 10 2014, 12:24:55) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import serpent
>>> class Foo:
... pass
...
>>> f = Foo()
>>> serpent.dumps(f)
"# serpent utf-8 python2.6\n{'__class__':'Foo'}"

Are you suggesting a WAMP based system should be able to use this custom
serialization format?

a) using this serialization format for the complete WAMP messages: I'
not convinced this makes sense, since the format is totally proprietory
and not even documented. Routers would need to fully implement the
format to do their job.

b) using this seriazation format for args/kwargs only within WAMP
messages (but not for the "surrounding" stuff in WAMP messages): this is
also problematic for different reasons:

- a WAMP router is supposed to be able to transparently translate
between different serialization formats, so clients using different
serialization formats are shielded from this.

- a WAMP router is supposed to ensure anything forwarded strictly
follows the serialization format used to protect innocent client
libraries from being attacked (security). if the router doesn't
understand args/kwargs serialization format, it no longer can provide
that service

- a WAMP router might be configured to do "schema validation" on app
payload (args/kwargs) - this is actually already partially implemented
in Crossbar.io. E.g. you can require the payload of a WAMP procedure
follow a schema, and the router will validate, and deny the forwarding
of a call if the payload doesn't match

==

Cheers,
/Tobias


Am 31.12.2014 um 23:29 schrieb Clayton Daley:
> Obviously, this is your thing... but I always assume that the amount of
> support (financial and volunteer, as well as commercial revenue) for a
> project is roughly proportional to the number of users.
>
> * Telling the large fraction of users -- who are building a single
> language system -- to go somewhere else for ease of use seems
> counter-productive. I figure it actually invites fragmentation by
> encouraging the development/ enhancement of competitive platforms.
> * Your JsonPickle response seems to suggest that the only barrier is a
> serializer... so the rest of the platform (and any future
> improvements) would be shared by the entire community.
>
> > Sure. You can create a custom JSON serializer that is capable of
> serializing Python timedelta. It will need to have *means to
> differentiate normal strings from timedelta strings [empahsis added]*
> for deserialization.
>
> Agreed. Perhaps a generic (at least standard to WAMP) solution to this
> problem would be warranted. My impression is the following (but you
> know your architecture better to criticize):
>
> * I believe Florian and I are really concerned about serializing args
> and kwargs.
> * The current architecture only serializes once... on the complete
> message. Changing the message-level serializer works, but it's
> massive overkill. Every receiver must use that substitute
> serializer... which means that the serializer must exist on every
> supported platform.
> * It would be better to implement a set of (optional) serializer for
> args/kwargs. Recurse through dicts and lists. Skip JSON native
> types. Add some metadata identifying the serializer (None for
> native, string identifying other serialized).
>
> Clayton Daley
>
> On Wed, Dec 31, 2014 at 6:35 AM, Tobias Oberstein
> <tobias.o...@gmail.com <mailto:tobias.o...@gmail.com>> wrote:
>
> Hi Clayton,
>
> > I understand how this approach aligns with platform independence. At
>
> the same time,
>
> * If I want to use Autobahn in a single-language environment, it
> should be easy to drop in a serializer that minimizes
> complexity.
> Better for the community if I'm using autobahn for everything I
> write... rather than switching to something that's simpler
> for a
> single-language implementation.
>
>
> If you want language specific, transparent type marshalling (like
> Python serpent) or stuff like transparent object remoting (like
> CORBA or Java RMI), then WAMP probably isn't for you.
>
> WAMP was deliberately designed _not_ following these approaches. We
> like to keep stuff simple and flexible.
>
> * For Python (at least), it might be better to use a "safe"
> serializer
> like serpent (https://pypi.python.org/pypi/__serpent
> <https://pypi.python.org/pypi/serpent>). Incidentally,
> https://github.com/tavendo/__AutobahnPython/blob/master/__autobahn/autobahn/wamp/__serializer.py#L206
> self.name <http://self.name> <http://self.name> = name
>
> class MyComponent(__ApplicationSession):
>
> @inlineCallbacks
> def onJoin(self, details):
> print("session ready")
>
> def printName(obj):
> print(obj.name <http://obj.name>
> <http://obj.name>)
>
> try:
> yield self.register(printName,
> u'com.myapp.printname')
> print("procedure registered")
> except Exception as e:
> print("could not register procedure:
> {0}".format(e))
>
> sleep(1)
>
> try:
> someObj = SomeClass("Oberstet")
> yield self.call(u'com.myapp.__printname',
> someObj)
> print("procedure called")
> except Exception as e:
> print("could not call procedure: {0}".format(e))
>
>
> from autobahn.twisted.wamp import ApplicationRunner
>
> runner = ApplicationRunner(url =
> "ws://localhost:8080/ws1", realm = "realm1")
> runner.run(MyComponent)
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from
> it, send
> an email to autobahnws+unsubscribe@__googlegroups.com
> <mailto:autobahnws%2Bunsu...@googlegroups.com>
> <mailto:autobahnws+_...@googlegroups.com
> <mailto:autobahnws%2Bunsu...@googlegroups.com>>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>
> <mailto:autobahnws@__googlegroups.com
> <mailto:autob...@googlegroups.com>>.
> To view this discussion on the web visit
> https://groups.google.com/d/__msgid/autobahnws/cc026982-__b503-4003-a907-a57c1a4e6484%__40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/cc026982-b503-4003-a907-a57c1a4e6484%40googlegroups.com>
> <https://groups.google.com/d/__msgid/autobahnws/cc026982-__b503-4003-a907-a57c1a4e6484%__40googlegroups.com?utm_medium=__email&utm_source=footer
> <https://groups.google.com/d/msgid/autobahnws/cc026982-b503-4003-a907-a57c1a4e6484%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "Autobahn" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/__topic/autobahnws/6Pk0c8JiSn0/__unsubscribe
> <https://groups.google.com/d/topic/autobahnws/6Pk0c8JiSn0/unsubscribe>.
> To unsubscribe from this group and all its topics, send an email to
> autobahnws+unsubscribe@__googlegroups.com
> <mailto:autobahnws%2Bunsu...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/__msgid/autobahnws/54A3DF91.__1060804%40gmail.com
> <https://groups.google.com/d/msgid/autobahnws/54A3DF91.1060804%40gmail.com>.
>
> For more options, visit https://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/CACKEAnVCh592WY1Hn07eDkJE7D%3DigXu6jtAJue65WtdBQGp-7w%40mail.gmail.com
> <https://groups.google.com/d/msgid/autobahnws/CACKEAnVCh592WY1Hn07eDkJE7D%3DigXu6jtAJue65WtdBQGp-7w%40mail.gmail.com?utm_medium=email&utm_source=footer>.
Message has been deleted

Clayton Daley

unread,
Jan 6, 2015, 6:59:29 PM1/6/15
to autob...@googlegroups.com
[Apologize for the duplicate... forgot to trim signature so I deleted my last post off the list]

Thanks for your thoughtful reply and happy new year to you too.  There are a lot of moving pieces, but my perfect world looks something like:
  • I have some channels that are Python-only.  I'd prefer to use a minimum-friction (Python only is fine) serializer so I don't have to think about the types/objects in function signatures.
    • JsonPickle has the right behavior.  Python consumers (who share JsonPickle) will deserialize these objects without breaking my line of communication with others. It's my problem to only send serialized objects to receivers that understand them.
    • However, Pickle is a recognized security vulnerability. I'd rather use something "safer" like Serpent in case one of the nodes in the system is breached.
  • I will definitely have some Python-JS channels and may add some Python-PHP channels.
    • If I ever want to transfer an object across one of these wires, I'd like a simple way to add support on both ends. I have to write the classes at both ends, but I'd rather not have to wrap every arg and every return that involves the object.
    • Something like the "py/object" syntax (e.g. "ab/<serializer>" for autobahn/<serializer>) would be fine for a representation, but I'd rather configure it in one place and have it automatically applied to any instance that crosses the wire.
It still seems to me like args/kwargs, the return, and both ends of the pub/sub are natural places to handle non-JSON types/classes. I think an ordered list of serializers has the right feel to it (iterate until one handles the type). The first can be a "serializer" that simply skips native types. I could follow that with custom Serializer for types supported in multiple platforms. Finally, I can toss in a catch-all serializer for Python classes for low friction.

If your only problem with Serpent is the binary (rather than string) representation, that's probably easy to fix. The standard is open-source and I've found Irmen accessible. Besides that, it seems like serpent is just as safe as a programmer who custom-serializes their own objects. If the receiver doesn't have serpent, they'll just see a meaningless representation.

Anyway... that's my mindset.

Clayton
 
[trimmed]

Tobias Oberstein

unread,
Jan 7, 2015, 4:49:49 AM1/7/15
to autob...@googlegroups.com
Hi Clayton,

1)
yes, both Pickle _and_ jsonpickle are a security vulnerability. e.g.
read here

https://blog.nelhage.com/2011/03/exploiting-pickle/

adjusted to jsonpickle:
https://gist.github.com/oberstet/3723147344967b9794c0

So while you should (technically) be able to use jsonpickle as a custom
serializer with AutobahnPython, I would _strongly_ recommend against
doing this (at least outside a strictly controlled environment).

2)
I have now tried it out, and I don't think it works like you expect:

https://gist.github.com/oberstet/6df18395c1c173a1cf89

As you can see, it'll happily serialize objects like timestamp or
DemoClass instances, but it will _not_ recreate instances of those classes.

So it (I guess) only solves 1 aspect of the 2 you are after: it can
serialize any Python object transparently, but _not_ the other direction.

Note that any system that is capable of doing the latter is likely a
security vulnerability. Please correct me if you think I am wrong!

Serpent: it's also an adhoc, non-standard serialization format. And I
have a problem with this, yes. It's not about binary/text.

Cheers,
/Tobias




Am 07.01.2015 um 00:59 schrieb Clayton Daley:
> [Apologize for the duplicate... forgot to trim signature so I deleted my
> last post off the list]
>
> Thanks for your thoughtful reply and happy new year to you too. There
> are a lot of moving pieces, but my perfect world looks something like:
>
> * I have some channels that are Python-only. I'd prefer to use a
> minimum-friction (Python only is fine) serializer so I don't have to
> think about the types/objects in function signatures.
> o JsonPickle has the right behavior. Python consumers (who share
> JsonPickle) will deserialize these objects without breaking my
> line of communication with others. It's my problem to only send
> serialized objects to receivers that understand them.
> o However, Pickle is a recognized security vulnerability. I'd
> rather use something "safer" like Serpent in case one of the
> nodes in the system is breached.
> * I will definitely have some Python-JS channels and may add some
> Python-PHP channels.
> o If I ever want to transfer an object across one of these wires,
> I'd like a simple way to add support on both ends. I have to
> write the classes at both ends, but I'd rather not have to wrap
> every arg and every return that involves the object.
> o Something like the "py/object" syntax (e.g. "ab/<serializer>"
> for autobahn/<serializer>) would be fine for a representation,
> but I'd rather configure it in one place and have it
> automatically applied to any instance that crosses the wire.
>
> It still seems to me like args/kwargs, the return, and both ends of the
> pub/sub are natural places to handle non-JSON types/classes. I think an
> ordered list of serializers has the right feel to it (iterate until one
> handles the type). The first can be a "serializer" that simply skips
> native types. I could follow that with custom Serializer for types
> supported in multiple platforms. Finally, I can toss in a catch-all
> serializer for Python classes for low friction.
>
> If your only problem with Serpent is the binary (rather than string)
> representation, that's probably easy to fix. The standard is open-source
> and I've found Irmen accessible. Besides that, it seems like serpent is
> just as safe as a programmer who custom-serializes their own objects. If
> the receiver doesn't have serpent, they'll just see a meaningless
> representation.
>
> Anyway... that's my mindset.
>
> Clayton
> [trimmed]
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/07af9de5-cf4d-49a8-9292-63482f65f397%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/07af9de5-cf4d-49a8-9292-63482f65f397%40googlegroups.com?utm_medium=email&utm_source=footer>.

Florian Conrady

unread,
Jan 11, 2015, 9:37:07 PM1/11/15
to autob...@googlegroups.com
Hi Tobias,

thanks for your reply!

>> WAMP is language agnostic. That means, given a WAMP procedure, _any_ WAMP client (in any language) is supposed to be able to call the procedure. <<

I realized later that jsonpickle is too Python specific for our purpose. We need communication between Java, Python and JavaScript.

>> With your code, change this

SERIALIZER_ID = "jsonpickle"

and leave it at "json". <<

I tried this and it did work! Now I know how to specialize Serializer. :-)

>> Don't follow the overrriding "call, register, .." approach .. this leads to insanity. <<

Why is making a custom Serializer better than overriding "call, register, ..."? Because then you have to write it only once?

We would like to go beyond the provided serializer, since we want to support simple nested classes of the type

class NestedClass:
   def __init__(self,name,id,anotherObj):
      self.name = name
      self.id = id
      self.anotherObj = anotherObj

class AnotherClass:
   def __init__(self,whatever):
      self.whatever = whatever

anotherObj = AnotherClass("Bla")
nestedObj = NestedClass("Bla",123456,anotherObj)

The challenge is that this is ambiguous when deserializing. The code would not be able to tell whether a JSON dictionary should become a Python object of some user-defined class or simply a Python dictionary.

To deal with this we are thinking of the following approach:
1. Arguments can be provided as is (e.g. of type NestedClass above) and will be converted to JSON using
json.dumps(obj, default=lambda o: o.__dict__)
2. Return values are produced by
json.loads(jsonString)
and are Python built-ins. When it is a dictionary, it is up to the user of our code to decide whether to convert it into an instance of a user-defined class or leave it as a dictionary.

Does this make sense?

Thanks
Florian
Reply all
Reply to author
Forward
0 new messages