object cache idea

18 views
Skip to first unread message

Brian Hammond

unread,
Dec 9, 2009, 5:16:07 PM12/9/09
to jsonpickle
Hello:

I'm playing around with an python object data store based on JSON
encoding, append-only journals, replication, etc.

Part of the data store is an in-memory object cache for "active"
objects, used to amortize the cost of disk reads. Inactive objects
are purged from the cache to save on memory. Inactive objects can
always be loaded from disk and put into this object cache.

The serialization format for these objects will be JSON, and I'll use
jsonpickle to serialize/deserialize them.

Blah, blah. Nothing new here.

Jsonpickle quite graciously encodes the entire object graph associated
with a given root object, but I'd like to store references to objects
in the graph instead of the object (well, serialization of the
object).

Such a reference would be tightly coupled to the aforementioned object
cache. Such a reference might simply be an "object id" (note: not id
() from python but say a UUID or equivalent concept).

Here, let's take an example:

class User(object):
def __init__(self, name):
self.name = name
self.friends = []

bob = User("bob")
al = User("al")
bob.friends.append(al)

import jsonpickle
print jsonpickle.encode(bob)

Running the above yields:

{"py/object": "__main__.User", "friends": [{"py/object":
"__main__.User", "friends": [], "name": "al"}], "name": "bob"}

Note that al has been flattened into bob's friends list as expected.

My goal is for the result to instead be:

{"py/object": "__main__.User", "friends": [{"myref":
"1bd5351a-75bc-45a8-93f6-0698692d389e"}], "name": "bob"}

That is, al's "id" (here, from str(uuid.uuid4())) is stored in bob's
friends list instead of al. I'm leaving out how I assign IDs as its
tangential; all that is important is that an object can be get/put by
such an ID using the cache interface.

On unpickling, if I encounter a "myref" I'd like to look up the object
by id in my object cache (and load from disk if not present, etc.).

I will fork the project on github to try and do this. But I figured
I'd throw the idea out there to see if anyone had any tips as I've
only been looking at jsonpickle for about 45 minutes now.

I saw the Handler registry and _mkref in pickler. My solution will
probably have something to do with these (I think).

Thanks for any tips/advice/info.

Brian

Reply all
Reply to author
Forward
0 new messages