Meteor ddp client for python

[Harrison Harnisch]

After trying a few Meteor clients written in python I found them to be missing features, missing documentation, or flat out didn't work. So I made this:

https://github.com/hharnisc/python-meteor

$ pip install python-meteor

It's got the new bcrypt login system implemented and keeps a local copy of subscribed data in memory (updated in real-time).  Currently it's got a basic implementation for `find` and `findOne` (data is stored/queried in/from a python dictionary), but I plan on implementing minimongo like features at some point (or preferably a python package that does something close)

I'd love to get some feedback on this if you get a chance to try it out!

[Dax]

Hi Harrison,

Tried out python-meteor. In linux pip install python-meteor works ok. But pip install fails when tried in osx mavericks using python installed via homebrew, though I am not sure if its just my osx/homebrew setup

https://github.com/hharnisc/python-meteor/issues/1

Regards,

Dax

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meteor-talk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Upcat Review @ upcat.org

[Harrison Harnisch]

I was using hashed and requiring it in the package. Hashed was made part of the python in v2.5 (it existed for some distros at some point pre 2.5).  I've made the change the package, should be fixed now:

Install and upgrade:

$ pip install python-meteor -U

Thanks for the report Dax

- Harrison

[Dax]

both upgrade and install of python-meteor now works. Thanks!

I tried creating a sample python script based on the python-meteor documentation. 

What is the best way to keep the script running and properly shutting it down (due to thread).

I seem not to get any results in find. 

Below is my python script and sample output, connecting to a running meteor (0.8.2) todos sample

---python---

from MeteorClient import MeteorClient

client = MeteorClient('ws://127.0.0.1:3000/websocket')

def subscribed(subscription):

    print '* SUBSCRIBED {}'.format(subscription)

def unsubscribed(subscription):

    print '* UNSUBSCRIBED {}'.format(subscription)

def added(collection, id, fields):

    print '* ADDED {} {}'.format(collection, id)

    for key, value in fields.items():

        print '  - FIELD {} {}'.format(key, value)

def connected():

    print '* CONNECTED'

def subscription_callback(error):

    if error:

        print error

client.on('subscribed', subscribed)

client.on('unsubscribed', unsubscribed)

client.on('added', added)

client.on('connected', connected)

client.connect()

client.subscribe('lists')

client.subscribe('todos')

print client, client.__dict__, client.collection_data.data

all_lists = client.find('lists', selector={})

all_todos = client.find('todos', selector={})

print all_lists, all_todos

client.unsubscribe('lists')

client.unsubscribe('todos')

---output---

(py27env)Daxs-MacBook-Pro:util dax$ python sample_todo_ddp.py 

* CONNECTED

<MeteorClient.MeteorClient object at 0x10f86c750> {'ddp_client': <DDPClient.DDPClient object at 0x10f86c7d0>, 'connected': True, '_events': defaultdict(<function <lambda> at 0x10f862c08>, {'added': [<function added at 0x10f86eaa0>], 'connected': [<function connected at 0x10f86eb18>], 'new_listener': [], 'subscribed': [<function subscribed at 0x10f86e9b0>], 'unsubscribed': [<function unsubscribed at 0x10f86ea28>]}), 'collection_data': <MeteorClient.CollectionData object at 0x10f86c790>, 'subscriptions': {'todos': '2', 'lists': '1'}} {}

[] []

* UNSUBSCRIBED lists

* UNSUBSCRIBED todos

Exception in thread WebSocketClient (most likely raised during interpreter shutdown):

Thanks,

Dax

[Harrison Harnisch]

I played around with your example a bit (see below)

There's still an exception thrown if you try to do anything with the client after the while loop (like unsubscibe). I'll have to put some more work into getting the websocket to close out gracefully.  But you should still be able to get the python logic flushed out in the mean time. Let me know how this works out for you! 

from MeteorClient import MeteorClient

client = MeteorClient('ws://127.0.0.1:3000/websocket')

def subscribed(subscription):

    print '* SUBSCRIBED {}'.format(subscription)

def unsubscribed(subscription):

    print '* UNSUBSCRIBED {}'.format(subscription)

def added(collection, id, fields):

    print '* ADDED {} {}'.format(collection, id)

    for key, value in fields.items():

        print '  - FIELD {} {}'.format(key, value)

    # query the data each time something has been added to

    # a collection to see the data `grow`

    all_lists = client.find('lists', selector={})

    print 'Lists: {}'.format(all_lists)

    print 'Num lists: {}'.format(len(all_lists))

    

    # if collection == 'list' you could subscribe to the list here

    # with something like

    # client.subscribe('todos', id)

    # all_todos = client.find('todos', selector={})

    # print 'Todos: {}'.format(all_todos)

def connected():

    print '* CONNECTED'

def subscription_callback(error):

    if error:

        print error

client.on('subscribed', subscribed)

client.on('unsubscribed', unsubscribed)

client.on('added', added)

client.on('connected', connected)

client.connect()

client.subscribe('lists')

# todos needs a list _id to subscribe to

# client.subscribe('todos')

# (sort of) hacky way to keep the client alive

# ctrl + c to kill the script

while True:

    try:

        client.ddp_client.run_forever()

    except KeyboardInterrupt:

        break

# Behind the scenes the client is getting data

# and updating a dictionary containing all the

# subscribed data. (currently) A query is only

# ran against the dictionary and is not

# reactive like a queryset. It's simply a snapshot.

[Dax]

The updated example works great! thanks a bunch.

Next will try experimenting on including some sqlalchemy/django orm code

[Harrison Harnisch]

That's awesome! I'm excited to see what you come up with, sounds worthy of a lightning talk when you get something going.

[Dax]

Modified your example, added sqlalchemy and saved to an sqlite database the lists collection. So far data flow is just one way, meteor/mongo -> sqlalchemy/sqlite.

I am currently looking at what others have done regarding integrating python apps and meteor.

Thanks a lot for python-meteor!

[Anzel Lai]

Very interesting. 

I have a project actually working across python and meteor and this seems a good start for bridging those apps.

Will keep an close eye on this, thank you.

[Harrison Harnisch]

Looks like the project is coming together! Thinking about how you get data flowing both ways you might have an easier time using the core python-ddp package. The python-meteor package will store all subscribed data in memory whereas the python-ddp package will just give you the add/change/remove events. Or even better you could modify the CollectionData in python-meteor class to use sqlalchemy rather than the "in-memory" backend. Love to hear your thoughts as backend plugins is something I want to get to!

- Harrison

On Tue, Jul 8, 2014 at 04:19 AM, Dax<d...@upcat.org>, wrote:

Modified your example, added sqlalchemy and saved to an sqlite database the lists collection. So far data flow is just one way, meteor/mongo -> sqlalchemy/sqlite.

I am currently looking at what others have done regarding integrating python apps and meteor.

Thanks a lot for python-meteor!

On Wed, Jul 2, 2014 at 11:57 PM, Harrison Harnisch <hhar...@gmail.com> wrote:

That's awesome! I'm excited to see what you come up with, sounds worthy of a lightning talk when you get something going.

On Wednesday, July 2, 2014 8:38:42 AM UTC-7, Dax wrote:

The updated example works great! thanks a bunch.

Next will try experimenting on including some sqlalchemy/django orm code

On Wed, Jul 2, 2014 at 9:40 AM, Harrison Harnisch <hhar...@gmail.com> wrote:

I played around with your example a bit (see below)

There's still an exception thrown if you try to do anything with the client after the while loop (like unsubscibe). I'll have to put some more work into getting the websocket to close out gracefully.  But you should still be able to get the python logic flushed out in the mean time. Let me know how this works out for you! 

from MeteorClient import MeteorClient

--

Upcat Review @ upcat.org

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meteor-talk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Upcat Review @ upcat.org

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meteor-talk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Upcat Review @ upcat.org

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meteor-talk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Upcat Review @ upcat.org

--
You received this message because you are subscribed to a topic in the Google Groups "meteor-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/meteor-talk/JLr1Ho_y3vE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to meteor-talk...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

<meteor2sqlite.py>

[Harrison Harnisch]

@anzel I currently use it as a bridge to control a bunch of test clients. So far it's been working like a charm.

- Harrison

[Dax]

Not sure if dataset will be helpful.

http://dataset.readthedocs.org/en/latest/index.html

dataset provides two key functions that make using SQL databases in Python a breeze:

A simple abstraction layer removes most direct SQL statements without the necessity for a full ORM model - essentially, databases can be used like a JSON file or NoSQL store.
Database contents can be exported (frozen) using a sophisticated plain file generator with JSON and CSV support. Exports can be configured to include metadata and dynamic file names depending on the exported data. The exporter can also be used as a command-line tool, datafreeze.

[Dax]

speaking of test clients, I modified the sample script to run X number of meteor clients. python meteorXclient.py <numclients>. Connected locally (loopback) for 1-2k number of clients and both the script and meteor seems ok. Will still have to check how it load and performance goes client is on a different machine

https://gist.github.com/anonymous/634aac5d77c31554ef88

Dax

[Harrison Harnisch]

I wonder what the result would be if you subscribe to a published collection. It's my understanding that the server keeps a copy of each client's subscribed data in memory. You've proven that this script and meteor can handle a fair amount of connections!

[Harrison Harnisch]

I've just added some new features that allow python-meteor to auto-reconnect on network errors or when meteor restarts during development.

When meteor-client reconnects it will log in (if a login was done during the session) and will re-subscribe to the data. This means the python clients can handle network issues, maintenance downtime, and meteor restarts when developing/debugging!

To get the updated version

$ pip install python-meteor -U

- Harrison