November Meeting Topic

[Mark Ramm]

I think there was a suggestion for a Python on Windows meeting in
december. Bruce, are you up for that?

And anybody else got topic ideas for future months? I'd like to
have a bit of a plan so that we can advertise a bit better.

--Mark

[Zach Steindler]

Mark, do you mean a Python on Windows meeting in November?

If the December 3rd slot is still open, I'd love to talk about how to
kick ass at network programming in Python. I'm hoping to spend most of
December in Michigan/Ann Arbor. Things like:

Twisted, the good parts:
- Using deferred and inline callbacks to load-test your website in
~25 lines of python

Thrift:
- The huge advantage of separating message serialization and
message handling
- Setting up your shop with web-service oriented architecture
(thanks, Amazon!)

Tornado:
- When evented programming matters
- How it's different from other Python web stacks (like cherrypy + nginx)

-Zach

--
Meet, drink, hack - http://coffeehousecoders.org/
Calling all geeks! - http://a2geeks.org/

[Mark Ramm]

Yea, I'm all for:

* windows in november,
* networking/even driven python in december.

Anything for january?

--
Mark Ramm-Christensen
email: mark at compoundthinking dot com
blog: www.compoundthinking.com/blog

[Bruce Webber]

Yes, I can do a presentation on Python in the Windows Environment at
the November meeting. I have arranged to borrow a laptop with Windows.

My presentation will include:

* creating a small app with pygtk (I'm not going to explain pygtk, but
I want to create a GUI app to be used in the following steps)
* turning that app into a Windows EXE with py2exe
* creating a Windows installer (an MSI file) for that app using the
open source WIX toolset

If time permits, I also want to touch on several ways to read and
write Excel files, using:

* pywin32 (and there are some other nice things you can do with
pywin32, like map network drives)
* xlrd
* pyExcelerator
* xlwt (I just came across this recently)

-- Bruce

[Bruce Webber]

I'm sorry, but I won't be able to give the presentation tomorrow
night. I'm home sick with a cold or flu. I'm getting better but I
don't think it's a good idea for me to drive to AA tomorrow evening
and give a talk.

Can we tentatively aim for January (Zach will present in December)?
Sorry for the late notice!

-- Bruce

[Kevin Dangoor]

On Wed, Nov 4, 2009 at 3:49 PM, Bruce Webber <webber...@gmail.com> wrote:

I'm sorry, but I won't be able to give the presentation tomorrow
night. I'm home sick with a cold or flu. I'm getting better but I
don't think it's a good idea for me to drive to AA tomorrow evening
and give a talk.

Can we tentatively aim for January (Zach will present in December)?
Sorry for the late notice!

Sorry to hear that, Bruce! Hope you feel better soon! (Obviously, late notice is the only kind possible when it comes to being sick!)

There are plenty of interesting potential topics, still. Seems like there's a lot going on out there.

I'm going to propose two possible hack sessions, if people are game:

1. Work on the Python support for BERT-RPC: http://bert-rpc.org/

See this for background on why BERT exists:
http://github.com/blog/531-introducing-bert-and-bert-rpc

2. Start a Python port of Resque, or at least see about interfacing Resque with Python code:

http://github.com/blog/542-introducing-resque

(yes, two GitHub technologies... but they both seem like good ideas that are proving themselves in real world use!)

Actually, it may be possible to set up Resque to just fire off jobs via BERT-RPC, though part of the point of Resque is that it gives you visibility into what the queue is doing.

Anyone game for a hack session? I don't think I personally have a presentation on deck that does not involve JavaScript ;)

Kevin

--
Kevin Dangoor

work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com

[Curt Micol]

On Wed, Nov 4, 2009 at 4:01 PM, Kevin Dangoor <dan...@gmail.com> wrote:
> 2. Start a Python port of Resque, or at least see about interfacing Resque
> with Python code:
>
> http://github.com/blog/542-introducing-resque
>
> (yes, two GitHub technologies... but they both seem like good ideas that are
> proving themselves in real world use!)
>
> Actually, it may be possible to set up Resque to just fire off jobs via
> BERT-RPC, though part of the point of Resque is that it gives you visibility
> into what the queue is doing.

In case you missed it, defunkt posted this gist:
http://gist.github.com/225369

Might be helpful.

--
# Curt Micol

[Kevin Dangoor]

Heh. That's the easy part. The harder part, I think, is running Python jobs.

[Curt Micol]

On Wed, Nov 4, 2009 at 4:08 PM, Kevin Dangoor <dan...@gmail.com> wrote:
> Heh. That's the easy part. The harder part, I think, is running Python jobs.
>
> Kevin

Oh for sure, just wanted to pass that along for the hack session. Wish
I could make it out.

Good luck, I look forward to seeing what materializes.

--
# Curt Micol

[Kevin Dangoor]

On Wed, Nov 4, 2009 at 4:10 PM, Curt Micol <ase...@gmail.com> wrote:

On Wed, Nov 4, 2009 at 4:08 PM, Kevin Dangoor <dan...@gmail.com> wrote:
> Heh. That's the easy part. The harder part, I think, is running Python jobs.
>
> Kevin

Oh for sure, just wanted to pass that along for the hack session. Wish
I could make it out.

Definitely thanks for passing that along. I wouldn't have noticed it otherwise.

Are people interested in doing this? Anyone have a different topic they'd like to propose?

Kevin

[Mark Ramm]

Hmm,

Not that I object to BERT

[Kevin Dangoor]

On Wed, Nov 4, 2009 at 9:42 PM, Mark Ramm <mark.mch...@gmail.com> wrote:

Hmm,

Not that I object to BERT

Resque is actually the one that's more interesting to me, personally. Might be cool if there was a useful combination of a Pythonic Resque and Norc: http://blog.perpetually.com/post/230779975/innovating-cron-announcing-norc

[Mark Ramm]

> I'm going to propose two possible hack sessions, if people are game:
>
> 1. Work on the Python support for BERT-RPC: http://bert-rpc.org/

I wonder how much this gets you over BSON (as used by mongodb)?

Both allow binary data, have more data types than json, and have fast
serializes, but it looks like right now there are more BSON bindings
in more languages...

> 2. Start a Python port of Resque, or at least see about interfacing Resque
> with Python code:
>
> http://github.com/blog/542-introducing-resque

RabbitMQ seems to fill this need in some ways, and is clusterable, and
bus/based rather than polling based. I'm not exactly clear where it
didn't work for them, but it does have some limitations. It does not
have a shiny web interface to tell you how full the queue is, but you
can get that information off of the command line.... And it's not
going to provide quite as much data as resque does.

I am interested in the use of redis as the backend for a queue, and
i'm sure there are some cases where resque is more suitable, I just
haven't quite gotten it yet.

--Mark

[Kevin Dangoor]

On Wed, Nov 4, 2009 at 10:22 PM, Mark Ramm <mark.mch...@gmail.com> wrote:

> I'm going to propose two possible hack sessions, if people are game:
>
> 1. Work on the Python support for BERT-RPC: http://bert-rpc.org/

I wonder how much this gets you over BSON (as used by mongodb)?

Both allow binary data, have more data types than json, and have fast
serializes, but it looks like right now there are more BSON bindings
in more languages...

This is a valid point, assuming BSON has a "binary" field type of some sort (and I think the comments on the GitHub blog mentioned BSON as well...)
 

> 2. Start a Python port of Resque, or at least see about interfacing Resque
> with Python code:
>
> http://github.com/blog/542-introducing-resque

RabbitMQ seems to fill this need in some ways, and is clusterable, and
bus/based rather than polling based.    I'm not exactly clear where it
didn't work for them, but it does have some limitations. It does not
have a shiny web interface to tell you how full the queue is, but you
can get that information off of the command line....  And it's not
going to provide quite as much data as resque does.

From what I recall, RabbitMQ is more of a message queue and not so much of a work queue, though I do seem to remember you mentioning something about RabbitMQ being able to do work queue duty. Most message queues try to deliver a message reliably to every listener on a given "channel"... a work queue, like beanstalkd which is what I've been using, just parcels out the work one at a time.

I am interested in the use of redis as the backend for a queue, and
i'm sure there are some cases where resque is more suitable, I just
haven't quite gotten it yet.

Where Resque is really different is in a couple of things:

1. the data it provides (as you note via a fancy web interface)
2. management of the workers themselves  (Resque itself will fork the worker processes and make sure they don't get out of control)

BTW, these were just suggestions. I'm really open to anything.

[Mark Ramm]

I like these suggestions, just priming the discussion a bit ;)

--Mark

[Ask Solem]

> >> > 2. Start a Python port of Resque, or at least see about interfacing
> >> > Resque
> >> > with Python code:

I'm not in Michigan, but felt I had a need to respond to this.

There's already a work queue implementation for python called Celery:
http://ask.github.com/celery

It recently gained support for using Redis as the message backend as
well:
http://ask.github.com/celery/tutorials/otherqueues.html

It would be great if we didn't work in parallel on this, a lot of help
is wanted.

[Kevin Dangoor]

Hi,

On Fri, Nov 6, 2009 at 10:25 AM, Ask Solem <asks...@gmail.com> wrote:

> >> > 2. Start a Python port of Resque, or at least see about interfacing
> >> > Resque
> >> > with Python code:

I'm not in Michigan, but felt I had a need to respond to this.

There's already a work queue implementation for python called Celery:
http://ask.github.com/celery

It recently gained support for using Redis as the message backend as
well:
http://ask.github.com/celery/tutorials/otherqueues.html

This is very cool. After taking a bit more of a look at what Resque does, we dropped the idea entirely. It had nice features along certain axes, but along others was not so good.

Celery actually looks quite good. We were talking about how scheduling+worker queue makes good sense, and it's nice to see that this has been done!

I may look at replacing my current beanstalkd setup with Celery at some point in the future.

Kevin

[Ask Solem]

=

> This is very cool. After taking a bit more of a look at what Resque does, we
> dropped the idea entirely. It had nice features along certain axes, but
> along others was not so good.

It does have a cool web interface, something we're currently working
on.
But with the limitation of not being able to peek at the queue, we're
building it around
messages not direct data access. Being able to peek is a nice feature,
but so is being
able to not use pull, and having message reliability. The default
polling interval in Resque
is 5 seconds, that certainly excludes a lot of the use cases Celery is
already being used for.

>
> Celery actually looks quite good. We were talking about how
> scheduling+worker queue makes good sense, and it's nice to see that this has
> been done!

Indeed. I've rewritten the scheduling now, and it's much better. It no
longer requires
polling or locks. http://wiki.github.com/ask/celery/rewriting-the-periodic-task-service

>
> I may look at replacing my current beanstalkd setup with Celery at some
> point in the future.
>

Great! Do tell if you need any help.

I don't want to occupy your list with this, so if it sparks your
interest
feel free to join us at our IRC channel or the mailing-list mentioned
here: http://ask.github.com/celery/introduction.html#getting-help


--
Ask Solem