CherryPy and connection pooling with sdb

[Grashopa]

Does boto do any connection pooling behind the scenes? ie can I call
this on every thread without wasting resources?:

sdb = boto.connect_sdb(...)
dom = sdb.get_domain('foo')

I assume I can't share a connection between multiple threads. But I
could create a connection and then 'release' it allowing that
connection to be returned to another thread right?

Can I 'pool' the domain object per connection as well so the thread
doesn't have to call get_domain each time?

[Alan Pound]

Does boto do any connection pooling behind the scenes? ie can I call
this on every thread without wasting resources?:

>>> No

sdb = boto.connect_sdb(...)
dom = sdb.get_domain('foo')

I assume I can't share a connection between multiple threads. But I
could create a connection and then 'release' it allowing that
connection to be returned to another thread right?

>>> Yes

Can I 'pool' the domain object per connection as well so the thread
doesn't have to call get_domain each time?

>>> Yes. I have a little tuple of (domainname, region, sdb, dom) that I can fetch and return to a shared pooling class for it to look after.

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To post to this group, send email to boto-...@googlegroups.com.
To unsubscribe from this group, send email to boto-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/boto-users?hl=en.

[Grashopa]

> Can I 'pool' the domain object per connection as well so the thread
> doesn't have to call get_domain each time?
>
> >>> Yes.  I have a little tuple of (domainname, region, sdb, dom) that I can fetch and return to a shared pooling class for it to look after.

Any reason we can't add a class like this to boto?

get_domain is the expensive operation - How many domains can you
'get_domain' for each connection? Any number? Downsides to keeping a
connection available with with multiple domain objects?

Can you retrieve the list of open domains from the connection object?
I'm currently keeping that list myself - I retrieve the first open
connection and if the requested domain isn't in it I do a get_domain
then return the connect and domain object.

Mark

[Alan Pound]

-----Original Message-----
From: boto-...@googlegroups.com on behalf of Grashopa
Sent: Thu 03/05/2012 19:02
To: boto-users
Subject: [boto-users] Re: CherryPy and connection pooling with sdb

> Can I 'pool' the domain object per connection as well so the thread
> doesn't have to call get_domain each time?
>
> >>> Yes.  I have a little tuple of (domainname, region, sdb, dom) that I can fetch and return to a shared pooling class for it to look after.

Any reason we can't add a class like this to boto?

>>> Probably not, but boto tends to give you fairly low-level tools that you build upon, and it isn't too difficult to build a pooling class to exactly suit your own usage.

get_domain is the expensive operation - How many domains can you
'get_domain' for each connection? Any number? Downsides to keeping a
connection available with with multiple domain objects?

>>> I'm pretty sure you must stick to one domain per connection.

Can you retrieve the list of open domains from the connection object?

>>> Not that I am aware. I got bitten when I started using boto - you can't open multiple domains on one connection then interleave accesses - it isn't thread-safe.

I'm currently keeping that list myself - I retrieve the first open
connection and if the requested domain isn't in it I do a get_domain
then return the connect and domain object.

>>> I do a loop of i) make connection, ii) get a domain, iii) store it in pool indexed by name (keeping those things as a tuple), then when I need one, get it, use it, put it back. If you don't have enough in the pool, then just make another one until you do...

Alan

[Grashopa]

What do you do for connections that time out?

It may also be possible to open too many connections as well. We
should probably add this to boto since everyone needs it and theres no
reason to go through these things more than once.

Mark

[Alan Pound]

Grashopa <markr...@gmail.com> wrote:
What do you do for connections that time out?

>>> You try it again.

It may also be possible to open too many connections as well.  

>>> Not really - it is the overall write/delete rate that seems to matter most.

We should probably add this to boto since everyone needs it and theres no
reason to go through these things more than once.

>>> Im certain that Mitch will accept any sensible contributions.

Mark

[Grashopa]

On May 4, 3:32 pm, Grashopa <markree...@gmail.com> wrote:
> What do you do for connections that time out?

This actually should be handled by boto anyway. Here is the error
message I see if I reuse a connection after an hour. Perhaps this
isn't some sort of timeout?

<Response><Errors><Error><Code>SignatureDoesNotMatch</
Code><Message>The request signature we calculated does not match the
signature you provided. Check your AWS Secret Access Key and signing
method. Consult the service documentation for details.</Message></
Error></Errors><RequestID>b7f137ff-7244-acdf-f6a3-7d7abb62a1f1</
RequestID></Response>

[Grashopa]

Ignore this last - the error is coming from not urlencoding first.

Mark

[Grashopa]

When can the connection be released back to the pool? ( Another reason
to put this into boto )

user = dom.get_item(email)

Can we now release the connection or does the following continue to
use the connection.

key = user['key']

[Alan Pound]

Yes, as far as I am aware, it continues to use the connection.

If you have a long-running thread that wants to keep doing sdb operations, there is no real reason to use a pool - just create the connection and use it.

If you want to use the connection in connection with, for example a web request where the lifetime of the thread is finite, then keep the connection until the thread is about to terminate, then re-pool it.  If the thread is due to get re-used, and the thread is *always* going to need a connection, then the thread may as well own the connection (if you can find somewhere useful to stash it).

There are usually plenty of ways of skinning each particular cat...

Alan

-----Original Message-----
From: boto-...@googlegroups.com on behalf of Grashopa
Sent: Sun 13/05/2012 18:11
To: boto-users
Subject: [boto-users] Re: CherryPy and connection pooling with sdb

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To post to this group, send email to boto-...@googlegroups.com.
To unsubscribe from this group, send email to boto-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/boto-users?hl=en.

Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

P Please consider the environment and don't print this e-mail unless you really need to