Retrive information from multiple redis instance using python

[kmg]

I have written a python code to pull metrics from Multiple redis instance. As a starting point, I wrote a very simple code that will contain the list of redis instance hosts details. Using for condition, I connected to each redis instance and pull the metrics.

Now I tried to enhance the code using python multiprocessing.

Below is the code I tried to enable multiprocessing. But it shows error.

Can someone help suggest what is wrong with this also please guide how to solve this process. Any sample code will really helps me a lot.

#!/usr/bin/python
import time
import redis
import multiprocessing

redis_host = ['localhost','192.168.56.10']

def conn_redis(host):
  r = redis.Redis(host=host,port=6379)
  return r

def main(r):
  try:
    print r.info()
  except redis.ConnectionError,e:
    print e

if __name__ == '__main__':

  p = multiprocessing.Pool(multiprocessing.cpu_count()) 
  redis_conn = p.map(conn_redis, redis_host) 
  while(True):
    main(redis_conn)
    time.sleep(5)

Error message:

Traceback (most recent call last):
  File "redis-metrics-thread.py", line 21, in <module>
    redis_conn = p.map(conn_redis, redis_host) 
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[Redis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>]'. Reason: 'PicklingError("Can't pickle <type 'function'>: attribute lookup __builtin__.function failed",)'

[Nick Farrell]

Your problem is nothing to do with redis. You are trying to create connection instances on different processes then pass them back to the controlling process. No way this will work, and it makes no sense.

I really like the async redis client. This is way more than you want, but here's something you can look at: https://gist.github.com/nicois/4079dec03d33a24f236ecaaf18ffecf9