Need help bootstrapping pymongo

103 views
Skip to first unread message

VJ

unread,
Feb 26, 2013, 3:59:09 PM2/26/13
to mr...@googlegroups.com
Hello.  I have a series of EMR jobs (using mrjob) that read and write from S3.  The last reduce step needs to write to Mongo.  Since the default EMR server instance does not seem to have pymongo (??), I'm trying to bootstrap a pymongo install in mrjob.conf (below).  The script runs fine at the command line (without -r emr).  Also, the script runs fine if the last reduce step writes to S3 (instead of Mongo).  Before I tried the bootstrapping, I got the error "ImportError: No module named pymongo"

Any advice would be greatly appreciated.  Thanks in advance.

Runtime errors
===========
No handlers could be found for logger "mrjob.emr"
Traceback (most recent call last):
  File "./loc_scr_gen.py", line 121, in <module>
    main()
  File "./loc_scr_gen.py", line 67, in main
    runner.run()
  File "/usr/lib/python2.6/site-packages/mrjob-0.4_dev-py2.6.egg/mrjob/runner.py", line 449, in run
    self._run()
  File "/usr/lib/python2.6/site-packages/mrjob-0.4_dev-py2.6.egg/mrjob/emr.py", line 840, in _run
    self._wait_for_job_to_complete()
  File "/usr/lib/python2.6/site-packages/mrjob-0.4_dev-py2.6.egg/mrjob/emr.py", line 1577, in _wait_for_job_to_complete
    raise Exception(msg)
Exception: Job on job flow j-3370FHLVR5WCR failed with status SHUTTING_DOWN: Shut down as step failed

mrjob.conf
========
runners:
  emr:
    aws_access_key_id: XXXXX
    aws_secret_access_key: XXXXX
    aws_region: us-east-1
#    ec2_instance_type: m1.xlarge
#    num_ec2_instances: 4
    s3_log_uri: s3://dsps-log/
    s3_scratch_uri: s3://dsps-tmp/
    base_tmp_dir: /data/emr_scratch
    ### setup cmds
    bootstrap_cmds:
    - sudo apt-get install -y python-pymongo
#    - sudo pip install pymongo
    ### Setup python packasges ###
#    bootstrap_python_packages:
#    - s3://emr-pypackages/pymongo.2.4.2.tar.gz
=====
Reply all
Reply to author
Forward
0 new messages