Boto service region issue (SonOfMMM)

[nikola]

Hello,

I am trying to setup an AMI to run boto's "sonofmmm" service. I have managed to do the following so far:

* Created input and output queues, buckets and sdb domain,

* Launched a new small instance of https://console.aws.amazon.com/ec2/home?region=eu-west-1#launchAmi=ami-1de8d369 AMI,

* Installed ffmpeg,

* Git cloned latest (2.3.0) boto source and installed it,

* Put the following into rc.local:

    sudo -b -i -u <username> /usr/bin/python /home/<username>/boto/boto/pyami/bootstrap.py
    sudo -b -i -u <username> /usr/bin/python /home/<username>/boto/boto/pyami/startup.py

* Bundled, uploaded and registered new AMI

I have the attached my "sonofmmm.cfg" to this post.

The problem I am experiencing is when I use the boto service "bs.py" script.

When I start a new instance like this:

    $ cd /home/<username>/boto/boto/services
    $ ./bs.py -k <my_pk> sonofmmm.cfg start

once the new instance is launched I get the following error in syslog:

Traceback (most recent call last):
  File "/home/<username>/boto/boto/pyami/startup.py", line 60, in <module>
    su.main()
  File "/home/<username>/boto/boto/pyami/startup.py", line 52, in main
    self.run_scripts()
  File "/home/<username>/boto/boto/pyami/startup.py", line 49, in run_scripts
    raise e
AttributeError: 'NoneType' object has no attribute 'read'

I have also logged in via ssh to this instance and looked at boto's log files, look at the boto source code and it seems that this issue has to do with the default region that is set for the input queue (sqs). It is set to "us-east-1" and I can't seem to find a proper way to set region for aws services to "eu-west-1".

Setting "ec2_region_name" and "ec2_region_endpoint" in the service config file doesn't seem to help.

I have also tried the following:

    $ cd /home/<username>/boto/boto/services
    $ ./bs.py -p ~/<dir_containing_movie_files> sonofmmm.cfg submit

It informs me that all the files were submitted and prints the "Batch Identifier". I can confirm that these files are stored in the input bucket but no new messages are created in the input queue.

Any help would be appreciated.

Kind Regards,

Nikola

[Gregory Taylor]

This may or may not be the "right" answer (in reality, it's a shameless plug), but I could suggest checking out media-nommer if you can't figure sonofmm out:

http://media-nommer.readthedocs.org/

Just another option, don't mind my plugging.

Greg

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/boto-users/-/beafXGkBFbcJ.
To post to this group, send email to boto-...@googlegroups.com.
To unsubscribe from this group, send email to boto-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/boto-users?hl=en.

[nikola]

Hello Greg, 

thanks for your quick reply. 

I am looking for a way to create a distributed application in the AWS cloud that could launch worker instances that will take input A and process it and produce output B, in other words "get the job(s) done". My architecture should be similar to the on this image. I chose boto and "sonofmmm" service as a starting point because I found that it provides exactly what I am looking for. My idea was to get this running and then write my own "sonofmmm"-like service to perform custom jobs. It actually doesn't have to do anything with converting video files. 

Nevertheless, I am going to look at the media-nommer as it seems very interesting.

Thanks,

Nikola

On Tuesday, May 8, 2012 10:43:16 PM UTC+2, Gregory Taylor wrote:

This may or may not be the "right" answer (in reality, it's a shameless plug), but I could suggest checking out media-nommer if you can't figure sonofmm out:

http://media-nommer.readthedocs.org/

Just another option, don't mind my plugging.

Greg

To unsubscribe from this group, send email to boto-users+unsubscribe@googlegroups.com.

[Mitchell Garnaat]

I think the basic pattern embodied in SonOfMMM is still extremely useful but the code is old.  I think we should either agree there are better, more modern alternatives like media-nommer and try to point people who read the SonOfMMM article at those alternatives or spend the time to update the SonOfMMM ami and code.  If the best answer is media-nommer (entirely possible) then perhaps we could even get a new article published that focuses on solving the same basic problem as the SonOfMMM article using media-nommer.

To view this discussion on the web visit https://groups.google.com/d/msg/boto-users/-/LIuYmV1yakoJ.

To post to this group, send email to boto-...@googlegroups.com.

To unsubscribe from this group, send email to boto-users+...@googlegroups.com.

[nikola]

Hello,

I did some quick googling and found this article where I read that I could accomplish my goal by writing a custom "Nommer". I guess your reply just verified this. It would be of great help to have some sort of tutorial-like article.

I spent a lot of time on getting to know and setting up SonOfMMM. I am wondering if it would be more feasible to switch to media-nommer or to pursue with the former one?

Thanks,

Nikola

On Tuesday, May 8, 2012 11:21:24 PM UTC+2, Mitchell Garnaat wrote:

I think the basic pattern embodied in SonOfMMM is still extremely useful but the code is old.  I think we should either agree there are better, more modern alternatives like media-nommer and try to point people who read the SonOfMMM article at those alternatives or spend the time to update the SonOfMMM ami and code.  If the best answer is media-nommer (entirely possible) then perhaps we could even get a new article published that focuses on solving the same basic problem as the SonOfMMM article using media-nommer.

[Gregory Taylor]

On Tue, May 8, 2012 at 5:54 PM, nikola <nik...@imadethis.no> wrote:

I did some quick googling and found this article where I read that I could accomplish my goal by writing a custom "Nommer". I guess your reply just verified this. It would be of great help to have some sort of tutorial-like article.

I do need to write something like that. I'll try to get something figured out in the near future.

 

I spent a lot of time on getting to know and setting up SonOfMMM. I am wondering if it would be more feasible to switch to media-nommer or to pursue with the former one?

I'm not sure there. As Mitch has mentioned, it is pretty old, and doesn't see a lot of attention. However, if you're already familiar with it, and can get it working, it may not be worth getting to learn yet another project. I don't want to direct you the wrong way, there. You definitely would need to write a Nommer for whatever you're doing. While it's not very difficult, it is another step.

If you do end up having more questions about media-nommer, shoot me an email at gta...@gc-taylor.com so I won't annoy boto-users. Otherwise, keep on keeping on with SonOfMMM!

Greg

[nikola]

Thank you very much. I will take a deeper look at media-nommer and send you an email once I have questions.

Kind Regards,

Nikola

On Wednesday, May 9, 2012 12:01:08 AM UTC+2, Gregory Taylor wrote: