How to make gems available for requiring inside a script

[Álvaro Martín Fraguas]

Greetings to all,

I have been trying to work with Hadoop Streaming without and with
Wukong. In both cases I have not found an easy way to make Ruby gems
available for requiring inside a mapper script.

One option is to put the files of all the lib directories of the
source of all the gems as zip archives in the HDFS. But that is error
prone and has problems with complex dependencies.

Any idea on how to solve this easily? It would be nice that Wukong
handled this gracefully, maybe reading a Gemfile from bundler and
uploading the proper files to the HDFS. What do you think? It may be
that I am just not using Hadoop properly and I could avoid these
problems, please tell me if that is the case :-)

Apart from that, it would be nice to fix the path of the Hadoop
streaming jar file in the Hadoop version 0.21, see this pull request
for details and minor testing: https://github.com/mrflip/wukong/pull/4

Thanks a lot :-)

Álvaro Martín Fraguas

[Álvaro Martín Fraguas]

I have ended up adding the corresponding lines to hadoop-env.sh in
order to load RVM into the session (which is a necessary step to
install RVM normally in a local environment) and use the installed
gems from every machine of the cluster:

[[ -s "/home/hadoop/.rvm/scripts/rvm" ]] && source "/home/hadoop/.rvm/
scripts/rvm" # This loads RVM into a shell session.

I still think it would be good to have Wukong automatically send the
libraries from the gems from a bundler Gemfile as archives in the HDFS
to avoid having to install and maintain all the gems in every machine
of the cluster. Maybe I can find the time to help with that in the
future...

Álvaro Martín Fraguas

On 26 abr, 12:41, Álvaro Martín Fraguas <amartinfrag...@gmail.com>
wrote: