Fuse fabric for hadoop+osgi

[somya singhal]

Hello

I have learnt to run hadoop applications as well as osgi applications on apache karaf.But now i have to integrate both hadoop and osgi.How can i use fuse fabric to integrate both so that i can deploy hadoop bundle in osfi container?

Somya Singhal
Btech(4th year,CSI)
IIT ROORKEE

[James Strachan]

If you follow the getting started guide:
http://fuse.fusesource.org/fabric/docs/getting-started.html

and try provision a hadoop or hadoop-datanode then that should be
deploying hadoop inside OSGi. Is that what you meant?

--
James
-------
FuseSource
Email: ja...@fusesource.com
Web: http://fusesource.com
Twitter: jstrachan, fusenews
Blog: http://macstrac.blogspot.com/

FuseSource: integration everywhere

[somya singhal]

The problem is that i have hadoop wordcount bundle with me.Now i am not getting the fact that how to deploy it in fuse fabric?That is what commands i should use to deploy it inside osgi container??

[Claus Ibsen]

Hi

To deploy a bundle in the OSGi container you can use the osgi:install command.
http://karaf.apache.org/manual/latest-2.2.x/commands/osgi-install.html

There is a lengthy guide on installing and deploying to the OSGi
container here as well
http://fusesource.com/docs/esbent/7.0/esb_deploy_osgi/front.html

But I guess you may use fabric and its profiles. Then you need to add
your bundle to the profile
you use for your containers. Then fabric will automatic provision this for you.
http://fuse.fusesource.org/fabric/docs/fabric-profiles.html

--
Claus Ibsen
-----------------
FuseSource
Email: cib...@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

[Guillaume Nodet]

If you have a local build of fabric, you can do the following steps:

  * modify the etc/system.properties to add the following

      hadoop.log.dir=${karaf.base}/data/hadoop/log/

  * (re-)start karaf

  * install hadoop by running the following commands in the karaf console:

  > osgi:install -s mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.commons-httpclient/3.1_7

  > features:install fabric-hadoop

  * configure hadoop by creating a file in etc/org.fusesource.fabric.hadoop.cfg with the following content:

fs.default.name=hdfs\://localhost\:9000

dfs.replication=1

mapred.job.tracker=localhost\:9001

dfs.name.dir=${karaf.data}/hadoop/dfs/name

dfs.http.address=localhost\:9002

dfs.data.dir=${karaf.data}/hadoop/dfs/data

dfs.name.edits.dir=${karaf.data}/hadoop/dfs/name

nameNode=true

dataNode=true

jobTracker=true

taskTracker=true

After those steps, you should have the hdfs web server available at

   http://localhost:9002/dfshealth.jsp

and the job tracker web server at

   http://localhost:50030/jobtracker.jsp

From this point, to deploy a job, you need to repackage your WordCount sample in an osgi bundle and modify it so that in a bundle activator (or using blueprint or any other similar technology), you create your job using the right configuration (the needed bits from the file above) and start it.

The hadoop support is still a bit work in progress and in particular, I have focused on hdfs rather than mapreduce, but I hope to simplify those steps asap.

--
------------------------
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
FuseSource, Integration everywhere
http://fusesource.com

[sudhee...@gmail.com]

Thanks,it's nice questioon and 123Trainings is a Global Interactive Learning company started by proven industry experts with an aim to provide Quality Training in the latest IT Technologies.

123Trainings is offering Corporate online Training services to Major IT giants and to individual students worldwide.

123Trainings has a pool of Expert Trainers worldwide on all the technologies to train the students.

[serg.s...@gmail.com]

Hi,

So does that mean that running hadoop map-reduce jobs is possible with current fuse fabric? or with fabric8, specifically.