hadoop eclipse plugin
Dave Bayer
supposed to do (especially if you connected it to the local
vmware image and saw a whole lot of nothing interesting),
the ibm site:
http://www.alphaworks.ibm.com/tech/mapreducetools
pointed me to the map reduce cheat sheet in eclipse, which
was almost very helpful. I thought the plugin would allow
me to find the hadoop dev area on the local node, but this
areas to not be the case. Do we need to download the apache
hadoop release for the single node? If so, will we need a
separate one for the ibm cluster?
Is there some abbreviated information out there on what the
plugin is doing and thus understand what else I need to do
to development and run a job without it?
dave bayer
Dave Bayer
> area on the local node, but this areas to not be the case. Do we need to
> download the apache hadoop release for the single node? If so, will we
> need a separate one for the ibm cluster?
Ok, so you do need to download the hadoop dev environment.
Now I have gone through the steps in the cheat sheet, except
that I cannot see the Run The MapReduce Application step due
to 'not having completed the prior steps' - which I had to do
manually because the plugin had some problems. Is there a way
to get around this?
dave bayer
Dave Bayer
> Now I have gone through the steps in the cheat sheet, except that I
> cannot see the Run The MapReduce Application step due to 'not having
> completed the prior steps' - which I had to do manually because the
> plugin had some problems. Is there a way to get around this?
I suppose I should give the error too:
The action could not be performed because plugin
'com.ibm.hipods.mapreduce' could not be located.
Google is of no help for this error.
dave bayer
barret rhoden
i spent a chunk of time messing with this too. in general, i found
that the cheat sheets aren't worth your time.
overall, there are two plugins out there. the hadoop one we are given
and the IBM map-reduce one. you will want the hadoop one (they are
very similar), and to switch into the map-reduce perspective and open a
map-reduce view. create a couple servers (one for the cloud and one
for your local machine).
from what i gather, the hadoop plugin provides some nice tools so you
can manage your hadoop jobs. you can browse your server's HDFS
(hadoop distributed file system) from the package explorer. you can
run your hadoop jobs directly from eclipse on your servers. it also
provides templates for common mapreduce classes (mapper, reducer,
driver) and allows you to make map-reduce projects.
if you are working from home, you will want to download and extract a
version of hadoop (i'm using 0.16.3 with no issues so far), so that you
can make mapreduce projects (it will want to point to the install
directory of hadoop).
there's another gotcha, at least for me. eclipse 3.3.1.1 does not work
well with the map-reduce plugin (either the apache hadoop one or the
IBM one), in that it does not allow you to tunnel a connection when
setting up a MR server. using eclipse 3.2 seems to work fine with
everything.
finally, regarding the cheat sheets - they aren't that great. or at
least i didn't like them. they were somewhat informative, but no more
than any website (the hadoop map-reduce tutorial is nice
(http://hadoop.apache.org/core/docs/r0.16.3/mapred_tutorial.html)).
the only nice thing about those cheat sheets is that they can generate
templates for you for Map, Reduce, and Driver classes. however, just
go to File->New->Whatever and you can get the ones provided by the
hadoop plugin.
the error you are getting is because the cheat sheets were made to work
with the IBM plugin. if you really want to use that plugin, then i can
provide it for you. but don't bother. i downloaded, jarred, and used
the thing, and it wasn't worth the time. plus there are other issues
with it. it doesn't let you edit MR servers, had some weird behavior
when connecting to the local VM HDFS, and a couple other minor things.
once i realized i didn't need the cheatsheet templates, i switched back
to the regular hadoop plugin. so my recommendation is to skip the IBM
plugin. if you really want it, email me.
just a disclaimer - i'm learning all of this too, so sorry if any of it
is wrong. i'm about a half step ahead, and will try to save you all
the headaches i've already gone through with this.
barret
Chunwei (Steven) Lai
just reference it from Eclipse without downloading the full hadoop.
Steven
wendylw...@gmail.com
Our HADOOP online training program is job oriented. After completion of HADOOP online training with us you should be able to work on any kind of project. After completion of HADOOP online training our dedicated team will be supporting you.
Please contact us for HADOOP online training Demo in our rstrainings.com is the bes HADOOP online training Institute in Hyderabad, India.
HADOOP Online Training Course Content :
1.INTRODUCTION
• What is Hadoop?
• History of Hadoop
• Building Blocks - Hadoop Eco-System
• Who is behind Hadoop?
• What Hadoop is good for and what it is not
2.HDFS
• Configuring HDFS
• Interacting With HDFS
• HDFS Permissions and Security
• Additional HDFS Tasks
• HDFS Overview and Architecture
• HDFS Installation
• Hadoop File System Shell
• File System Java API
3.MAPREDUCE
• Map/Reduce Overview and Architecture
• Installation
• Developing Map/Red Jobs
• Input and Output Formats
• Job Configuration
• Job Submission
• Practicing Map Reduce Programs (atleast 10 Map Reduce Algorithms )
4.Getting Started With Eclipse IDE
• Configuring Hadoop API on Eclipse IDE
• Connecting Eclipse IDE to HDFS
5.Hadoop Streaming
6.Advanced MapReduce Features
• Custom Data Types
• Input Formats
• Output Formats
• Partitioning Data
• Reporting Custom Metrics
• Distributing Auxiliary Job Data
7.Distributing Debug Scripts
8.Using Yahoo Web Services
9.Pig
• Pig Overview
• Installation
• Pig Latin
• Pig with HDFS
10. Hive
• Hive Overview
• Installation
• Hive QL
• Hive Unstructured Data Analyzation
• Hive Semistructured Data Analyzation
11.HBase
• HBase Overview and Architecture
• HBase Installation
• HBase Shell
• CRUD operations
• Scanning and Batching
• Filters
• HBase Key Design
12.ZooKeeper
• Zoo Keeper Overview
• Installation
• Server Mantainace
13.Sqoop
• Sqoop Overview
• Installation
• Imports and Exports
14.CONFIGURATION
• Basic Setup
• Important Directories
• Selecting Machines
• Cluster Configurations
• Small Clusters: 2-10 Nodes
• Medium Clusters: 10-40 Nodes
• Large Clusters: Multiple Racks
15.Integrations
16.Putting it all together
• Distributed installations
• Best Practices
http://www.rstrainings.com/hadoop-online-training.html
sudhee...@gmail.com
renuka...@gmail.com
anitha...@gmail.com
I was really impressed by your article,hadooponlinetrainings.com provides best hadoop online training from hyderabad.we have experienced faculty from all around India regarding hadoop online trainings.
anitha...@gmail.com
renuka...@gmail.com
anitha...@gmail.com
hi you have gathered a valuable information on hadoop...., I am looking for content like this and i am much impressed with the information and nice course content, thanks a lot for the Information regarding hadoop Online Training.
anitha...@gmail.com
It was very nice that u have very much interested regarding HADOOP .If u have more doubts better to take HADOOP ONLINE TRAINING
sudhee...@gmail.com
This is nice information.hadoop online trainings also provides<a href="http://hadooponlinetrainings.com/hadoop-online-training/">hadoop online training</a>
sudhee...@gmail.com
sudhee...@gmail.com
sudheer...@gmail.com
123Trainings is offering Corporate online Training services to Major IT giants and to individual students worldwide.
http://123trainings.com/it-hadoop-bigdata-online-training.html