Calling external application

[iduy]

Hi All,
Is it possible for cascading to call external application or library?
For example I have an application written in C and I want to call it
from a cascading function. I'm thinking about writing a custom
function which will call an external library for every Tuple it
process. Is it possible?

Thanks!

[Ivan Brusic]

Nothing is stopping you from calling out to an external library via JNI provided that the shared object is installed on every machine in the cluster. However, the overhead for JNI is substantial. Considering the nature of MapReduce, processing every tuple in JNI, the overheads add up.

Then again, I haven't touched JNI in almost a decade, so perhaps things are better nowadays.

Ivan

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.423 / Virus Database: 270.14.25/2450 - Release Date: 10/22/09 08:51:00

[Nick Dimiduk]

I recommend JNI over using external program execution (commons-exec works, but is slooow). Thunking Objects across the language barrier is a little painful but passing simple types is quite fast. IIRC, most of the time in overhead is consumed with loading the library's symbols into the jvm's namespace - once the lib is loaded (should be done once/jvm), calling C methods is no slower than any other java method. Hadoop m/r is high latency anyway, so I doubt you'll notice much slow-down if you can keep your tight loops from crossing the boundary. As always, Your Milage May Vary.

[Dwinanda Prayudi]

Guys, many thanks for the answer.
I will look into using JNI.

> > 08:51:00- Hide quoted text -
>
> - Show quoted text -

[Dwinanda Prayudi]

Just want to share. Tried the JNI call and it works like a charm :)
One thing we have to do: put the native library (.so file) into
$HADOOP_HOME/lib/native/Linux-xxxx/lib on every data node.

> > - Show quoted text -- Hide quoted text -