Scribe - HDFS Integration

[GK]

Hi,

I am trying to integrate Scribe with HDFS for storing the log
files. The Scribe version i'm using is 2.01 and Hadoop version is
0.20.1. There was no built-in HDFS support in Scribe 2.01. So now I am
trying with Scribe 2.1 which I got from github.com. I'm trying to
install it with the following command.

$ ./configure --with-hadooppath=/usr/local/hadoop --enable-hdfs

$ make

I am getting an errors in hdfs.c file located in hadoop/src/c++/
libhdfs

Is there any problem with the versions I am using??
Or any patches available?

Please help me out in the Scribe-HDFS connection.

[Anthony Giardullo]

What is the error you are running into?

-Anthony

[GK]

These is my error sequence.

HdfsFile.h:14:18: error: hdfs.h: No such file or directory
HdfsFile.h:14:40: error: hdfsFS does not name a type
HdfsFile.h:14:41: error: hdfsFile does not name a type
HdfsFile.h:14:42: error: hdfsFS does not name a type
make[3]: *** [file.o] Error 1
make[3]: Leaving directory `/root/gokul/scribe-2.1/src'
make[2]: *** [all] Error 2

So I included the full path of the hdfs.h (i.e., /root/app/hadoop/src/c
++/libhdfs/hdfs.h) in the file HdfsFile.h(located in /root/gokul/
scribe-2.1/src).

Then I got the following error.

HdfsFile.o: In function `HdfsFile::connectToPath(char const*)':
HdfsFile.cpp:(.text+0x166): undefined reference to `hdfsConnectAsUser`
followed by hdfsFlush , hdfsDisconnect, hdfsGetPathInfo ,
hdfsFreeFileInfo. etc..

And so I tried to build the libhdfs. So in /root/app/hadoop/src/c++/
libhdfs I tried the following.

$ sh ./configure
$ make
.....
.....
cc1: error: unrecognized command line option "-m"
make: *** [hdfs.lo] Error 1

If you know any other procedure for scribe-hdfs integration, please
suggest.

> > Please help me out in the Scribe-HDFS connection.- Hide quoted text -
>
> - Show quoted text -

[Anthony Giardullo]

How did you configure Scribe before building? Take a look at the
readme for manually specifying the path to hdfs.

Sorry this build process is a little obtuse right now. We are hoping
to get this cleaned up a bit.

-Anthony

[tsuna]

On Mon, Jan 11, 2010 at 9:08 PM, GK <gokulak...@gmail.com> wrote:
> These is my error sequence.
>
> HdfsFile.h:14:18: error: hdfs.h: No such file or directory

This error indicates that you're missing a -I flag to specify where
are the include files of libhdfs.
This comes from the fact that you passed
--with-hadooppath=/usr/local/hadoop but
/usr/local/hadoop/include/hdfs.h doesn't exist, probably because you
didn't install libhdfs under /usr/local/hadoop. If you don't want to
install libhdfs (you don't have to), then the alternative for you is
to call configure like this:

$ ./configure --enable-hdfs CPPFLAGS=-I/root/app/hadoop/src/c++/libhdfs

> HdfsFile.h:14:40: error: hdfsFS does not name a type
> HdfsFile.h:14:41: error: hdfsFile does not name a type
> HdfsFile.h:14:42: error: hdfsFS does not name a type
> make[3]: *** [file.o] Error 1
> make[3]: Leaving directory `/root/gokul/scribe-2.1/src'
> make[2]: *** [all] Error 2
>
> So I included the full path of the hdfs.h (i.e., /root/app/hadoop/src/c
> ++/libhdfs/hdfs.h) in the file HdfsFile.h(located in /root/gokul/
> scribe-2.1/src).
>
> Then I got the following error.
>
> HdfsFile.o: In function `HdfsFile::connectToPath(char const*)':
> HdfsFile.cpp:(.text+0x166): undefined reference to `hdfsConnectAsUser`
> followed by hdfsFlush , hdfsDisconnect, hdfsGetPathInfo ,
> hdfsFreeFileInfo. etc..

This is a linker error. It means the linker is unable to find the
libhdfs.a – because you didn't install it.
Once again, your quick solution would be to pass an extra variable to
configure, like so:

$ ./configure --enable-hdfs
CPPFLAGS=-I/root/app/hadoop/src/c++/libhdfs
LDFLAGS=-L/root/app/hadoop/src/c++/libhdfs/.libs

> And so I tried to build the libhdfs. So in /root/app/hadoop/src/c++/
> libhdfs I tried the following.
>
> $ sh ./configure
> $ make
> .....
> .....
> cc1: error: unrecognized command line option "-m"
> make: *** [hdfs.lo] Error 1

This is because the configure script of libhdfs doesn't tell you but
it needs to be given a JVM_ARCH variable.

Try to configure it like this:
./configure --enable-static --disable-shared
--with-java=path/to/your/java/hom JVM_ARCH=tune=k8
k8 will work on many modern CPUs, but you can replace it with i686 if
you want to remain more generic or with the architecture of your
choice (see man gcc for the possible values of the -mtune flag).

--
Benoit Sigoure aka Tsuna
Software Engineer @ www.StumbleUpon.com

[Gokul]

Thanks a lot Tsuna..

Thanks anthony..

--
Thanks and Regards,
Gokulakannan.M

[GK]

I am getting the following error after running the following command
sequences ..

./configure --enable-static --disable-shared
--with-java=$JAVA_HOME JVM_ARCH=tune=k8

configuration is successful. No errors or warnings..

but after running 'make' I am getting the folowing error

/bin/sh ./libtool --mode=link --tag=CC gcc -g -O2 -DOS_LINUX -
DDSO_DLFCN -DCPU=\"i386\" -mtune=k8 -I/opt/jdk6/include -I/opt-rpath /
root/app/hadoop-0.20.1/src/c++/libhdfs/../install/lib hdfs.lo
hdfsJniHelper.lo -ldl -lpthread
libtool: link: can not build a shared library
libtool: link: see the libtool documentation for more information
libtool: link: Fatal configuration error.
make: *** [libhdfs.la] Error 1

These are my -mtune options < v0 , v8 , v10 , k8 , i386 >
I tried all of the options but it results in the same error

On Jan 12, 3:08 pm, tsuna <tsuna...@gmail.com> wrote:

> Software Engineer @www.StumbleUpon.com- Hide quoted text -

[tsuna]

On Tue, Jan 12, 2010 at 10:30 PM, GK <gokulak...@gmail.com> wrote:
> but after running 'make' I am getting the folowing error
>
> /bin/sh ./libtool --mode=link --tag=CC gcc -g -O2 -DOS_LINUX -
> DDSO_DLFCN -DCPU=\"i386\" -mtune=k8 -I/opt/jdk6/include -I/opt-rpath /
> root/app/hadoop-0.20.1/src/c++/libhdfs/../install/lib hdfs.lo
> hdfsJniHelper.lo -ldl -lpthread

This command doesn't seem complete. Are you sure you copy/pasted it
in its entirety?

> libtool: link: can not build a shared library
> libtool: link: see the libtool documentation for more information
> libtool: link: Fatal configuration error.
> make: *** [libhdfs.la] Error 1

Which version of libtool are you using (what's the output of libtool --version)

> These are my -mtune options < v0 , v8 , v10 , k8 , i386 >
> I tried all of the options but it results in the same error

Yeah this setting doesn't have anything to do with this error.

[Gokul]

Yes this is the exact error message I got. 

libtool version is 1.5.24

[GK]

@tsuna
this was the complete error message I got. Actually I couldn't copy/
paste the error.
I should type it from the remote system. That's why some part was left
previously.

This is my full error message.

/bin/sh ./libtool --mode=link --tag=CC gcc -g -O2 -DOS_LINUX -

DDSO_DLFCN -DCPU=\"i386\" -mtune=k8 -I/opt/jdk1.6.0_14/include -I/opt/
jdk1.6.0_14/include/linux -Wall -Wstrict-prototypes -mtune=k8 -L/opt/
jdk1.6.0_14/jre/lib/server -ljvm -shared -Wl, -x -o -rpath /root/gokul/
hadoop/src/c++/libhdfs/../install/lib hdfs.lo hdfsJniHelper.lo -ldl -
lpthread

libtool: link: can not build a shared library
libtool: link: see the libtool documentation for more information
libtool: link: Fatal configuration error.
make: *** [libhdfs.la] Error 1

The libtool version is 1.5.22

On Jan 13, 9:32 pm, tsuna <tsuna...@gmail.com> wrote:

[IRFAN]

I got the following error while building the scribe available at
http://github.com/huyphan/Scribe-with-HDFS-support/tree/master

usr/lib/gcc/i586-suse-linux/4.1.2/../../../../i586-suse-linux/bin/ld:
cannot find -ljvm
collect2: ld returned 1 exit status

Appreciate any help on this.

[Anthony Giardullo]

Sounds like the linker cannot find jvm in your environment. See the
README about how to specify the extra CPPFLAGS/LDFLAGS to specify
where jvm is installed on your machine.

Also, we are working on getting this build process more streamlined.

-Anthony

[GK]

I am also trying to build the same..

I included --with-jvmpath while configuring and got resolved from that
-ljvm error.

but I am still getting an error saying:

"cannot find -lhdfs"

Before "make" I included libhdfs path in CPPFLAGS and LD_LIBRARY_PATH.
and also LDFLAGS="-Ljvm -Lhdfs" . Still the same error

> >>> Software Engineer @www.StumbleUpon.com-Hide quoted text -
>
> >> - Show quoted text -- Hide quoted text -

[Anthony Giardullo]

Hmmm... setting LD_LIBRARY_PATH to include the location of your hdfs
shared lib should work. Are you sure this directory contains
libhdfs.so?

Also, you could try configuring scribe with "--with-hadoop=/[path-to-
your-libhdfs-dir]" or passing the libhdfs.so directory using LDFLAGS="-
L/[path-to-your-libhdfs-dir]".

-Anthony

[Anthony Giardullo]

Oh, for your LDFAGS, I believe you want to use a lowercase l instead
of uppercase:

LDFLAGS="-ljvm -lhdfs"

-Anthony

[GK]

Thanks anthony..
it is working..

The LD_LIBRARY_PATH path was not set correctly..
The problem was not in LDFLAGS... this gcc(in SUSE) accepts options
only with -L

On Jan 20, 12:06 pm, Anthony Giardullo <agiardu...@facebook.com>
wrote:

> >>>>> Software Engineer @www.StumbleUpon.com-Hidequoted text -

[Chris Trezzo]

Hi All,

I seem to be having a similar linker problem when trying to build
Scribe with HDFS.

I am running the configure script with the following command:

./configure --prefix=/home/ctrezzo/usr/local --enable-hdfs --with-
thriftpath=/home/ctrezzo/usr/local --with-hadooppath=/home/ctrezzo/
hadoop-0.19.1 --with-fb303path=/home/ctrezzo/usr/local --with-jvmpath=/
home/ctrezzo/usr/java/jdk1.6.0_05 CPPFLAGS="-I/home/ctrezzo/usr/java/
jdk1.6.0_05/include -I/home/ctrezzo/usr/java/jdk1.6.0_05/include/linux
-I/home/ctrezzo/hadoop-0.19.1/libhdfs -I/home/ctrezzo/hadoop-0.19.1/
src/c++/libhdfs" LDFLAGS="-ljvm -lhdfs"

I have edited my LD_LIBRARY_PATH as follows:
LD_LIBRARY_PATH=$HOME/hadoop-0.19.1/libhdfs:$HOME/usr/java/
jdk1.6.0_05:$HOME/usr/java/jdk1.6.0_05/include/linux:$HOME/usr/java/
jdk1.6.0_05/jre/lib/
i386/server:$HOME/usr/java/jdk1.6.0_05/jre/lib/i386/client:
$LD_LIBRARY_PATH

When I run the configure command, I receive the following error inside
the config.log file:

configure:2333: checking for C compiler default output file name
configure:2336: gcc -I/home/ctrezzo/usr/java/jdk1.6.0_05/include -I/
home/ctrezzo/usr/java/jdk1.6.0_05/include/linux -I/home/ctrezzo/
hadoop-0.19.1/libhdfs
-I/home/ctrezzo/hadoop-0.19.1/src/c++/libhdfs -I/home/ctrezzo/usr/
java/jdk1.6.0_05/jre/lib/i386/client -ljvm -lhdfs conftest.c >&5
/usr/bin/ld: cannot find -ljvm

collect2: ld returned 1 exit status

configure:2339: $? = 1

The libhdfs.so file is in /home/ctrezzo/hadoop-0.19.1/libhdfs.
There is an libjvm.so file in /home/ctrezzo/usr/java/jdk1.6.0_05/jre/
lib/i386/client and /home/ctrezzo/usr/java/jdk1.6.0_05/jre/lib/i386/
server

Is there something I am missing?

Thanks in advance for the help.
Chris

> > >>>>> Software Engineer @www.StumbleUpon.com-Hidequotedtext -

[tsuna]

On Sat, Feb 20, 2010 at 8:31 PM, Chris Trezzo <ctr...@gmail.com> wrote:
> When I run the configure command, I receive the following error inside
> the config.log file:
>
> configure:2333: checking for C compiler default output file name
> configure:2336: gcc  -I/home/ctrezzo/usr/java/jdk1.6.0_05/include -I/
> home/ctrezzo/usr/java/jdk1.6.0_05/include/linux -I/home/ctrezzo/
> hadoop-0.19.1/libhdfs
>  -I/home/ctrezzo/hadoop-0.19.1/src/c++/libhdfs -I/home/ctrezzo/usr/
> java/jdk1.6.0_05/jre/lib/i386/client -ljvm -lhdfs conftest.c  >&5
> /usr/bin/ld: cannot find -ljvm
> collect2: ld returned 1 exit status
> configure:2339: $? = 1

ld can't find libjvm.so because you used a lower-case `-l' flag
instead of `-L' to specify the paths to the libraries used.

[Chris Trezzo]

Ah. Thanks!
Chris

[djy19...@gmail.com]

hello,It seems that you are familiar with scribe,I have accout some problems,

[Sun Jun 23 15:51:19 2013] "STATUS: STARTING" 
[Sun Jun 23 15:51:19 2013] "STATUS: configuring" 
[Sun Jun 23 15:51:19 2013] "got configuration data from file <examplesfs_examplefeng.conf>" 
[Sun Jun 23 15:51:19 2013] "CATEGORY : default" 
[Sun Jun 23 15:51:19 2013] "Creating default store" 
[Sun Jun 23 15:51:19 2013] "configured <1> stores" 
[Sun Jun 23 15:51:19 2013] "STATUS: " 

error as follows:

[Sun Jun 23 15:51:19 2013] "STATUS: ALIVE" 
[Sun Jun 23 15:51:19 2013] "Starting scribe server on port 1463" 
Thrift: Sun Jun 23 15:51:19 2013 libevent 2.0.12-stable method epoll
[Sun Jun 23 15:51:23 2013] "[test] Creating new category store from model default" 
[Sun Jun 23 15:51:23 2013] "store thread starting" 
[Sun Jun 23 15:51:23 2013] "[hdfs] Connecting to HDFS for hdfs://localhost:9000/scribedata/test/ndn-VirtualBox" 
[Sun Jun 23 15:51:23 2013] "[hdfs] Before hdfsConnectNewInstance(localhost, 9000)" 
There was an error trying to initialize the HPI library.
Call to JNI_CreateJavaVM failed with error: -1
[Sun Jun 23 15:51:23 2013] "[hdfs] After hdfsConnectNewInstance" 
[Sun Jun 23 15:51:23 2013] "[hdfs] ERROR: HDFS is not configured for file: hdfs://localhost:9000/scribedata/test/ndn-VirtualBox" 
[Sun Jun 23 15:51:23 2013] "[hdfs] Connecting to HDFS for hdfs://localhost:9000/scribedata/test/ndn-VirtualBox/test-2013-06-23_00000" 

I have tried to set JAVA_HOME CLASSPATH but it didn't work ,what shoud I do?

Please help me ,thank you!

在 2010年1月14日星期四UTC+8上午12时32分10秒，tsuna写道：

[Stig Bakken]

Looks like you need to set your LD_LIBRARY_PATH or configure ld.so.conf so your scribed executable can find libjvm.so.  On my system it is in /usr/lib/jvm/java-1.6.0/jre/lib/amd64/server.

 - Stig

--
 
---
You received this message because you are subscribed to the Google Groups "Scribe Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scribe-serve...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.