HDFS support feedback

[sebest]

Hello,

I compiled scribe with hdfs support using cloudera packages hadoop
0.20.1 on ubuntu 9.10

First i had issues with some ClassNotFoundException, even if the jar
were in the classpath it could find it, i had to unzip them (hadoop-
core hadoop-hdfs and common-logging)

then i have this error message:

[Thu Nov 19 18:29:58 2009] "setrlimit error (setting max fd size)"
[Thu Nov 19 18:29:58 2009] "STATUS: STARTING"
[Thu Nov 19 18:29:58 2009] "STATUS: configuring"
[Thu Nov 19 18:29:58 2009] "got configuration data from file
<hdfs_example.conf>"
[Thu Nov 19 18:29:58 2009] "CATEGORY : default"
[Thu Nov 19 18:29:58 2009] "Creating default store"
[Thu Nov 19 18:29:58 2009] "configured <1> stores"
[Thu Nov 19 18:29:58 2009] "STATUS: "
[Thu Nov 19 18:29:58 2009] "STATUS: ALIVE"
[Thu Nov 19 18:29:58 2009] "Starting scribe server on port 1463"
Thrift: Thu Nov 19 18:29:58 2009 libevent 1.4.11-stable method epoll
[Thu Nov 19 18:29:59 2009] "[categ1] Creating new category from model
default"
[Thu Nov 19 18:29:59 2009] "store thread starting"
[Thu Nov 19 18:29:59 2009] "[hdfs] Connecting to HDFS"
*** glibc detected *** ./scribed: munmap_chunk(): invalid pointer:
0x0000000001ea19c3 ***
======= Backtrace: =========
/lib/libc.so.6[0x7f8859a37dd6]
./scribed[0x42f6a8]
./scribed[0x42fdc4]
./scribed[0x42ce19]
./scribed[0x415fba]
./scribed[0x4182e7]
./scribed[0x40f5c3]
./scribed[0x4238bc]
./scribed[0x4241e9]
/lib/libpthread.so.0[0x7f885a4e2a04]
/lib/libc.so.6(clone+0x6d)[0x7f8859aa17bd]
======= Memory map: ========
00400000-00473000 r-xp 00000000 08:01
1028671 /home/sebest/Work/hadoop/scribe/src/
scribed
00673000-00674000 r--p 00073000 08:01
1028671 /home/sebest/Work/hadoop/scribe/src/
scribed
00674000-00675000 rw-p 00074000 08:01
1028671 /home/sebest/Work/hadoop/scribe/src/
scribed
00675000-00676000 rw-p 00000000 00:00 0
01ea0000-02355000 rw-p 00000000 00:00
0 [heap]
7f8820000000-7f8820049000 rw-p 00000000 00:00 0
7f8820049000-7f8824000000 ---p 00000000 00:00 0
7f8828000000-7f88280fa000 rw-p 00000000 00:00 0
7f88280fa000-7f882c000000 ---p 00000000 00:00 0
7f882eb4c000-7f882eb53000 r-xp 00000000 08:01
273525 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libnio.so
7f882eb53000-7f882ec52000 ---p 00007000 08:01
273525 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libnio.so
7f882ec52000-7f882ec54000 rw-p 00006000 08:01
273525 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libnio.so
7f882ec54000-7f882ec59000 r-xp 00000000 08:01
599 /lib/libnss_dns-2.10.1.so
7f882ec59000-7f882ee58000 ---p 00005000 08:01
599 /lib/libnss_dns-2.10.1.so
7f882ee58000-7f882ee59000 r--p 00004000 08:01
599 /lib/libnss_dns-2.10.1.so
7f882ee59000-7f882ee5a000 rw-p 00005000 08:01
599 /lib/libnss_dns-2.10.1.so
7f882ee5a000-7f882ee5c000 r-xp 00000000 08:01
607 /lib/libnss_mdns4_minimal.so.2
7f882ee5c000-7f882f05b000 ---p 00002000 08:01
607 /lib/libnss_mdns4_minimal.so.2
7f882f05b000-7f882f05c000 r--p 00001000 08:01
607 /lib/libnss_mdns4_minimal.so.2
7f882f05c000-7f882f05d000 rw-p 00002000 08:01
607 /lib/libnss_mdns4_minimal.so.2
7f882f05d000-7f882f070000 r-xp 00000000 08:01
273524 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libnet.so
7f882f070000-7f882f171000 ---p 00013000 08:01
273524 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libnet.so
7f882f171000-7f882f174000 rw-p 00014000 08:01
273524 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libnet.so
7f882f174000-7f882f17a000 r-xp 00000000 08:01
273565 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libmanagement.so
7f882f17a000-7f882f279000 ---p 00006000 08:01
273565 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libmanagement.so
7f882f279000-7f882f27b000 rw-p 00005000 08:01
273565 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libmanagement.so
7f882f27b000-7f882f27e000 ---p 00000000 00:00 0
7f882f27e000-7f882f37c000 rwxp 00000000 00:00 0
7f882f37c000-7f882f37d000 ---p 00000000 00:00 0
7f882f37d000-7f882f47d000 rwxp 00000000 00:00 0
7f882f47d000-7f882f480000 ---p 00000000 00:00 0
7f882f480000-7f882f57e000 rwxp 00000000 00:00 0
7f882f57e000-7f882f581000 ---p 00000000 00:00 0
7f882f581000-7f882f67f000 rwxp 00000000 00:00 0
7f882f67f000-7f882f682000 ---p 00000000 00:00 0
7f882f682000-7f882f780000 rwxp 00000000 00:00 0
7f882f780000-7f882f783000 ---p 00000000 00:00 0
7f882f783000-7f882f881000 rwxp 00000000 00:00 0
7f882f881000-7f882f888000 r--s 00000000 08:01
394566 /usr/lib/gconv/gconv-modules.cache
7f882f888000-7f882f8c7000 r--p 00000000 08:01
918358 /usr/lib/locale/en_US.utf8/LC_CTYPE
7f882f8c7000-7f882f8ca000 ---p 00000000 00:00 0
7f882f8ca000-7f882f9c8000 rwxp 00000000 00:00 0
7f882f9c8000-7f882f9cb000 ---p 00000000 00:00 0
7f882f9cb000-7f882fac9000 rwxp 00000000 00:00 0
7f882fac9000-7f882faca000 ---p 00000000 00:00 0
7f882faca000-7f882fbca000 rwxp 00000000 00:00 0
7f882fbca000-7f882fc3f000 rw-p 00000000 00:00 0
7f882fc3f000-7f882fdd5000 r--s 02fb3000 08:01
273520 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
rt.jar
7f882fdd5000-7f882fdfd000 rw-p 00000000 00:00 0
7f882fdfd000-7f882fdfe000 ---p 00000000 00:00 0
7f882fdfe000-7f882fefe000 rwxp 00000000 00:00 0
7f882fefe000-7f882feff000 ---p 00000000 00:00 0
7f882feff000-7f883000a000 rwxp 00000000 00:00 0
7f883000a000-7f8830029000 rwxp 00000000 00:00 0
7f8830029000-7f8830034000 rwxp 00000000 00:00 0
7f8830034000-7f88300ce000 rwxp 00000000 00:00 0
7f88300ce000-7f88300d9000 rwxp 00000000 00:00 0
7f88300d9000-7f88300f8000 rwxp 00000000 00:00 0
7f88300f8000-7f8830103000 rwxp 00000000 00:00 0
7f8830103000-7f883019c000 rwxp 00000000 00:00 0
7f883019c000-7f88301a2000 rwxp 00000000 00:00 0
7f88301a2000-7f88301ef000 rwxp 00000000 00:00 0
7f88301ef000-7f88316b0000 rwxp 00000000 00:00 0
7f88316b0000-7f88355f0000 rwxp 00000000 00:00 0
7f88355f0000-7f8836a90000 rwxp 00000000 00:00 0
7f8836a90000-7f8849f50000 rwxp 00000000 00:00 0
7f8849f50000-7f884a990000 rwxp 00000000 00:00 0
7f884a990000-7f88543f0000 rwxp 00000000 00:00 0
7f88543f0000-7f8854660000 rwxp 00000000 00:00 0
7f8854660000-7f88573f0000 rwxp 00000000 00:00 0
7f88573f0000-7f88573fe000 r-xp 00000000 08:01
273534 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libzip.so
7f88573fe000-7f8857500000 ---p 0000e000 08:01
273534 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libzip.so
7f8857500000-7f8857503000 rw-p 00010000 08:01
273534 /usr/lib/jvm/java-6-sun-1.6.0.15/jre/lib/
amd64/libzip.so
7f8857503000-7f8857504000 rw-p 00000000 00:00 0
7f8857504000-7f8857510000 r-xp 00000000 08:01
601 /lib/libnss_files-2.10.1.so
7f8857510000-7f885770f000 ---p 0000c000 08:01
601 /lib/libnss_files-2.10.1.so
7f885770f000-7f8857710000 r--p 0000b000 08:01
601 /lib/libnss_files-2.10.1.so
7f8857710000-7f8857711000 rw-p 0000c000 08:01
601 /lib/libnss_files-2.10.1.so
7f8857711000-7f885771b000 r-xp 00000000 08:01
611 /lib/libnss_nis-2.10.1.so
7f885771b000-7f885791a000 ---p 0000a000 08:01
611 /lib/libnss_nis-2.10.1.so
7f885791a000-7f885791b000 r--p 00009000 08:01
611 /lib/libnss_Aborted

[TomTom]

You can compile Scribe with HDFS and Hadoop Support? That is awesome.
I hadn't come across that yet.

Anthony do you facebook guys write directly to an HDFS file system, or
do you suck it in later with hadoop?

Would someone mind describing how to compile Scribe with hadoop, and
hdfs. Is it one of the configure options?
-Tom

[Dhruba Borthakur]

Here is a blog post I wrote a couple of months back on how scribe and hdfs works together in our environment. Scribe writes to a small HDFS cluster, and then the data is copied asynchronosly into the big (petabyte-size) HDFS/Hive warehouse

http://hadoopblog.blogspot.com/2009/06/hdfs-scribe-integration.html

thanks,
dhruba

--
Connect to me at http://www.facebook.com/dhruba

[Kimsterv]

Here's another post that should help:

http://www.cloudera.com/blog/2008/11/02/configuring-and-using-scribe-for-hadoop-log-collection/

[sebest]

This blog post is about writing hadoop logs into scribe, i want to do
the opposite :)

On 19 nov, 22:35, Kimsterv <kim.v...@gmail.com> wrote:
> Here's another post that should help:
>

> http://www.cloudera.com/blog/2008/11/02/configuring-and-using-scribe-...

[sebest]

Thanx,

I also found this link that seems to have your patches:
http://github.com/lenn0x/Hadoop-Append/

i'll try to apply them on cloudera's packages

[Kimsterv]

My bad :-)

[Kevin Weil]

At Twitter we write scribe data directly into HDFS, and have also
added (optional) LZO compression on the way in. It works great. We
have some patches to Dhruba's fantastic work, which are collectively
up at http://github.com/emaland/scribe. More coming there too, but we
use it 24 hours a day to write lots of TBs of logs a day right now.

Kevin

[sebest]

Kevin,

The git repos is about scribe, do you also have to patch hadoop/hdfs
or do you use a stock release?
Which version of hadoop do you use?

On a side note, does anyone have packaged thrift and scribe for debian/
ubuntu?

This could be interesting to make a community PPA with all theses
stuff hdfs / thrift / scribe / hive

On 20 nov, 09:01, Kevin Weil <kevinw...@gmail.com> wrote:
> At Twitter we write scribe data directly into HDFS, and have also
> added (optional) LZO compression on the way in.  It works great.  We
> have some patches to Dhruba's fantastic work, which are collectively

> up athttp://github.com/emaland/scribe.  More coming there too, but we

> ...
>
> plus de détails »

[Thomas A. McGonagle]

Hey Kevin,
 Thanks for the reply. Thanks for contributing back. Your patches are exactly what I was looking for. I look forward to trying them out.
 Is anyone else branching scribe. Are there any other branches doing cool things? The LZO compression, was exactly what I was looking for. Did you add a configuration file option or a run time server option?
 Thanks a lot.
-Tom

--
Thomas A. McGonagle
t...@dataero.com
Cell: 781- 526-4706

[eric maland]

We did add a config option that only applies to the HDFS store -

lzo_compression=N

Currently 0 == off, 1-8 are synonymous with 3 (the default in LZOP),
9=better/slower

This is currently only in my set of patches at http://github.com/emaland/scribe/
so far as I know.

Eric

On Nov 20, 5:20 pm, "Thomas A. McGonagle" <t...@dataero.com> wrote:
> Hey Kevin,
>  Thanks for the reply. Thanks for contributing back. Your patches are
> exactly what I was looking for. I look forward to trying them out.
>  Is anyone else branching scribe. Are there any other branches doing cool
> things? The LZO compression, was exactly what I was looking for. Did you add
> a configuration file option or a run time server option?
>  Thanks a lot.
> -Tom
>
>
>
> On Fri, Nov 20, 2009 at 3:01 AM, Kevin Weil <kevinw...@gmail.com> wrote:
> > At Twitter we write scribe data directly into HDFS, and have also
> > added (optional) LZO compression on the way in.  It works great.  We
> > have some patches to Dhruba's fantastic work, which are collectively

> > up athttp://github.com/emaland/scribe.  More coming there too, but we

> ...
>
> read more »

[TomTom]

Hey Thanks for the explanation Eric. It sounds like you did some
excellent computer science. Ill dig in and see how far I can get.
Using github, is there an easy way to see what is different/new/
patched? Ill send up the white flag to the scribe group, if I run into
any trouble. Have a nice week, and I hope everyone has a lovely
thanksgiving.
-Tom

On Nov 21, 6:22 pm, eric maland <eric.mal...@gmail.com> wrote:
> We did add a config option that only applies to the HDFS store -
>
> lzo_compression=N
>
> Currently 0 == off, 1-8 are synonymous with 3 (the default in LZOP),
> 9=better/slower
>

> This is currently only in my set of patches athttp://github.com/emaland/scribe/

> ...
>
> read more »

[Kevin Weil]

Tom,

Eric took the time to break his patches apart into multiple branches
-- check out the branches to that repository on github. There's
pretty much one feature per branch, which should make it easy to see
where ours is different and what's been added.

HTH,
Kevin

> ...
>
> read more »

[Thomas A. McGonagle]

Kevin,
 Thanks for the explanation. Ill dig into github and try to figure it all out. I am most interested in the LZO compression, Ill see if I can only use that branch.
 Thanks again. I hope you had a nice Thanksgiving weekend.
 Have a great day!
-Tom

[tsuna]

On Nov 19 2009, 9:29 am, sebest <sebastien.estie...@gmail.com> wrote:
> [Thu Nov 19 18:29:59 2009] "[hdfs] Connecting to HDFS"
> *** glibc detected *** ./scribed: munmap_chunk(): invalid pointer: 0x0000000001ea19c3 ***

This is due to a gross bug in HdfsFile::connectToPath - it's moving a
pointer to the middle of a buffer and then free()ing it. I don't know
why such a bug is still in everyone's master branch but I fixed it in
http://github.com/tsuna/scribe/commit/e888dcd245722f71be53e709b791ffe26e914ae2
(I forked Eric Maland's repository so my tree also gives you LZO
support).

[Anthony Giardullo]

Thanks for the bug report. Taking a look...

-Anthony

[lossnopro lossnopro]

Thanks tsuna!