[scribe-server] Server crashing when attempting to write to HDFS
Josh
with HDFS support. However, whenever the server receives a message and
attempts to write it out, I get this crash:
Mon Jan 25 21:37:23 2010] "STATUS: STARTING"
[Mon Jan 25 21:37:23 2010] "STATUS: configuring"
[Mon Jan 25 21:37:23 2010] "got configuration data from file </usr/
local/scribe/scribe.conf>"
[Mon Jan 25 21:37:23 2010] "CATEGORY : default"
[Mon Jan 25 21:37:23 2010] "Creating default store"
[Mon Jan 25 21:37:23 2010] "configured <1> stores"
[Mon Jan 25 21:37:23 2010] "STATUS: "
[Mon Jan 25 21:37:23 2010] "STATUS: ALIVE"
[Mon Jan 25 21:37:23 2010] "Starting scribe server on port 1463"
Thrift: Mon Jan 25 21:37:23 2010 TServerSocket::listen() IPV6_V6ONLY
Thrift: Mon Jan 25 21:37:23 2010 libevent 1.3e method epoll
[Mon Jan 25 21:37:25 2010] "[Ruby] Creating new category from model
default"
[Mon Jan 25 21:37:25 2010] "store thread starting"
[Mon Jan 25 21:37:25 2010] "[hdfs] Connecting to HDFS"
could not find method newInstance from class org/apache/hadoop/fs/
FileSystem with signature (Ljava/net/URI;Lorg/apache/hadoop/conf/
Configuration;)Lorg/apache/hadoop/fs/FileSystem;
Exception in thread "main" java.lang.NoSuchMethodError: newInstance
Call to org.apache.hadoop.fs.Filesystem::newInstance(URI,
Configuration) failed!
*** glibc detected *** scribed: free(): invalid pointer: 0x09fadc24
***
======= Backtrace: =========
/lib/tls/i686/nosegneg/libc.so.6[0x454274]
/lib/tls/i686/nosegneg/libc.so.6(cfree+0x96)[0x456296]
scribed[0x8088ab1]
scribed[0x80891d8]
scribed[0x80729a5]
scribed[0x8072c17]
scribed[0x8063c0a]
scribed[0x80644d3]
scribed[0x8051611]
scribed[0x8055429]
scribed[0x806a410]
scribed[0x806ac51]
/lib/tls/i686/nosegneg/libpthread.so.0[0x3d1545]
/lib/tls/i686/nosegneg/libc.so.6(clone+0x5e)[0x4caa8e]
======= Memory map: ========
00110000-0018f000 r-xp 00000000 08:01 363810 /usr/local/lib/
libthrift.so.0.0.0
0018f000-00192000 r--p 0007f000 08:01 363810 /usr/local/lib/
libthrift.so.0.0.0
00192000-00193000 rw-p 00082000 08:01 363810 /usr/local/lib/
libthrift.so.0.0.0
00193000-001a0000 r-xp 00000000 08:01 417945 /lib/libgcc_s.so.1
001a0000-001a1000 r--p 0000c000 08:01 417945 /lib/libgcc_s.so.1
001a1000-001a2000 rw-p 0000d000 08:01 417945 /lib/libgcc_s.so.1
001a2000-001b7000 r-xp 00000000 08:01 418733 /lib/tls/i686/
nosegneg/libnsl-2.9.so
001b7000-001b8000 r--p 00014000 08:01 418733 /lib/tls/i686/
nosegneg/libnsl-2.9.so
001b8000-001b9000 rw-p 00015000 08:01 418733 /lib/tls/i686/
nosegneg/libnsl-2.9.so
001b9000-001bb000 rw-p 001b9000 00:00 0
001bb000-001bd000 r-xp 00000000 08:01 418730 /lib/tls/i686/
nosegneg/libdl-2.9.so
001bd000-001be000 r--p 00001000 08:01 418730 /lib/tls/i686/
nosegneg/libdl-2.9.so
001be000-001bf000 rw-p 00002000 08:01 418730 /lib/tls/i686/
nosegneg/libdl-2.9.so
001bf000-001e2000 r-xp 00000000 08:01 476892 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/libjava.so
001e2000-001e4000 rw-p 00023000 08:01 476892 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/libjava.so
001e4000-001ea000 r-xp 00000000 08:01 476880 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/native_threads/libhpi.so
001ea000-001eb000 rw-p 00006000 08:01 476880 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/native_threads/libhpi.so
001eb000-001f2000 r-xp 00000000 08:01 418734 /lib/tls/i686/
nosegneg/libnss_compat-2.9.so
001f2000-001f3000 r--p 00006000 08:01 418734 /lib/tls/i686/
nosegneg/libnss_compat-2.9.so
001f3000-001f4000 rw-p 00007000 08:01 418734 /lib/tls/i686/
nosegneg/libnss_compat-2.9.so
001f4000-001fd000 r-xp 00000000 08:01 418738 /lib/tls/i686/
nosegneg/libnss_nis-2.9.so
001fd000-001fe000 r--p 00008000 08:01 418738 /lib/tls/i686/
nosegneg/libnss_nis-2.9.so
001fe000-001ff000 rw-p 00009000 08:01 418738 /lib/tls/i686/
nosegneg/libnss_nis-2.9.so
001ff000-00209000 r-xp 00000000 08:01 418736 /lib/tls/i686/
nosegneg/libnss_files-2.9.so
00209000-0020a000 r--p 00009000 08:01 418736 /lib/tls/i686/
nosegneg/libnss_files-2.9.so
0020a000-0020b000 rw-p 0000a000 08:01 418736 /lib/tls/i686/
nosegneg/libnss_files-2.9.so
0020b000-0021a000 r-xp 00000000 08:01 476895 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/libzip.so
0021a000-0021c000 rw-p 0000e000 08:01 476895 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/libzip.so
0021d000-0022f000 r-xp 00000000 08:01 344946 /usr/lib/
libboost_filesystem-mt.so.1.37.0
0022f000-00230000 r--p 00011000 08:01 344946 /usr/lib/
libboost_filesystem-mt.so.1.37.0
00230000-00231000 rw-p 00012000 08:01 344946 /usr/lib/
libboost_filesystem-mt.so.1.37.0
00231000-00238000 rwxp 00231000 00:00 0
00252000-0025d000 r-xp 00000000 08:01 476891 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/libverify.so
0025d000-0025e000 rw-p 0000b000 08:01 476891 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/libverify.so
0025e000-0025f000 rwxp 0025e000 00:00 0
0025f000-00260000 rwxp 0025f000 00:00 0
00260000-00263000 rwxp 00260000 00:00 0
00263000-0027e000 rwxp 00263000 00:00 0
0027e000-00284000 rwxp 0027e000 00:00 0
00284000-0029e000 rwxp 00284000 00:00 0
0029e000-002a9000 rwxp 0029e000 00:00 0
002c9000-002dc000 r-xp 00000000 08:01 345848 /usr/lib/
libevent-1.3e.so.1.0.3
002dc000-002de000 rw-p 00013000 08:01 345848 /usr/lib/
libevent-1.3e.so.1.0.3
002de000-002df000 rw-p 002de000 00:00 0
002df000-002e1000 rwxp 002df000 00:00 0
002e1000-0035f000 rwxp 002e1000 00:00 0
0035f000-00362000 ---p 0035f000 00:00 0
00362000-003b0000 rwxp 00362000 00:00 0
003cb000-003e0000 r-xp 00000000 08:01 418741 /lib/tls/i686/
nosegneg/libpthread-2.9.so
003e0000-003e1000 r--p 00014000 08:01 418741 /lib/tls/i686/
nosegneg/libpthread-2.9.so
003e1000-003e2000 rw-p 00015000 08:01 418741 /lib/tls/i686/
nosegneg/libpthread-2.9.so
003e2000-003e4000 rw-p 003e2000 00:00 0
003e4000-00543000 r-xp 00000000 08:01 418727 /lib/tls/i686/
nosegneg/libc-2.9.so
00543000-00544000 ---p 0015f000 08:01 418727 /lib/tls/i686/
nosegneg/libc-2.9.so
00544000-00546000 r--p 0015f000 08:01 418727 /lib/tls/i686/
nosegneg/libc-2.9.so
00546000-00547000 rw-p 00161000 08:01 418727 /lib/tls/i686/
nosegneg/libc-2.9.so
00547000-0054a000 rw-p 00547000 00:00 0
00579000-00595000 r-xp 00000000 08:01 417925 /lib/ld-2.9.so
00595000-00596000 r--p 0001b000 08:01 417925 /lib/ld-2.9.so
00596000-00597000 rw-p 0001c000 08:01 417925 /lib/ld-2.9.so
005aa000-0068e000 r-xp 00000000 08:01 346668 /usr/lib/libstdc++.so.
6.0.10
0068e000-00692000 r--p 000e3000 08:01 346668 /usr/lib/libstdc++.so.
6.0.10
00692000-00693000 rw-p 000e7000 08:01 346668 /usr/lib/libstdc++.so.
6.0.10
00693000-00699000 rw-p 00693000 00:00 0
00699000-0069a000 ---p 00699000 00:00 0
0069a000-0079a000 rwxp 0069a000 00:00 0
0079a000-0079b000 ---p 0079a000 00:00 0
0079b000-0081b000 rwxp 0079b000 00:00 0
00829000-0084d000 r-xp 00000000 08:01 418731 /lib/tls/i686/
nosegneg/libm-2.9.so
0084d000-0084e000 r--p 00023000 08:01 418731 /lib/tls/i686/
nosegneg/libm-2.9.so
0084e000-0084f000 rw-p 00024000 08:01 418731 /lib/tls/i686/
nosegneg/libm-2.9.so
0084f000-00852000 ---p 0084f000 00:00 0
00852000-008a0000 rwxp 00852000 00:00 0
008a0000-008a3000 ---p 008a0000 00:00 0
008a3000-008f1000 rwxp 008a3000 00:00 0
008f1000-008f4000 ---p 008f1000 00:00 0
008f4000-00972000 rwxp 008f4000 00:00 0
00972000-00975000 ---p 00972000 00:00 0
00975000-009c3000 rwxp 00975000 00:00 0
009c3000-009c4000 ---p 009c3000 00:00 0
009c4000-00a44000 rwxp 009c4000 00:00 0
00aaa000-00aab000 r-xp 00aaa000 00:00 0 [vdso]
00b0a000-00b0d000 rwxp 00b0a000 00:00 0
00b0d000-00b28000 rwxp 00b0d000 00:00 0
00bfe000-00c01000 r-xp 00000000 08:01 344945 /usr/lib/
libboost_system-mt.so.1.37.0
00c01000-00c02000 r--p 00002000 08:01 344945 /usr/lib/
libboost_system-mt.so.1.37.0
00c02000-00c03000 rw-p 00003000 08:01 344945 /usr/lib/
libboost_system-mt.so.1.37.0
00c7e000-00c85000 rwxp 00c7e000 00:00 0
00c85000-00c9f000 rwxp 00c85000 00:00 0
00d24000-00d36000 r-xp 00000000 08:01 418742 /lib/tls/i686/
nosegneg/libresolv-2.9.so
00d36000-00d37000 r--p 00011000 08:01 418742 /lib/tls/i686/
nosegneg/libresolv-2.9.so
00d37000-00d38000 rw-p 00012000 08:01 418742 /lib/tls/i686/
nosegneg/libresolv-2.9.so
00d38000-00d3a000 rw-p 00d38000 00:00 0
00d52000-00d59000 r-xp 00000000 08:01 418743 /lib/tls/i686/
nosegneg/librt-2.9.so
00d59000-00d5a000 r--p 00006000 08:01 418743 /lib/tls/i686/
nosegneg/librt-2.9.so
00d5a000-00d5b000 rw-p 00007000 08:01 418743 /lib/tls/i686/
nosegneg/librt-2.9.so
00dd9000-00dde000 rwxp 00dd9000 00:00 0
00dfc000-00e05000 r-xp 00000000 08:01 346057 /usr/lib/libhdfs.so.
0.0.0
00e05000-00e06000 r--p 00008000 08:01 346057 /usr/lib/libhdfs.so.
0.0.0
00e06000-00e07000 rw-p 00009000 08:01 346057 /usr/lib/libhdfs.so.
0.0.0
00e40000-00e41000 ---p 00e40000 00:00 0
00e41000-00f41000 rwxp 00e41000 00:00 0
00f50000-00f5d000 r-xp 00000000 08:01 363876 /usr/local/lib/
libthriftnb.so.0.0.0
00f5d000-00f5e000 r--p 0000c000 08:01 363876 /usr/local/lib/
libthriftnb.so.0.0.0
00f5e000-00f5f000 rw-p 0000d000 08:01 363876 /usr/local/lib/
libthriftnb.so.0.0.0
00f5f000-013c5000 r-xp 00000000 08:01 476886 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/client/libjvm.so
013c5000-013e5000 rw-p 00465000 08:01 476886 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/client/libjvm.so
013e5000-01805000 rw-p 013e5000 00:00 0
01805000-01885000 rwxp 01805000 00:00 0
01885000-03805000 rwxp 01885000 00:00 0
073fa000-073fd000 ---p 073fa000 00:00 0
073fd000-07bfb000 rwxp 073fd000 00:00 0
08048000-080b5000 r-xp 00000000 08:01 363902 /usr/local/bin/
scribed
080b5000-080b6000 r--p 0006c000 08:01 363902 /usr/local/bin/
scribed
080b6000-080b7000 rw-p 0006d000 08:01 363902 /usr/local/bin/
scribed
09fac000-0a23a000 rw-p 09fac000 00:00 0
8b7c0000-8b8c0000 rwxp 8b7c0000 00:00 0
8b8c0000-8bca0000 rwxp 8b8c0000 00:00 0
8bca0000-8c0a0000 rwxp 8bca0000 00:00 0
8c0a0000-8f7c0000 rwxp 8c0a0000 00:00 0
8f7c0000-903c0000 rwxp 8f7c0000 00:00 0
903c0000-937c0000 rwxp 903c0000 00:00 0
937c0000-93dbb000 r--s 00001000 08:01 477820 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/client/classes.jsa
93dbb000-93fc0000 rwxp 93dbb000 00:00 0
93fc0000-946d8000 rw-p 005fc000 08:01 477820 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/client/classes.jsa
946d8000-94bc0000 rwxp 946d8000 00:00 0
94bc0000-94c9e000 rw-p 00d14000 08:01 477820 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/client/classes.jsa
94c9e000-94fc0000 rwxp 94c9e000 00:00 0
94fc0000-94fc8000 r-xs 00df2000 08:01 477820 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/i386/client/classes.jsa
94fc8000-953c0000 rwxp 94fc8000 00:00 0
b7900000-b7921000 rw-p b7900000 00:00 0
b7921000-b7a00000 ---p b7921000 00:00 0
b7a13000-b7a21000 r--s 000f0000 08:01 420409 /usr/lib/hadoop-0.18/
hadoop-0.18.3+76-test.jar
b7a21000-b7a23000 r--s 0000e000 08:01 500468 /usr/lib/hadoop-0.18/
lib/oro-2.0.8.jar
b7a23000-b7a24000 r--s 00008000 08:01 500485 /usr/lib/hadoop-0.18/
lib/hadoop-0.18.3+76-fairscheduler.jar
b7a24000-b7a2d000 r--s 000a4000 08:01 500472 /usr/lib/hadoop-0.18/
lib/hsqldb.jar
b7a2d000-b7a34000 r--s 0009c000 08:01 500483 /usr/lib/hadoop-0.18/
lib/jetty-5.1.4.jar
b7a34000-b7a36000 r--s 00016000 08:01 500477 /usr/lib/hadoop-0.18/
lib/servlet-api.jar
b7a36000-b7a37000 r--s 00004000 08:01 500475 /usr/lib/hadoop-0.18/
lib/hadoop-0.18.3+76-scribe-log4j.jar
b7a37000-b7a3d000 r--s 0002b000 08:01 500474 /usr/lib/hadoop-0.18/
lib/junit-4.5.jar
b7a3d000-b7a3f000 r--s 0000a000 08:01 500484 /usr/lib/hadoop-0.18/
lib/commons-codec-1.3.jar
b7a3f000-b7a40000 r--s 00003000 08:01 500486 /usr/lib/hadoop-0.18/
lib/xmlenc-0.52.jar
b7a40000-b7a45000 r--s 00028000 08:01 500473 /usr/lib/hadoop-0.18/
lib/commons-net-1.4.1.jar
b7a45000-b7a48000 r--s 0000f000 08:01 500471 /usr/lib/hadoop-0.18/
lib/libthrift.jar
b7a48000-b7a4a000 r--s 00001000 08:01 500469 /usr/lib/hadoop-0.18/
lib/slf4j-log4j12-1.4.3.jar
b7a4a000-b7a4f000 r--s 00040000 08:01 500480 /usr/lib/hadoop-0.18/
lib/commons-httpclient-3.0.1.jar
b7a4f000-b7a55000 r--s 0007f000 08:01 500481 /usr/lib/hadoop-0.18/
lib/mysql-connector-java-5.0.8-bin.jar
b7a55000-b7a5a000 r--s 0004a000 08:01 500460 /usr/lib/hadoop-0.18/
lib/jets3t-0.6.1.jar
b7a5a000-b7a5b000 r--s 00003000 08:01 500461 /usr/lib/hadoop-0.18/
lib/slf4j-api-1.4.3.jar
b7a5b000-b7a5d000 r--s 00005000 08:01 500482 /usr/lib/hadoop-0.18/
lib/commons-logging-api-1.0.4.jar
b7a5d000-b7a64000 r--s 00059000 08:01 500470 /usr/lib/hadoop-0.18/
lib/log4j-1.2.15.jar
b7a64000-b7a67000 r--s 00019000 08:01 500465 /usr/lib/hadoop-0.18/
lib/jetty-ext/commons-el.jar
b7a67000-b7a6c000 r--s 00051000 08:01 500464 /usr/lib/hadoop-0.18/
lib/jetty-ext/jasper-compiler.jar
b7a6c000-b7a6e000 r--s 00018000 08:01 500466 /usr/lib/hadoop-0.18/
lib/jetty-ext/jasper-runtime.jar
b7a6e000-b7a70000 r--s 0000b000 08:01 500463 /usr/lib/hadoop-0.18/
lib/jetty-ext/jsp-api.jar
b7a70000-b7a72000 r--s 0000f000 08:01 500478 /usr/lib/hadoop-0.18/
lib/libfb303.jar
b7a72000-b7a74000 r--s 00008000 08:01 500479 /usr/lib/hadoop-0.18/
lib/commons-logging-1.0.4.jar
b7a74000-b7a79000 r--s 0003b000 08:01 500467 /usr/lib/hadoop-0.18/
lib/commons-cli-2.0-SNAPSHOT.jar
b7a79000-b7a7a000 r--s 00002000 08:01 500476 /usr/lib/hadoop-0.18/
lib/kfs-0.1.3.jar
b7a7a000-b7a7c000 r--s 0000a000 08:01 420410 /usr/lib/hadoop-0.18/
hadoop-0.18.3+76-tools.jar
b7a7c000-b7a7e000 r--s 0000e000 08:01 500669 /usr/lib/hadoop-0.18/
contrib/sqoop/hadoop-0.18.3+76-sqoop.jar
b7a7e000-b7a80000 r--s 0000e000 08:01 500671 /usr/lib/hadoop-0.18/
contrib/index/hadoop-0.18.3+76-index.jar
b7a80000-b7a81000 r--s 00008000 08:01 500667 /usr/lib/hadoop-0.18/
contrib/fairscheduler/hadoop-0.18.3+76-fairscheduler.jar
b7a81000-b7a83000 r--s 00002000 08:01 500661 /usr/lib/hadoop-0.18/
contrib/datajoin/hadoop-0.18.3+76-datajoin.jar
b7a83000-b7a84000 r--s 00005000 08:01 500659 /usr/lib/hadoop-0.18/
contrib/mrunit/hadoop-0.18.3+76-mrunit.jar
b7a84000-b7a86000 r--s 00019000 08:01 500663 /usr/lib/hadoop-0.18/
contrib/streaming/hadoop-0.18.3+76-streaming.jar
b7a86000-b7a87000 r--s 00004000 08:01 500665 /usr/lib/hadoop-0.18/
contrib/scribe-log4j/hadoop-0.18.3+76-scribe-log4j.jar
b7a87000-b7a88000 r--s 00001000 08:01 420413 /usr/lib/hadoop-0.18/
hadoop-0.18.3+76-ant.jar
b7a88000-b7a8b000 r--s 00016000 08:01 420417 /usr/lib/hadoop-0.18/
hadoop-0.18.3+76-examples.jar
b7a8b000-b7aa7000 r--s 001da000 08:01 420412 /usr/lib/hadoop-0.18/
hadoop-0.18.3+76-core.jar
b7aa7000-b7ada000 rw-p b7aa7000 00:00 0
b7ada000-b7c70000 r--s 02fb3000 08:01 395874 /usr/lib/jvm/java-6-
sun-1.6.0.16/jre/lib/rt.jar
b7c70000-b7db4000 r--p 00000000 08:01 352812 /usr/lib/locale/
locale-archive
b7db4000-b7dbc000 rw-s 00000000 08:01 286724 /tmp/hsperfdata_root/
21113
b7dbc000-b7dbd000 rw-p b7dbc000 00:00 0
b7dbd000-b7dbe000 r--p b7dbd000 00:00 0
b7dbe000-b7f03000 rw-p b7dbe000 00:00 0
b7f09000-b7f0c000 rw-p b7f09000 00:00 0
bfd4d000-bfd61000 rwxp bfd4d000 00:00 0 [stack]
bfd61000-bfd62000 rw-p bfd61000 00:00 0
Aborted
Anybody have any ideas about this?
Thanks,
Josh
tsuna
> Hello, I've finally after days of trying managed to compile Scribe
> with HDFS support. However, whenever the server receives a message and
> attempts to write it out, I get this crash:
> *** glibc detected *** scribed: free(): invalid pointer: 0x09fadc24 ***
> Anybody have any ideas about this?
Yes. It's an obvious bug in the code that initializes the connection to HDFS.
I provided a patch in
http://github.com/tsuna/scribe/commit/dec093654ebe9f5c7ed14cebf4482615d51c0bf0
and it's being reviewed by Anthony for integration in facebook's tree.
If you just want a quick hack you can initialize `buf' to NULL and
comment out every line that does `free(buf);' in
HdfsFile::connectToPath in src/HdfsFile.cpp
--
Benoit Sigoure aka Tsuna
Software Engineer @ www.StumbleUpon.com
Gautam Roy
Just pushed the fix Tsuna did to the facebook tree. You can pull the
latest code and that should fix your problem.
Best,
Gautam