how to limit range size in 200MB without core dump

[swin]

In my APP, if a range up to 200MB, we hope it doesn't accept now data,
but not core dump.
in range.cc:
if (m_split_row < m_start_row || m_split_row >= m_end_row) {
HT_FATALF("Unable to determine split row for range %s[%s..
%s]", m_identifier.name, m_start_row.c_str(), m_end_row.c
_str());
DUMP_CORE;
}

How can I do this?

Thanks.

[Doug Judd]

Hi Swin,

I'll fix it today and push it out to the public repository tonight.  We're shooting for a 0.9.0.5 release by the end of the week.  The fix will be in that release as well.

- Doug

[swin]

I hope we can test it in this week, we have been distressed about
this
for weeks.

thanks.

On 5月14日, 上午6时33分, "Doug Judd" <d...@zvents.com> wrote:
> Hi Swin,
>
> I'll fix it today and push it out to the public repository tonight. We're
> shooting for a 0.9.0.5 release by the end of the week. The fix will be in
> that release as well.
>
> - Doug
>

[Doug Judd]

Hi Swin,

It's looking a little more involved that I initially thought.  I'm 100% focused on fixing this right now and as soon as I'm finished I'll push out the fix and let you know.  Thanks for your patience.

- Doug

[Doug Judd]

Hi Swin,

I've made the changes that you've requested and have pushed them into the public repository.  Take a look at the apache_log/apache_log_load.cc to see how it now works.  In a nutshell, the client will now throw a RANGESERVER_ROW_OVERFLOW exception upon encountering this situation.  You can obtain the failed mutations with a call to TableMutator::get_failed().  You need to wrap your flush() call in a loop to ensure that all of the internal send buffers get properly flushed.  Here's an excerpt from the apache_log example:

  // Flush pending updates
  while (true) {
    try {
      mutator_ptr->flush();
    }
    catch (Exception &e) {
      report_mutation_error(e, mutator_ptr);
      // this continue is necessary since the flush may
      // not have completed when exception was thrown
      continue;
    }
    break;
  }

And here's the code for report_mutation_error:

  void report_mutation_error(Exception &e, TableMutatorPtr &mutator_ptr) {
    std::vector<std::pair<Cell, int> > failed_mutations;

    mutator_ptr->get_failed(failed_mutations);
    if (!failed_mutations.empty()) {
      for (size_t i=0; i<failed_mutations.size(); i++) {
        cerr << "Failed: (" << failed_mutations[i].first.row_key << ","
             << failed_mutations[i].first.column_family;
        if (failed_mutations[i].first.column_qualifier)
          cerr << ":" << failed_mutations[i].first.column_qualifier;
        cerr << "," << failed_mutations[i].first.timestamp << ") - "
             << Error::get_text(failed_mutations[i].second) << endl;
      }
    }
    else
      cerr << "Error: " << Error::get_text(e.code()) << endl;
  }

Upon receiving this exception, you need to take care of these failures immediately (e.g. log them or something).  Any subsequent call into the API will invalidate the failed_mutations vector since it points into the internal send buffers.  These internal send buffers will get garbage collected on the subsequent API call.

One other thing to note.  I've changed CellT typedef to a class called Cell, so you'll need to update your code to reflect this change.  Thanks for your patience and let me know if you have any other issues.

- Doug

[swin]

Ok, thanks,we we will test it this week.

[swin]

hello
We update the soucecode,but it still core dump:
$ tail -n 20 DfsBroker.hadoop.log
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:206)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:
207)
at
org.hypertable.AsyncComm.IOHandlerData.run(IOHandlerData.java:100)
at org.hypertable.AsyncComm.Reactor.run(Reactor.java:83)
at java.lang.Thread.run(Thread.java:595)
May 20, 2008 3:56:20 PM org.hypertable.DfsBroker.hadoop.HdfsBroker
Create
INFO: I/O exception while creating file '/hypertable/tables/METADATA/
default/36289BE250857072500CF0DD/cs1' -
org.apache.hadoop.dfs.AlreadyBeingCreatedException: failed to create
file /hypertable/tables/METADATA/default/36289BE250857072500CF0DD/cs1
for DFSClient_321576484 on client 10.23.3.30 because current
leaseholder is trying to recreate file.
at
org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:
1030)
at
org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:986)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:301)
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)

May 20, 2008 3:56:20 PM org.hypertable.DfsBroker.hadoop.HdfsBroker
Create
SEVERE: Problem sending response to 'create' command - COMM not
connected


$ tail -n 30 Hypertable.RangeServer.log
1211269004 INFO Hypertable.RangeServer : (/home/hadoop/hypertable/src/
cc/Hypertable/RangeServer/RangeServer.cc:1160) drj mem=162594546
items=556842 vm-est229415586
RangeServer::update
Table Name = site_index
Table ID = 2
Generation = 1
1211269019 INFO Hypertable.RangeServer : (/home/hadoop/hypertable/src/
cc/Hypertable/RangeServer/RangeServer.cc:1091) Added 8241 (0 split
off) updates to 'site_index'
1211269027 INFO Hypertable.RangeServer : (/home/hadoop/hypertable/src/
cc/Hypertable/RangeServer/RangeServer.cc:1091) Added 15420 (0 split
off) updates to 'site_index'
1211269028 INFO Hypertable.RangeServer : (/home/hadoop/hypertable/src/
cc/Hypertable/RangeServer/RangeServer.cc:1574) Compacting METADATA[..
0:ÿÿ].default because cumulative log size of 201450801 exceeds
threshold 200000000
1211269029 INFO Hypertable.RangeServer : (/home/hadoop/hypertable/src/
cc/Hypertable/RangeServer/RangeServer.cc:1574) Compacting METADATA[..
0:ÿÿ].location because cumulative log size of 201450801 exceeds
threshold 200000000
1211269029 INFO Hypertable.RangeServer : (/home/hadoop/hypertable/src/
cc/Hypertable/RangeServer/RangeServer.cc:1574) Compacting
METADATA[0:ÿÿ..ÿÿ].default because cumulative log size of 201450801
exceeds threshold 200000000
1211269029 INFO Hypertable.RangeServer : (/home/hadoop/hypertable/src/
cc/Hypertable/RangeServer/RangeServer.cc:1574) Compacting
METADATA[0:ÿÿ..ÿÿ].location because cumulative log size of 201450801
exceeds threshold 200000000
1211269029 INFO Hypertable.RangeServer : (/home/hadoop/hypertable/src/
cc/Hypertable/RangeServer/AccessGroup.cc:270) Starting Minor
Compaction (METADATA.default)
1211269037 INFO Hypertable.RangeServer : (/home/hadoop/hypertable/src/
cc/Hypertable/RangeServer/RangeServer.cc:1091) Added 22591 (0 split
off) updates to 'site_index'
1211269071 INFO Hypertable.RangeServer : (/home/hadoop/hypertable/src/
cc/Hypertable/RangeServer/HyperspaceSessionHandler.cc:37) Hyperspace
session state change: JEOPARDY
RangeServer::create_scanner
Table Name = METADATA
Table ID = 0
Generation = 1
StartRow = ""
EndRow = "0:ÿÿ"
RowLimit = 1211269173 ERROR Hypertable.RangeServer : (/home/hadoop/
hypertable/src/cc/Hypertable/RangeServer/HyperspaceSessionHandler.cc:
44) Hyperspace session expired. Exiting...
0
MaxVersions = 0
Columns = Files
StartRow =
StartRowInclusive = 0
EndRow = ÿÿ
EndRowInclusive = 0
MinTime = 0
MaxTime = 18446744073709551615

[Doug Judd]

Hi Wang,

Can you pull the latest code and try it?  We've checked in a few fixes within the last day or two that may be related.  Also, if you still see the problem, can you tar up the log files and post them to the mailing list?  That would help me better diagnose the problem.

Also, what version of Hadoop are you using?

- Doug

[donald]

Hi Doug,

We installed hyperspace 0.9.0.5-alpha release on 4 servers (1 master +
hyperspace, 3 range servers, HDFS is deployed on some other servers
within the same cluster) today, and tried to insert batches of data
into it. The process goes well at first. However we experienced
several problems later.

Each batch of our input data contains around 500 text log files, which
are about 350MB each. We wrote a LogLoad program to load a single text
log file, which is very similar to the apach_log_load.cc example, only
with a different parser. We also wrote a script to invoke LogLoad to
load every log file.

After the first 4 hours, our first batch of 167GB data were
successfully inserted, despite seeing some "RANGE SERVER row overflow"
exceptions. We simply ignored those exceptions. At this time, each of
the 3 range servers takes about 65% of the available 4GB physical
memory.

However, when we tried to load a second batch of data, we were out of
luck. At first the LogLoad program gave a lot of core dumps. gdb says
the core was caused by a segmentation fault in
TableMutatorScatterBuffer::send().

After some further inspection, we found that, during running the
following code segment:

// Flush pending updates
while (true) {
try {
mutator_ptr->flush();

} catch ( Hypertable::Exception &e ) {
handle_mutation_failure(e, mutator_ptr);
continue ;
}
break;
}

The TableMutator::flush() may still generate "RANGE SERVER row
overflow" exceptions, and upon catching it, we call
TableMutator::flush() again, and catches another "RANGE SERVER row
overflow" exception soon after, so we try again and again and again...
Somehow after a lot of retries, some internal data structure gets
corrupted, and thus we get a segmentation fault. We are not sure how
many retries it takes to actually cause the fault, it seems sometimes
the flush() call succeeds before it is too late. We also tried to use
the new TableMutator::retry() function instead of
TableMutator::flush(), as the new version of apache_log_load.cc did,
but the same problem remains.

The second problem we met is: if we ignore the core dumps and continue
loading more data, the memory usage of the range servers goes higher
and higher, and eventually takes over 95% of the 4GB physical memory.
After that the servers became very slow, probably because the os was
busy swapping pages. Minutes later, the servers began killing
themselves by complaining "1212573610 ERROR Hypertable.RangeServer : (/
home/work/hypertable-0.9.0.5-alpha/src/cc/Hypertable/RangeServer/
HyperspaceSessionHandler.cc:44) Hyperspace session expired.
Exiting..." in their log. But weird enough we can see the
Hypertable.RangeServer processes still running in the output of "ps -
A" or "top" after that.

Now we wonder:
1. Is there a safe way to flush all buffered table mutations, ignoring
those that cause exceptions?
2. Is there any global control on memory usuage of the range servers?
We only see two related configuration
Hypertable.RangeServer.AccessGroup.MaxMemory and
Hypertable.RangeServer.BlockCache.MaxMemory, which default to 50MB and
200MB. Does that mean if one range server is serving 100 ranges, each
containing just 1 access group, the whole server process might take up
to 50MBx100+200MB=5.2GB memory? Is there a simple way to observe how
many ranges are served on a specific range server?

Thanks.

Donald

On 5月22日, 上午9时23分, "Doug Judd" <d...@zvents.com> wrote:
> Hi Wang,
>
> Can you pull the latest code and try it? We've checked in a few fixes
> within the last day or two that may be related. Also, if you still see the
> problem, can you tar up the log files and post them to the mailing

> list<http://groups.google.com/group/hypertable-dev/files?upload=1>?

> That would help me better diagnose the problem.
>
> Also, what version of Hadoop are you using?
>
> - Doug
>

[Doug Judd]

Hi Donald,

Thanks for the excellent write-up!  Is there anyway you could send me a stack trace from the core file?  If you build a debug version of your LogLoad program, that will provide the best information.  When it dumps core, load the core file in gdb and issue the following command which will dump out a stack trace for all threads in the program at the time of the crash.

(gdb) thread apply all where

Also, be sure to use the new retry() method as described in the latest apache_log_load.cc example.  The retry() method is supposed to flush all valid mutations, ignoring the failed ones.

I'll look into the memory issue.  There currently is no throttling based on memory consumption, but an issue has been filed for it (see issue 84).

- Doug

Also, there may be a memory leak.  I'll look into

[donald]

Hi Doug,

Well, I was stupid enough that I kept all the core files, but modified
and recompiled the LogLoad program that generates them... so now if I
load them in gdb the symbol table doesn't match, that's why I didn't
include a stack trace :(

I will try to regenerate the same problem and provide more information
soon. Also I changed Hypertable.RangeServer.AccessGroup.MaxMemory to
10M to see if we still have that memory problem.

Thanks.

Donald

On Jun 4, 9:44 pm, "Doug Judd" <d...@zvents.com> wrote:
> Hi Donald,
>
> Thanks for the excellent write-up! Is there anyway you could send me a
> stack trace from the core file? If you build a debug version of your
> LogLoad program, that will provide the best information. When it dumps
> core, load the core file in gdb and issue the following command which will
> dump out a stack trace for all threads in the program at the time of the
> crash.
>
> (gdb) thread apply all where
>
> Also, be sure to use the new retry() method as described in the latest
> apache_log_load.cc example. The retry() method is supposed to flush all
> valid mutations, ignoring the failed ones.
>
> I'll look into the memory issue. There currently is no throttling based on
> memory consumption, but an issue has been filed for it (see issue

> 84<http://code.google.com/p/hypertable/issues/detail?id=84>
> ).

>
> - Doug
>
> Also, there may be a memory leak. I'll look into
>

> ...
>
> read more »

[donald]

Hi Doug,

Here comes the gdb backtrace:

(gdb) thread apply all where

Thread 7 (process 28139):
#0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
#1 0x00000000004b4e50 in Hypertable::ReactorRunner::operator() (
this=0x40a001a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ReactorRunner.cc:59
#2 0x00000000004c97db in
boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (function_obj_ptr=@0x40a001a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/boost-1_34-fix/boost/
function/function_template.hpp:158
#3 0x0000002a95577dc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#4 0x0000002a95577407 in boost::thread_group::join_all ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#5 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 6 (process 28140):
#0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
#1 0x00000000004b4e50 in Hypertable::ReactorRunner::operator() (
this=0x414011a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ReactorRunner.cc:59
#2 0x00000000004c97db in
boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (function_obj_ptr=@0x414011a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/boost-1_34-fix/boost/
function/function_template.hpp:158
#3 0x0000002a95577dc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#4 0x0000002a95577407 in boost::thread_group::join_all ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#5 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 5 (process 28141):
#0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
#1 0x00000000004b4e50 in Hypertable::ReactorRunner::operator() (
this=0x41e021a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ReactorRunner.cc:59
#2 0x00000000004c97db in
boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (function_obj_ptr=@0x41e021a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/boost-1_34-fix/boost/
function/function_template.hpp:158
#3 0x0000002a95577dc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#4 0x0000002a95577407 in boost::thread_group::join_all ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#5 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 4 (process 28142):
#0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
#1 0x00000000004b4e50 in Hypertable::ReactorRunner::operator() (
this=0x428031a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ReactorRunner.cc:59
#2 0x00000000004c97db in
boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (function_obj_ptr=@0x428031a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/boost-1_34-fix/boost/
function/function_template.hpp:158
#3 0x0000002a95577dc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#4 0x0000002a95577407 in boost::thread_group::join_all ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#5 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 3 (process 28143):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/tls/libpthread.so.0
#1 0x0000000000419124 in boost::condition::do_wait<boost::mutex> (
this=0x8a6090, mutex=@0x8a6068)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x00000000004190ef in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0x8a6090, lock=@0x432040d0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x00000000004bfcad in Hypertable::ConnectionManager::operator() (
this=0x432041a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ConnectionManager.cc:293
#4 0x0000000000419c45 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ConnectionManager,
void>::invoke (function_obj_ptr=@0x432041a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/boost-1_34-fix/boost/
function/function_template.hpp:158
#5 0x0000002a95577dc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95577407 in boost::thread_group::join_all ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 2 (process 28144):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/tls/libpthread.so.0
#1 0x0000000000419124 in boost::condition::do_wait<boost::mutex> (
this=0x8b9c50, mutex=@0x8b9c00)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x00000000004190ef in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0x8b9c50, lock=@0x43c050c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000418b3c in
Hypertable::ApplicationQueue::Worker::operator() (
this=0x43c051a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x0000000000418aa1 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (
function_obj_ptr=@0x43c051a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/boost-1_34-fix/boost/
function/function_template.hpp:158
#5 0x0000002a95577dc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95577407 in boost::thread_group::join_all ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 1 (process 28137):
#0 0x000000302af718e3 in memcpy () from /lib64/tls/libc.so.6
#1 0x000000000048ca75 in Hypertable::TableMutatorScatterBuffer::send
(
this=0x2a96000cd0)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/Hypertable/Lib/
TableMutatorScatterBuffer.cc:197
#2 0x000000000048a517 in Hypertable::TableMutator::flush
(this=0x8bf410)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/Hypertable/Lib/
TableMutator.cc:194
#3 0x000000000048a685 in Hypertable::TableMutator::retry
(this=0x8bf410,
timeout=30)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/Hypertable/Lib/
TableMutator.cc
#4 0x000000000040a57f in main (argc=2, argv=0x7fbffff448) at
LogLoad.cpp:181

We are using TableMutator::retry() exactly the same way as the latest
apach_log_load.cc example:
const int RETRY_TIMEOUT = 30;

try {
mutator_session_raw_ptr->flush();
} catch (Hypertable::Exception &e) {
report_mutation_error(e, mutator_session_raw_ptr);
mutator_session_raw_ptr->retry(RETRY_TIMEOUT);
do {
report_mutation_error(e,
mutator_session_raw_ptr);
} while (!mutator_session_raw_ptr-
>retry(RETRY_TIMEOUT));
}
And about the memory problem, now memory usage on the range servers is
about 30-40%, after insert our first batch of data, but the insertion
speed become much slower. Now we'll continue to load the second batch
and see what happens.

Donald

> ...
>
> 阅读更多 »

[Doug Judd]

Thanks, Donald.  If you can send me those stack traces it will save me a lot of work.  I look forward to hearing back from you.

- Doug

[Doug Judd]

Perfect.  Thanks!

- Doug

[donald]

Hi, I've slightly modified the flush&retry code to output a message

try {
mutator_session_raw_ptr->flush();
} catch (Hypertable::Exception &e) {

do {
cerr << "while flushing raw, " ;

report_mutation_error(e,
mutator_session_raw_ptr);
} while (!mutator_session_raw_ptr-
>retry(RETRY_TIMEOUT));
}

Having looked at the output, I find that the program usually core
dumps after printing "while flushing raw, ..." for the second time,
although sometimes it just goes into a dead loop.
So I guess I should only retry() once instead of using a while loop?

Donald

On 6月5日, 上午11时04分, "Doug Judd" <d...@zvents.com> wrote:
> Perfect. Thanks!
>
> - Doug
>

> ...
>
> 阅读更多 >>

[donald]

Hi Doug,

While I was debugging LogLoad with gdb this afternoon, I accidently
made Hyperspace die:

$ tail -n 20 Hyperspace.log
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/ServerConnectionHandler.cc:54) Event:
type=MESSAGE protocol=hyperspace id=16 gid=8215 headerLen=16
totalLen=26 from=10.65.25.146:33546
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/Master.cc:634) close(session=1204,
handle=8215)
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/ServerConnectionHandler.cc:54) Event:
type=MESSAGE protocol=hyperspace id=17 gid=1618 headerLen=16
totalLen=49 from=10.65.25.146:33546
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/Master.cc:434) open(sessionId=1204, fname=/
hypertable/root, flags=0x1, eventMask=0x1)
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/ServerConnectionHandler.cc:54) Event:
type=MESSAGE protocol=hyperspace id=18 gid=2425 headerLen=16
totalLen=60 from=10.65.25.146:33546
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/Master.cc:434) open(sessionId=1204, fname=/
hypertable/tables/METADATA, flags=0x1, eventMask=0x0)
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/ServerConnectionHandler.cc:54) Event:
type=MESSAGE protocol=hyperspace id=19 gid=8217 headerLen=16
totalLen=35 from=10.65.25.146:33546
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/Master.cc:713) attrget(session=1204,
handle=8217, name=schema)
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/ServerConnectionHandler.cc:54) Event:
type=MESSAGE protocol=hyperspace id=20 gid=8217 headerLen=16
totalLen=26 from=10.65.25.146:33546
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/Master.cc:634) close(session=1204,
handle=8217)
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/ServerConnectionHandler.cc:54) Event:
type=MESSAGE protocol=hyperspace id=21 gid=8213 headerLen=16
totalLen=37 from=10.65.25.146:33546
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/Master.cc:713) attrget(session=1204,
handle=8213, name=location)
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/ServerConnectionHandler.cc:54) Event:
type=MESSAGE protocol=hyperspace id=24 gid=8216 headerLen=16
totalLen=37 from=10.65.25.146:33546
1212659809 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/Master.cc:713) attrget(session=1204,
handle=8216, name=location)
1212659850 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/Master.cc:248) Expiring session 1204
1212659850 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/Master.cc:258) Destroying handle 8211
1212659850 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/Master.cc:258) Destroying handle 8213
1212659850 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/Master.cc:258) Destroying handle 8216
1212659971 INFO Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/Hyperspace/ServerKeepaliveHandler.cc:101) Session handle
1204 expired
1212659971 FATAL Hyperspace.Master : (/home/work/hypertable-0.9.0.5-
alpha/src/cc/AsyncComm/CommBuf.h:125) failed expectation: (data_ptr -
data.base) == (int)data.size

And the backtrace:
Thread 26 (process 18107):
#0 0x000000302b806ffb in pthread_join () from /lib64/tls/
libpthread.so.0
#1 0x0000002a957c11be in boost::thread::join () from /usr/local/lib/
libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#2 0x0000002a957c1336 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#3 0x00000000004867ac in Hypertable::ApplicationQueue::join
(this=0x5ee5f0) at /home/work/hypertable-0.9.0.5-alpha/src/cc/
AsyncComm/ApplicationQueue.h:176
#4 0x0000000000486395 in main (argc=3, argv=0x7fbffff8c8) at /home/
work/hypertable-0.9.0.5-alpha/src/cc/Hyperspace/main.cc:186

Thread 25 (process 18109):

#0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6

#1 0x000000000048e730 in Hypertable::ReactorRunner::operator()
(this=0x40a001a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ReactorRunner.cc:59
#2 0x00000000004a2ca7 in

boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (function_obj_ptr=@0x40a001a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/boost-1_34-fix/boost/
function/function_template.hpp:158

#3 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#4 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/

local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#5 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 24 (process 18110):

#0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6

#1 0x000000000048e730 in Hypertable::ReactorRunner::operator()
(this=0x414011a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ReactorRunner.cc:59
#2 0x00000000004a2ca7 in

boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (function_obj_ptr=@0x414011a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/boost-1_34-fix/boost/
function/function_template.hpp:158

#3 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#4 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/

local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#5 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 23 (process 18111):

#0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6

#1 0x000000000048e730 in Hypertable::ReactorRunner::operator()
(this=0x41e021a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ReactorRunner.cc:59
#2 0x00000000004a2ca7 in

boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (function_obj_ptr=@0x41e021a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/boost-1_34-fix/boost/
function/function_template.hpp:158

#3 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#4 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/

local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#5 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 22 (process 18113):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ec1c0, mutex=@0x5ec198)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ec1c0, lock=@0x432040d0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000499605 in Hypertable::ConnectionManager::operator()
(this=0x432041a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ConnectionManager.cc:293
#4 0x000000000048976f in

boost::detail::function::void_function_obj_invoker0<Hypertable::ConnectionManager,
void>::invoke (function_obj_ptr=@0x432041a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/boost-1_34-fix/boost/
function/function_template.hpp:158

#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 21 (process 18114):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x43c050c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in

Hypertable::ApplicationQueue::Worker::operator() (this=0x43c051a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97

#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (
function_obj_ptr=@0x43c051a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158

#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 20 (process 18115):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x446060c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x446061a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x446061a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 19 (process 18116):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x450070c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x450071a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x450071a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 18 (process 18117):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x45a080c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x45a081a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x45a081a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 17 (process 18118):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x464090c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x464091a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x464091a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 16 (process 18119):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x46e0a0c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x46e0a1a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x46e0a1a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 15 (process 18120):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4780b0c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4780b1a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4780b1a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 14 (process 18121):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4820c0c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4820c1a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4820c1a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 13 (process 18122):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x48c0d0c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x48c0d1a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x48c0d1a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 12 (process 18123):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4960e0c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4960e1a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4960e1a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 11 (process 18124):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4a00f0c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4a00f1a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4a00f1a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 10 (process 18125):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4aa100c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4aa101a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4aa101a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 9 (process 18126):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4b4110c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4b4111a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4b4111a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 8 (process 18127):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4be120c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4be121a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4be121a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 7 (process 18128):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4c8130c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4c8131a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4c8131a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 6 (process 18129):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4d2140c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4d2141a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4d2141a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 5 (process 18130):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4dc150c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4dc151a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4dc151a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 4 (process 18131):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4e6160c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4e6161a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4e6161a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 3 (process 18132):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4f0170c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4f0171a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4f0171a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 2 (process 18133):

#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from /
lib64/tls/libpthread.so.0

#1 0x00000000004702c6 in boost::condition::do_wait<boost::mutex>
(this=0x5ee6b0, mutex=@0x5ee660)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000046e759 in

boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>

> (this=0x5ee6b0, lock=@0x4fa180c0)
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000488806 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4fa181a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ApplicationQueue.h:97
#4 0x000000000048876b in

boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke (

function_obj_ptr=@0x4fa181a8) at /home/work/hypertable-0.9.0.5-
alpha/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#6 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#7 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 1 (process 18112):
#0 0x0000000000491a7b in Hypertable::CommBuf::reset_data_pointers
(this=0x2a95c02090) at /home/work/hypertable-0.9.0.5-alpha/src/cc/
AsyncComm/CommBuf.h:125
#1 0x0000000000490091 in Hypertable::Comm::send_datagram
(this=0x5ec140, addr=@0x2a95c02800, send_addr=@0x5edea0,
cbuf_ptr=@0x428022c0)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/Comm.cc:
305
#2 0x0000000000484b40 in Hyperspace::ServerKeepaliveHandler::handle
(this=0x5ede80, eventPtr=@0x42802350)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/Hyperspace/
ServerKeepaliveHandler.cc:103
#3 0x000000000049c9a0 in Hypertable::IOHandler::deliver_event
(this=0x5f0960, event=0x2a95c027f0)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
IOHandler.h:80
#4 0x000000000049eb89 in Hypertable::IOHandlerDatagram::handle_event
(this=0x5f0960, event=0x42802480)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
IOHandlerDatagram.cc:76
#5 0x000000000048e823 in Hypertable::ReactorRunner::operator()
(this=0x428031a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/AsyncComm/
ReactorRunner.cc:65
#6 0x00000000004a2ca7 in

boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (function_obj_ptr=@0x428031a8)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/boost-1_34-fix/boost/
function/function_template.hpp:158

#7 0x0000002a957c1dc7 in boost::function0<void,

std::allocator<boost::function_base> >::operator() ()
from /usr/local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1

#8 0x0000002a957c1407 in boost::thread_group::join_all () from /usr/
local/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#9 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#10 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#11 0x0000000000000000 in ?? ()

Any idea why this happens?

Donald

> ...
>
> 阅读更多 >>

[Doug Judd]

Hi Donald,

Unfortunately I haven't been able to reproduce this problem.  I've done some cleanup in the TableMutator retry logic, which may fix the problem.  Before you pull the changes, however, can you inspect some of the variables in the core file?  In particular, can you run the following gdb commands and send me the output:

(gdb) p this
(gdb) p ptr
(gdb) p i
(gdb) p kvec[i].ptr
(gdb) p bs.ptr
(gdb) p *send_buffer_ptr.p_

That would be a big help.  After you do that, could you pull the most recent changes that I made from the main repository and try your test again?  This page gives instructions for how to pull the latest code from the public repository:  http://code.google.com/p/hypertable/wiki/SourceCode?tm=4  The latest log message (run 'git log') should look like:

    Cleaned up TableMutator retry logic; update release number to 0.9.0.6

The new source code will install the binaries under the 0.9.0.6/ install directory.  Be sure to blow away your old install on all machines with something like:

rm -rf ~/hypertable/0.9.0.5

to ensure that you're not inadvertently executing the wrong binaries.  Thanks a lot for your help on this, it is really helping to make this code base rock solid.

Now, I'm on to the other problem that you reported ...

- Doug

On Wed, Jun 4, 2008 at 7:16 PM, donald <donal...@gmail.com> wrote:

[Doug Judd]

Hi Donald,

I tracked this one down.  The problem was detected by a recent assert that was added to the code.  It only occurred during a Hyperspace session expiration.  I've pushed out a fix for this one to the git repository.  Thanks.

- Doug

[donald]

Hi Doug,

Here's the gdb output you wanted:

(gdb) p this
No symbol "this" in current context.
(gdb) bt
#0 0x000000302af718e6 in memcpy () from /lib64/tls/libc.so.6

#1 0x000000000048ca75 in Hypertable::TableMutatorScatterBuffer::send
(

this=0x8d8820)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/Hypertable/Lib/
TableMutatorSca

tterBuffer.cc:197
#2 0x000000000048a517 in Hypertable::TableMutator::flush
(this=0x8bf410)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/Hypertable/Lib/
TableMutator.cc :
194
#3 0x000000000048a685 in Hypertable::TableMutator::retry
(this=0x8bf410,
timeout=30)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/Hypertable/Lib/

TableMutator.cc :
227

#4 0x000000000040a57f in main (argc=2, argv=0x7fbffff448) at
LogLoad.cpp:181

(gdb) frame 1

#1 0x000000000048ca75 in Hypertable::TableMutatorScatterBuffer::send
(

this=0x8d8820)
at /home/work/hypertable-0.9.0.5-alpha/src/cc/Hypertable/Lib/
TableMutatorScatterBuffer.cc:197
197 memcpy(ptr, kvec[i].ptr, bs.ptr - kvec[i].ptr);
(gdb) p this
$1 = (class Hypertable::TableMutatorScatterBuffer * const) 0x8d8820
(gdb) p ptr
$2 = (
uint8_t *) 0x2a96454ff7 "\034tools.go" <Address 0x2a96455000 out
of bounds>
(gdb) p i
$3 = 7373

(gdb) p kvec[i].ptr

You can't do that without a process to debug.
(gdb) p bs.ptr
$4 = (uint8_t *) 0x2a96970450 "\031www.qq163.com"
(gdb) p *send_buffer_ptr.p_
$5 = {<Hypertable::ReferenceCount> =
{<boost::noncopyable_::noncopyable> = {<No data fields>},
_vptr.ReferenceCount = 0x4e3810, refCount = {counter = 2}},
key_offsets = {<std::_Vector_base<uint64_t,std::allocator<uint64_t>
>> = {
_M_impl = {<std::allocator<uint64_t>> =
{<__gnu_cxx::new_allocator<uint64_t>> = {<No data fields>}, <No data
fields>}, _M_start = 0x2a96851010,
_M_finish = 0x2a96879f90,
_M_end_of_storage = 0x2a96891010}}, <No data fields>}, accum =
{
base = 0x2a96892010 "\035user.qzone.qq.com",
ptr = 0x2a96892010 "\035user.qzone.qq.com", size = 1326180, own =
true},
pending_updates = {base = 0x2a96404970 "\0313w.cga.com.cn", size =
0,
own = true}, addr = {sin_family = 2, sin_port = 44180, sin_addr =
{
s_addr = 2518237450}, sin_zero =
"\000\000\000\000\000\000\000"},
counterp = 0x8d88c0, dispatch_handler_ptr = {p_ = 0x2a964078e0},
failed_regions =
{<std::_Vector_base<Hypertable::FailedRegionT,std::allocator<Hypertable::FailedRegionT>
>> = {
_M_impl = {<std::allocator<Hypertable::FailedRegionT>> =
{<__gnu_cxx::new_allocator<Hypertable::FailedRegionT>> = {<No data
fields>}, <No data fields>},
_M_start = 0x8cd500, _M_finish = 0x8cd518,
_M_end_of_storage = 0x8cd518}}, <No data fields>},
m_table_identifier = 0x8d8880, m_range_locator = 0x8c0250}

Thanks.

Donald

On 6月6日, 上午1时23分, "Doug Judd" <d...@zvents.com> wrote:
> Hi Donald,
>

> ...
>
> 阅读更多 >>

[Doug Judd]

Hi Donald,

The changes that I pushed out earlier today should take care of this problem (as well as the Hyperspace problem).  When you get a chance, if you could re-run your test with the latest code and report back, I'd appreciate it.  Thanks!

- Doug

[donald]

Hi Doug,

Thanks very much. I'll try it next week.

BTW, after tuning Hypertable.RangeServer.AccessGroup.MaxMemory to 5M,
we've successfully inserted about 400MB data into 3 range servers,
despite the failed ones (because of range server row overflow). Now
the range servers each takes about 60% memory, and seems to work
pretty stable. The insertion speed did drop by 10%-20%, but we think
that's tolerable.

Thanks again for your great support!

Donald

On 6月6日, 下午12时05分, "Doug Judd" <d...@zvents.com> wrote:
> Hi Donald,
>

> The changes that I pushed out earlier today should take care of this problem
> (as well as the Hyperspace problem). When you get a chance, if you could
> re-run your test with the latest code and report back, I'd appreciate it.
> Thanks!
>
> - Doug
>

> On Thu, Jun 5, 2008 at 6:41 PM, donald <donaldl...@gmail.com> wrote:
>
> > Hi Doug,
>

> ...
>
> 阅读更多 >>