failed expectation: strcmp(m_split_row.c_str(), m_metalog_entity->spec.end_row) < 0 && strcmp(m_split_row.c_str(), m_metalog_entity->spec.start_row) > 0

[David]

A RangeServer crashed suddenly in the Hypertable 0.9.7.5 cluster,  the log report the expectation as title.  Meanwhile, a core file was made, use gdb to trace it, get the following ouput:

(gdb) bt

#0  0x00000030b340f38b in raise () from /lib64/libpthread.so.0

#1  0x00000000006a4146 in Hypertable::Range::split_install_log() ()

#2  0x00000000006a55b5 in Hypertable::Range::split() ()

#3  0x0000000000586364 in Hypertable::MaintenanceQueue::Worker::operator()() ()

#4  0x00007f040a5cdf3d in thread_proxy () from /dinglicom/hypertable/current/lib/libboost_thread.so.1.44.0

#5  0x00000030b34077e1 in start_thread () from /lib64/libpthread.so.0

#6  0x00000030b30e153d in clone () from /lib64/libc.so.6

[Doug Judd]

Thanks for the report.  I'd like you to run our integrity checker on your database.  I'll send you an e-mail off-list with instructions for how to do that.

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Doug Judd

CEO, Hypertable Inc.

[David]

/opt/hypertable/0.9.7.5.com/bin/ht htck check /home/cloudil/metadata.tsv

METADATA entries: 6116

1369111367 WARN htck : verify_backup (/root/src/hypertable-commercial/src/cc/Hypertable/Lib/MetaLogReader.cc:124): MetaLog backup file '/opt/hypertable/0.9.7.5.com/run/log_backup/rsml/rs1/25' doesnt exist

1369111368 ERROR htck : main (/root/src/hypertable-commercial/src/cc/Tools/htck/htck.cc:918): Hypertable::Exception: Error getting length of DFS file: /hypertable/servers/rs1/log/rsml/25 - DFS BROKER i/o error

        at virtual int64_t Hypertable::DfsBroker::Client::length(const Hypertable::String&, bool) (/root/src/hypertable-commercial/src/cc/DfsBroker/Lib/Client.cc:451)

        at virtual int64_t Hypertable::DfsBroker::Client::length(const Hypertable::String&, bool) (/root/src/hypertable-commercial/src/cc/DfsBroker/Lib/Client.cc:445): java.io.IOException: Cannot obtain block length for LocatedBlock{BP-182269171-172.16.40.121-1366352314110:blk_-8023451511163314470_2718680; getBlockSi

[David]

[Doug Judd]

Is it possible that HDFS is having problems?  Are all the DataNodes up and does hadoop fsck /hypertable run clean?

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[David]

All datanodes are up, but maybe HDFS has problem. When executing the 'hadoop fsck /hypertable' on every datanode, get the following similar ouput:

.....Status: CORRUPT

 Total size:    1465571625672 B

 Total dirs:    168992

 Total files:   146814 (Files currently being written: 20)

 Total blocks (validated):      151232 (avg. block size 9690883 B) (Total open file blocks (not validated): 20)

  ********************************

  CORRUPT FILES:        65

  MISSING BLOCKS:       13

  MISSING SIZE:         11055104 B

  CORRUPT BLOCKS:       65

  ********************************

 Minimally replicated blocks:   151219 (99.9914 %)

 Over-replicated blocks:        0 (0.0 %)

 Under-replicated blocks:       8049 (5.322286 %)

 Mis-replicated blocks:         0 (0.0 %)

 Default replication factor:    2

 Average block replication:     2.9394772

 Corrupt blocks:                65

 Missing replicas:              8085 (1.7860812 %)

 Number of data-nodes:          6

 Number of racks:               1

FSCK ended at Tue May 21 13:24:17 CST 2013 in 6760 milliseconds

The filesystem under path '/hypertable' is CORRUPT

Does the problem of HDFS cause the expectation?

[David]

[David]

I inspect the HDFS file system via browsing namenode:50070 monitor page, find that some files have corrupted all blocks of all replications, so i sure that HDFS has a trouble.  can the trouble trigger the assert?

[David]

[Doug Judd]

It's possible that the HDFS problems triggered the assert.  It really depends on how the corruption manifests itself.  Were there any other errors in the log prior to the failed expectation message?

- Doug

--

You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[David]

No other errors.
A question: can the operators  change from "<" ">" to "<=" ">=" in the assert?  i.e. "strcmp(m_split_row.c_str(), m_metalog_entity->spec.end_row) <= 0 && strcmp(m_split_row.c_str(), m_metalog_entity->spec.start_row) => 0".  If there exists a row which contains so multiple timestamp that its data is over a range, of course, it's an extreme case.

By the way, there is a word "ScanSpec column predicate validation fix" in the release notes of the 0.9.7.5 version, what is it mean?

[David]

[Doug Judd]

It's unlikely, that's why my first thought was metadata corruption.  As I mentioned before, there was a bug that was recently fixed that could lead to metadata corruption.

- Doug

--

You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[David]

[David]

[Doug Judd]

It was a fix to validate the query predicate.  This commit doesn't have anything to do with the error you're seeing.

- Doug

On Tue, May 21, 2013 at 10:02 PM, David <cwh...@gmail.com> wrote:

By the way, there is a word "ScanSpec column predicate validation fix" in the release notes of the 0.9.7.5 version, what is it mean?

--

You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.