java hql_query failed: out of sequence response

[Azhar Jassal]

Hi,

I'm using the Java client library to query Hypertable.

The most common exception I get in my program is caused by "out of sequence" responses. I'm not sure about how to correct this as the applications are not threaded and there is just a single process making SELECT queries. There is no pattern to when I get this error, it happens randomly, sometimes long periods apart.

My application is just a single thread, it is looping through a preloaded set of row keys (120million). In each loop it's querying Hypertable (SELECT) for all cells with that ROW ID (maximum result will be 200 cells). There is no sleep between the loops. How does the response become out of sequence?

Stacktrace below:

org.apache.thrift.TApplicationException: hql_query failed: out of sequence response

        at org.hypertable.thriftgen.HqlService$Client.recv_hql_query(HqlService.java:191)

        at org.hypertable.thriftgen.HqlService$Client.hql_query(HqlService.java:168)

        at dataindexer.IndexQueue.hypertableObject(IndexQueue.java:44)

        at dataindexer.IndexQueue.processUniques(IndexQueue.java:114)

        at dataindexer.IndexQueue.doIndex(IndexQueue.java:130)

        at dataindexer.Main.main(Main.java:11)

Any tips?

[Doug Judd]

What version of Hypertable are you running and what system (e.g. OS, kernel  version, architecture)?

Are there any errors in the ThriftBroker.log file?

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To post to this group, send email to hyperta...@googlegroups.com.
To unsubscribe from this group, send email to hypertable-us...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/hypertable-user?hl=en.

[Azhar Jassal]

Hi,

Hypertable version 0.9.4.3

OS: Ubuntu 10.04 LTS (installed with deb file)

Only a single machine (using local file system storage)

(using default config)

It's not a system resource/ swapping issue because while the program is half way through the machine has about 1GB free memory...

No errors in ThriftBroker.log, I've just run it again and got the following stacktrace 1,300 seconds into its run. One of the SELECT's must of failed but the exception was caught and on the next loop (still same open connection) everything continued as normal... There is no pattern I've been able to identify with these errors. What is the common cause of this exception?

I am doing approx 1000 queries (SELECT by ROW ID) a second... As I got this at 1300 seconds, only 1 SELECT failed out of 1000x13000 = 1,300,000

org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out

        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)

        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)

        at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)

        at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)

        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)

        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)

        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)

        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)

        at org.hypertable.thriftgen.HqlService$Client.recv_hql_query(HqlService.java:184)

        at org.hypertable.thriftgen.HqlService$Client.hql_query(HqlService.java:168)

        at dataindexer.IndexQueue.hypertableObject(IndexQueue.java:42)

        at dataindexer.IndexQueue.processUniques(IndexQueue.java:107)

        at dataindexer.IndexQueue.doIndex(IndexQueue.java:122)

        at dataindexer.Main.main(Main.java:11)

Caused by: java.net.SocketTimeoutException: Read timed out

        at java.net.SocketInputStream.socketRead0(Native Method)

        at java.net.SocketInputStream.read(SocketInputStream.java:146)

        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)

        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)

        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)

        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)

        ... 13 more

org.apache.thrift.TApplicationException: hql_query failed: out of sequence response

        at org.hypertable.thriftgen.HqlService$Client.recv_hql_query(HqlService.java:191)

        at org.hypertable.thriftgen.HqlService$Client.hql_query(HqlService.java:168)

        at dataindexer.IndexQueue.hypertableObject(IndexQueue.java:42)

        at dataindexer.IndexQueue.processUniques(IndexQueue.java:107)

        at dataindexer.IndexQueue.doIndex(IndexQueue.java:122)

        at dataindexer.Main.main(Main.java:11)

[Doug Judd]

Hi Azhar,

The "out of sequence response" exception is a Thrift exception.  I suspect it is a result of the "java.net.SocketTimeoutException: Read timed out" exception.  This looks suspiciously like the Ubuntu hang problem.

Is there any way you could try your application out on a different system?  I'm happy to set up CentOS 5 machine in EC2 and give you access to run your test.

- Doug

[Azhar Jassal]

Hi,

I've left some tests running to try and get get a better understanding of what's going on just before I try another platform... Sometimes in ThriftBroker.log, I see things like:

cells.size=cells.size=34138

cells.size=cells.size=7386

whereas the rest of the statements just have a single cells.size, e.g.

cells.size=317

What does the "cells.size=cells.size" mean?