Failed to load data from the table - ThriftClient throws exception

[Evgeny Pasynkov]

Hi,

I'm trying to iterate my table and get the following problem:

org.apache.thrift.transport.TTransportException: Frame size (336913670) larger than max length (16384000)!

at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)

at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)

at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)

at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)

at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)

at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)

at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)

at org.hypertable.thriftgen.ClientService$Client.recv_scanner_get_cells_serialized(ClientService.java:2286)

at org.hypertable.thriftgen.ClientService$Client.scanner_get_cells_serialized(ClientService.java:2273)

at HtTests.main(HtTests.java:30)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

Database logs have no errors.

This exception also occurs when I'm trying to iterate some rows inside that table.

Snippet:

final ThriftClient client = ThriftClient.create("host", 15867);

final long ns = client.open_namespace("namespacename");

final ScanSpec scanSpec = new ScanSpec();

final long scanner = client.scanner_open(ns, "tablename", scanSpec);

while (true) {

  final ByteBuffer serializedCells = client.scanner_get_cells_serialized(scanner);

  final SerializedCellsReader reader = new SerializedCellsReader(serializedCells);

  while (reader.next()) {

    /* do some stuff */

  }

  if (reader.eos())

    break;

}

client.scanner_close(scanner);

client.close();

What's wrong and how can I handle this problem?

I could try to reduce db size to reasonable values (by dropping all other tables) and provide you with the db snapshot, if becessary

[Doug Judd]

The problem will go away if you call the client factory method that accepts a frame size and supply it a frame size that is large enough.  For example, in the above case, the frame size needs to be at least 336,913,670, so in the call below I've passed it 500M.

final ThriftClient client = ThriftClient.create("host", 15867, 1600000, true, 500000000);

- Doug

--
You received this message because you are subscribed to the Google Groups "Hypertable User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypertable-us...@googlegroups.com.
To post to this group, send email to hyperta...@googlegroups.com.
Visit this group at http://groups.google.com/group/hypertable-user.
For more options, visit https://groups.google.com/d/optout.

--

Doug Judd

CEO, Hypertable Inc.

[Evgeny Pasynkov]

Hi Doug,

Thank you, modifying the client constructor I was able to read the table.

Occasionally, one cell contains 300Mb data.

The question is: Is the frame size relates only to *receiving* data from HT? To write data I was using constructor with default frame size, and it was able to write such HUGE cell. Is it correct?

Thank you