org.apache.thrift.transport.TTransportException: Frame size (24719069) larger than max length (16777

[cavi...@gmail.com]

Hi,

We encounter an error when listing file status via the filesystem API indicating the thrift frame size is larger than the max length (16MB). This seems to occur only when the number of files exceeds about 50000. I have included below the stack trace:

[ERROR] 2016-08-29 18:51:46,635 alluxio.logger.type Thread ID : pool-1-thread-7 12889 - Frame size (24719069) larger than max length (16777216)!
org.apache.thrift.transport.TTransportException: Frame size (24719069) larger than max length (16777216)!
    at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
    at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
    at org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
    at alluxio.thrift.FileSystemMasterClientService$Client.recv_listStatus(FileSystemMasterClientService.java:503)
    at alluxio.thrift.FileSystemMasterClientService$Client.listStatus(FileSystemMasterClientService.java:489)
    at alluxio.client.file.FileSystemMasterClient$8.call(FileSystemMasterClient.java:220)
    at alluxio.client.file.FileSystemMasterClient$8.call(FileSystemMasterClient.java:216)
    at alluxio.AbstractClient.retryRPC(AbstractClient.java:324)
    at alluxio.client.file.FileSystemMasterClient.listStatus(FileSystemMasterClient.java:216)
    at alluxio.client.file.BaseFileSystem.listStatus(BaseFileSystem.java:195)

We use Alluxio 1.2 and currently use block size of 8MB and worker size of 64GB.

Thanks.

[Jiří Šimša]

Hi there,

This error can occur when the transport between client and server is mismatched (e.g. using incorrect service port or different client and server configuration). However, since things seem to be working correctly for you for smaller number of files, I suspect the issue is in fact that to the amount of information the list status call sending back exceeds the Thrift frame size.

You should be able to solve this problem by changing the maximum Thrift frame size Alluxio uses through the alluxio.network.thrift.frame.​size.bytes.max configuration property to say 64MB. I recommend adding this setting to your alluxio-site.properties file on both the client-side and the server-side.

Best,

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jiří Šimša

Alluxio, Inc.

[Jiří Šimša]

Hi there,

Were you able to resolve your issue?

Best,