Frame size (25168883) larger than max length (16384000)

[Mohammed Azad]

Dear All,

I am hitting a weird exception when listing large set of parquet files... i have around 89000 parquet files in multiple directories.. and this is the exception i get.

logs/user.log:2016-02-29 16:40:24,287 ERROR USER_LOGGER (FileSystemMasterClient.java:getFileInfoList) - Frame size (25168883) larger than max length (16384000)!

logs/user.log:org.apache.thrift.transport.TTransportException: Frame size (25168883) larger than max length (16384000)!

logs/user.log:2016-02-29 16:40:25,209 ERROR USER_LOGGER (FileSystemMasterClient.java:getFileInfoList) - Frame size (25168883) larger than max length (16384000)!

logs/user.log:org.apache.thrift.transport.TTransportException: Frame size (25168883) larger than max length (16384000)!

logs/user.log:2016-02-29 16:40:26,134 ERROR USER_LOGGER (FileSystemMasterClient.java:getFileInfoList) - Frame size (25168883) larger than max length (16384000)!

logs/user.log:org.apache.thrift.transport.TTransportException: Frame size (25168883) larger than max length (16384000)!

logs/user.log:2016-02-29 16:40:27,059 ERROR USER_LOGGER (FileSystemMasterClient.java:getFileInfoList) - Frame size (25168883) larger than max length (16384000)!

logs/user.log:org.apache.thrift.transport.TTransportException: Frame size (25168883) larger than max length (16384000)!

logs/user.log:2016-02-29 16:40:27,984 ERROR USER_LOGGER (FileSystemMasterClient.java:getFileInfoList) - Frame size (25168883) larger than max length (16384000)!

logs/user.log:org.apache.thrift.transport.TTransportException: Frame size (25168883) larger than max length (16384000)!

logs/user.log:2016-02-29 16:40:28,913 ERROR USER_LOGGER (FileSystemMasterClient.java:getFileInfoList) - Frame size (25168883) larger than max length (16384000)!

logs/user.log:org.apache.thrift.transport.TTransportException: Frame size (25168883) larger than max length (16384000)!

logs/user.log:2016-02-29 16:40:29,843 ERROR USER_LOGGER (FileSystemMasterClient.java:getFileInfoList) - Frame size (25168883) larger than max length (16384000)!

logs/user.log:org.apache.thrift.transport.TTransportException: Frame size (25168883) larger than max length (16384000)!

logs/user.log:2016-02-29 16:40:30,754 ERROR USER_LOGGER (FileSystemMasterClient.java:getFileInfoList) - Frame size (25168883) larger than max length (16384000)!

logs/user.log:org.apache.thrift.transport.TTransportException: Frame size (25168883) larger than max length (16384000)!

any idea how to increase the Frame Size in tachyon?..

Thanks

Azad

[Jiří Šimša]

You are likely using web port (19999), instead of using data port (19998).

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pengfei Xuan]

Also, for some reason, Thrift v0.9.3 changes the default size of largest allowable frame from 64MB to 16MB. It could become an issue in current implementation. I'm going to submit a PR to address this changes.   

--

Pengfei Xuan

School of Computing
McAdams Hall 202A
Clemson University
Clemson SC  29634-0974

[Mohammed Azad]

Nope.. i double checked, its not the port number.. i am using the 19998... as mentioned by Xuan, something wrong with default frame size... wonder whether i can configure some parameter to get it increased.. or should i recompile thrift for this to work?. please advice.

Thanks

Azad

[Pengfei Xuan]

Hi Mohammed,

Your requested feature has been merged into Alluxio master branch. The default frame size is set to 64MB, which should be a good size for your case. If you still want to adjust this value, you can override the parameter alluxio.keyvalue.partition.size.bytes.max. 

Best,
Pengfei 

[Bin Fan]

Hi Mohanmmed,

Could you provide a little bit more information for this problem?

e.g., which Alluxio or (Tachyon) version are you using?

what application do you run on top of Alluxio (or Tachyon) when you see this error?

I am asking to try to reproduce this problem

Thanks

- Bin

[Zaicheng Wang]

Hi,
I see the default size for alluxio.keyvalue.partition.

size.bytes.max is 512MB which is larger than 16MB or 64MB. Is this the right parameter to change?

Thanks

在 2016年3月1日星期二 UTC-8上午8:59:16，Pengfei Xuan写道：

[Calvin Jia]

Hi,

The key-value partition size parameter is not related to the thrift frame size. Are you seeing/trying to resolve frame size issues?

Thanks,

Calvin

[Zaicheng Wang]

Yes, I am trying to increase the frame size. I saw a commit solved this https://github.com/Alluxio/alluxio/commit/e5e0eeb40bc5b7d1647175bb2ef9a4c3d7535ca0 but been reverted.
Is there anyway I can do this?

Thanks

在 2016年4月8日星期五 UTC-7下午12:14:47，Calvin Jia写道：

[Calvin Jia]

Hi,

Is there a particular reason you want to increase the frame size? Or are you just getting errors about the frame size?

If you are just getting errors, are you using the secure transport in Alluxio?

Thanks,

Calvin

[Carl H]

Hi, I am wondering if you guys have solved the issue.

I have met the same question. While after I alter the property 'alluxio.network.thrift.frame.size.bytes.max' to 128M, the error is still occurs.

16/07/22 17:46:44 ERROR type: Frame size (791752241) larger than max length (16777216)!

alluxio.org.apache.thrift.transport.TTransportException: Frame size (791752241) larger than max length (16777216)!

But the file size I want to process is only 23k.


在 2016年4月12日星期二 UTC+8上午8:33:56，Calvin Jia写道：

[Gene Pang]

Hi,

This error typically happens if the incorrect port number is used. Could you verify you are using the correct port?

Thanks,

Gene

[Carl H]

val a = sc.textFile("alluxio://10.175.128.28:19998/hdfs_data/nohup.out")

This is what I defined, the IP:port is the master info. Is it correct?

在 2016年7月22日星期五 UTC+8下午9:14:23，Gene Pang写道：

[Gene Pang]

Hi,

Yes, the default rpc port is 19998.

Could you restart your spark-shell and run the commands with the correct port?

Thanks,

Gene

[Chanh Le]

Why don't put a catch or message that user can understand easier?

[Gene Pang]

Were you able to resolve the frame size issue by trying again with the correct port?

I agree that that the error message is difficult to understand. It should be improved, but I think the challenge is that a ThriftException is thrown, and not all ThriftExceptions are attributed to the incorrect port number.

Thanks,

Gene

[Chanh Le]

Yes,

In the first time I use Alluxio I have experienced with that already. It’s OK when I change the port but It also takes me a lot time to really understand because it really weird.

Hope you guys can fix it to make Alluxio greater.

 

--
You received this message because you are subscribed to a topic in the Google Groups "Alluxio Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/alluxio-users/CZc1mhWmAZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to alluxio-user...@googlegroups.com.

[Gene Pang]

Thanks for the confirmation! Yes, the messages should be updated to help clarify this issue.

-Gene