Do I need switch to FT mode?

69 views
Skip to first unread message

Chanh Le

unread,
Jul 19, 2016, 3:27:03 AM7/19/16
to Alluxio Users

We have 5 nodes Alluxio and total capacity is 90GB still free a lot.


I running a summary job for daily report from hourly data already store in Alluxio.
But when I run it stuck very long time and throw error


16/07/19 14:12:32 INFO type: open(alluxio://master1:19998/AD_COOKIE_REPORT/time=2016-07-17-10/network_id=14310/part-r-00000-48fd1057-1e30-4554-b4fa-fedc8e392c66.snappy.parquet, 65536)
16/07/19 14:12:32 INFO ParquetRelation$$anonfun$buildInternalScan$1$$anon$1: Input split: ParquetInputSplit{part: alluxio://master1:19998/AD_COOKIE_REPORT/time=2016-07-17-19/network_id=22100/part-r-00000-7a60e53b-f84c-466d-9f19-3688c3136ea6.snappy.parquet start: 0 end: 38545099 length: 38545099 hosts: []}
16/07/19 14:12:32 INFO type: getFileStatus(alluxio://master1:19998/AD_COOKIE_REPORT/time=2016-07-17-19/network_id=22100/part-r-00000-7a60e53b-f84c-466d-9f19-3688c3136ea6.snappy.parquet)
16/07/19 14:12:32 INFO type: open(alluxio://master1:19998/AD_COOKIE_REPORT/time=2016-07-17-19/network_id=22100/part-r-00000-7a60e53b-f84c-466d-9f19-3688c3136ea6.snappy.parquet, 65536)
16/07/19 14:12:32 INFO type: Connecting to remote worker @ master2/10.197.0.4:29998
16/07/19 14:12:32 INFO type: Connecting to remote worker @ master2/10.197.0.4:29998
16/07/19 14:12:32 INFO type: Connected to remote machine master2/10.197.0.4:29999
16/07/19 14:12:32 INFO type: Data 3757727285248 from remote machine master2/10.197.0.4:29999 received
16/07/19 14:12:32 INFO type: Connected to remote machine master2/10.197.0.4:29999
16/07/19 14:12:32 INFO type: status: WRITE_ERROR from remote machine master2/10.197.0.4:29999 received
16/07/19 14:12:32 WARN type: The block with ID 3757727285248 could not be cached into Alluxio storage.

16/07/19 14:22:32 ERROR type: java.net.SocketTimeoutException: Read timed out
alluxio.org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at alluxio.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at alluxio.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at alluxio.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at alluxio.org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
at alluxio.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at alluxio.thrift.BlockWorkerClientService$Client.recv_cancelBlock(BlockWorkerClientService.java:282)
at alluxio.thrift.BlockWorkerClientService$Client.cancelBlock(BlockWorkerClientService.java:268)
at alluxio.client.block.BlockWorkerClient$4.call(BlockWorkerClient.java:167)
at alluxio.client.block.BlockWorkerClient$4.call(BlockWorkerClient.java:164)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:327)
at alluxio.client.block.BlockWorkerClient.cancelBlock(BlockWorkerClient.java:164)
at alluxio.client.block.RemoteBlockOutStream.cancel(RemoteBlockOutStream.java:65)
at alluxio.client.file.FileInStream.closeOrCancelCacheStream(FileInStream.java:339)
at alluxio.client.file.FileInStream.handleCacheStreamIOException(FileInStream.java:397)
at alluxio.client.file.FileInStream.read(FileInStream.java:214)
at alluxio.client.file.FileInStream.readCurrentBlockToPos(FileInStream.java:617)
at alluxio.client.file.FileInStream.seekInternalWithCachingPartiallyReadBlock(FileInStream.java:562)
at alluxio.client.file.FileInStream.seek(FileInStream.java:247)
at alluxio.hadoop.HdfsFileInputStream.seek(HdfsFileInputStream.java:324)
at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:62)
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:417)
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:385)
at org.apache.spark.sql.execution.datasources.parquet.SpecificParquetRecordReaderBase.initialize(SpecificParquetRecordReaderBase.java:98)
at org.apache.spark.sql.execution.datasources.parquet.UnsafeRowParquetRecordReader.initialize(UnsafeRowParquetRecordReader.java:130)
at org.apache.spark.sql.execution.datasources.parquet.UnsafeRowParquetRecordReader.tryInitialize(UnsafeRowParquetRecordReader.java:117)
at org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.<init>(SqlNewHadoopRDD.scala:169)
at org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:126)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.CoalescedRDD$$anonfun$compute$1.apply(CoalescedRDD.scala:96)
at org.apache.spark.rdd.CoalescedRDD$$anonfun$compute$1.apply(CoalescedRDD.scala:95)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:376)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at alluxio.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 51 more
16/07/19 14:22:32 INFO type: Connecting to remote worker @ master2/10.197.0.4:29998


Chanh Le

unread,
Jul 19, 2016, 4:47:40 AM7/19/16
to Alluxio Users
2016-07-19 08:45:31,163 ERROR logger.type (BlockDataServerHandler.java:handleBlockWriteRequest) - Error writing remote block : Temp blockId 3,792,036,691,968 is not available, because it is already committed
alluxio.exception.BlockAlreadyExistsException: Temp blockId 3,792,036,691,968 is not available, because it is already committed
 at alluxio.worker.block.TieredBlockStore.checkTempBlockIdAvailable(TieredBlockStore.java:397)
 at alluxio.worker.block.TieredBlockStore.createBlockMetaInternal(TieredBlockStore.java:525)
 at alluxio.worker.block.TieredBlockStore.createBlockMeta(TieredBlockStore.java:188)
 at alluxio.worker.block.BlockWorker.createBlockRemote(BlockWorker.java:341)
 at alluxio.worker.netty.BlockDataServerHandler.handleBlockWriteRequest(BlockDataServerHandler.java:142)
 at alluxio.worker.netty.DataServerHandler.channelRead0(DataServerHandler.java:75)
 at alluxio.worker.netty.DataServerHandler.channelRead0(DataServerHandler.java:41)
 at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
 at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
 at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:831)
 at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:322)
 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254)
 at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
 at java.lang.Thread.run(Thread.java:745)


This is a log from a worker.

Chanh Le

unread,
Jul 19, 2016, 5:09:39 AM7/19/16
to Alluxio Users
It passed all the tests

[root@master1:/home/spark/alluxio-1.1.0]# ./bin/alluxio runTests
2016-07-19 16:08:27,006 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemMasterClient master @ master1/10.197.0.3:19998
2016-07-19 16:08:27,015 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemMasterClient master @ master1/10.197.0.3:19998
runTest Basic CACHE_PROMOTE MUST_CACHE
2016-07-19 16:08:27,073 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with BlockMasterClient master @ master1/10.197.0.3:19998
2016-07-19 16:08:27,076 INFO  type (AbstractClient.java:connect) - Client registered with BlockMasterClient master @ master1/10.197.0.3:19998
2016-07-19 16:08:27,098 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to local worker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,130 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_MUST_CACHE took 70 ms.
2016-07-19 16:08:27,281 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_MUST_CACHE took 151 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE MUST_CACHE
2016-07-19 16:08:27,346 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master3/10.197.0.5:29998
2016-07-19 16:08:27,395 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master3/10.197.0.5:29999
2016-07-19 16:08:27,440 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master3/10.197.0.5:29999 received
2016-07-19 16:08:27,480 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master3/10.197.0.5:29998
2016-07-19 16:08:27,484 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master3/10.197.0.5:29998
2016-07-19 16:08:27,489 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine master3/10.197.0.5:29999
2016-07-19 16:08:27,500 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3801650036736 from remote machine master3/10.197.0.5:29999 received
Passed the test!
runTest Basic CACHE_PROMOTE CACHE_THROUGH
2016-07-19 16:08:27,510 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,510 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,522 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master2/10.197.0.4:29998
2016-07-19 16:08:27,542 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master2/10.197.0.4:29999
2016-07-19 16:08:27,551 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master2/10.197.0.4:29999 received
2016-07-19 16:08:27,553 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_CACHE_THROUGH took 50 ms.
2016-07-19 16:08:27,589 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master2/10.197.0.4:29998
2016-07-19 16:08:27,593 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master2/10.197.0.4:29998
2016-07-19 16:08:27,596 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master2/10.197.0.4:29998
2016-07-19 16:08:27,597 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine master2/10.197.0.4:29999
2016-07-19 16:08:27,605 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3801666813952 from remote machine master2/10.197.0.4:29999 received
2016-07-19 16:08:27,607 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master2/10.197.0.4:29999
2016-07-19 16:08:27,615 INFO  type (NettyRemoteBlockWriter.java:write) - status: WRITE_ERROR from remote machine master2/10.197.0.4:29999 received
2016-07-19 16:08:27,627 INFO  type (FileInStream.java:closeOrCancelCacheStream) - Closing or cancelling the cache stream encountered IOExecption java.io.IOException: Error writing blockId: 3,801,666,813,952, sessionId: 5,654,815,959,648,460,859, address: master2/10.197.0.4:29999, message: Failed to write block., reading from the regular stream won't be affected.
2016-07-19 16:08:27,627 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_CACHE_THROUGH took 74 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE CACHE_THROUGH
2016-07-19 16:08:27,629 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,630 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,632 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:27,643 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:27,651 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:27,691 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:27,694 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:27,696 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:27,698 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:27,700 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3801683591168 from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:27,701 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:27,703 INFO  type (NettyRemoteBlockWriter.java:write) - status: WRITE_ERROR from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:27,704 INFO  type (FileInStream.java:closeOrCancelCacheStream) - Closing or cancelling the cache stream encountered IOExecption java.io.IOException: Error writing blockId: 3,801,683,591,168, sessionId: 2,256,656,202,840,086,824, address: master4/10.197.0.6:29999, message: Failed to write block., reading from the regular stream won't be affected.
Passed the test!
runTest Basic CACHE_PROMOTE THROUGH
2016-07-19 16:08:27,706 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,706 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,712 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_THROUGH took 7 ms.
2016-07-19 16:08:27,754 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,754 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,782 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_THROUGH took 70 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE THROUGH
2016-07-19 16:08:27,784 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,785 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,828 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,829 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:27,832 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:27,844 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:27,845 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master4/10.197.0.6:29999 received
Passed the test!
runTest Basic CACHE_PROMOTE ASYNC_THROUGH
2016-07-19 16:08:27,849 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:27,850 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:27,851 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:27,858 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_PROMOTE_ASYNC_THROUGH took 11 ms.
2016-07-19 16:08:27,895 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:27,898 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:27,900 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ slave5.dev-etl.ants.vn/10.197.0.7:29998
2016-07-19 16:08:27,902 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:27,903 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3801733922816 from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:27,905 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999
2016-07-19 16:08:27,907 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999 received
2016-07-19 16:08:27,908 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_PROMOTE_ASYNC_THROUGH took 50 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE_PROMOTE ASYNC_THROUGH
Passed the test!
runTest Basic CACHE MUST_CACHE
2016-07-19 16:08:28,022 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ slave5.dev-etl.ants.vn/10.197.0.7:29998
2016-07-19 16:08:28,023 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999
2016-07-19 16:08:28,025 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999 received
2016-07-19 16:08:28,026 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_MUST_CACHE took 6 ms.
2016-07-19 16:08:28,053 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ slave5.dev-etl.ants.vn/10.197.0.7:29998
2016-07-19 16:08:28,055 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master2/10.197.0.4:29998
2016-07-19 16:08:28,056 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999
2016-07-19 16:08:28,064 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3801767477248 from remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999 received
2016-07-19 16:08:28,066 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master2/10.197.0.4:29999
2016-07-19 16:08:28,069 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master2/10.197.0.4:29999 received
2016-07-19 16:08:28,070 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_MUST_CACHE took 43 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE MUST_CACHE
2016-07-19 16:08:28,072 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:28,074 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:28,076 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:28,107 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:28,109 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ slave5.dev-etl.ants.vn/10.197.0.7:29998
2016-07-19 16:08:28,111 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:28,112 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3801784254464 from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:28,116 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999
2016-07-19 16:08:28,117 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999 received
Passed the test!
runTest Basic CACHE CACHE_THROUGH
2016-07-19 16:08:28,120 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,120 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,123 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master3/10.197.0.5:29998
2016-07-19 16:08:28,127 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master3/10.197.0.5:29999
2016-07-19 16:08:28,128 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master3/10.197.0.5:29999 received
2016-07-19 16:08:28,130 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_CACHE_THROUGH took 11 ms.
2016-07-19 16:08:28,155 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master3/10.197.0.5:29998
2016-07-19 16:08:28,157 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:28,159 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine master3/10.197.0.5:29999
2016-07-19 16:08:28,160 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3801801031680 from remote machine master3/10.197.0.5:29999 received
2016-07-19 16:08:28,162 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:28,163 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:28,164 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_CACHE_THROUGH took 34 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE CACHE_THROUGH
2016-07-19 16:08:28,166 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,167 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
Passed the test!
runTest Basic CACHE THROUGH
2016-07-19 16:08:28,204 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,205 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,210 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_THROUGH took 7 ms.
2016-07-19 16:08:28,240 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,241 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,245 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master2/10.197.0.4:29998
2016-07-19 16:08:28,259 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master2/10.197.0.4:29999
2016-07-19 16:08:28,261 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master2/10.197.0.4:29999 received
2016-07-19 16:08:28,263 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_THROUGH took 52 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE THROUGH
2016-07-19 16:08:28,265 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,265 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,308 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,308 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,313 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:28,322 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:28,323 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master4/10.197.0.6:29999 received
Passed the test!
runTest Basic CACHE ASYNC_THROUGH
2016-07-19 16:08:28,327 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master3/10.197.0.5:29998
2016-07-19 16:08:28,329 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master3/10.197.0.5:29999
2016-07-19 16:08:28,330 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master3/10.197.0.5:29999 received
2016-07-19 16:08:28,331 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_CACHE_ASYNC_THROUGH took 6 ms.
2016-07-19 16:08:28,372 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master3/10.197.0.5:29998
2016-07-19 16:08:28,375 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ slave5.dev-etl.ants.vn/10.197.0.7:29998
2016-07-19 16:08:28,376 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine master3/10.197.0.5:29999
2016-07-19 16:08:28,377 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3801868140544 from remote machine master3/10.197.0.5:29999 received
2016-07-19 16:08:28,379 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999
2016-07-19 16:08:28,380 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999 received
2016-07-19 16:08:28,381 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_CACHE_ASYNC_THROUGH took 49 ms.
Passed the test!
runTest BasicNonByteBuffer CACHE ASYNC_THROUGH
Passed the test!
runTest Basic NO_CACHE MUST_CACHE
2016-07-19 16:08:28,429 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_MUST_CACHE took 2 ms.
2016-07-19 16:08:28,472 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_MUST_CACHE took 43 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE MUST_CACHE
2016-07-19 16:08:28,474 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ slave5.dev-etl.ants.vn/10.197.0.7:29998
2016-07-19 16:08:28,476 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999
2016-07-19 16:08:28,477 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999 received
2016-07-19 16:08:28,579 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ slave5.dev-etl.ants.vn/10.197.0.7:29998
2016-07-19 16:08:28,582 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999
2016-07-19 16:08:28,583 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3801918472192 from remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999 received
Passed the test!
runTest Basic NO_CACHE CACHE_THROUGH
2016-07-19 16:08:28,585 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,585 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,588 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:28,593 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:28,595 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:28,596 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_CACHE_THROUGH took 12 ms.
2016-07-19 16:08:28,622 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:28,624 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:28,626 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3801935249408 from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:28,626 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_CACHE_THROUGH took 30 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE CACHE_THROUGH
2016-07-19 16:08:28,628 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,629 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
Passed the test!
runTest Basic NO_CACHE THROUGH
2016-07-19 16:08:28,662 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,662 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,668 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_THROUGH took 7 ms.
2016-07-19 16:08:28,692 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,692 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,700 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_THROUGH took 32 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE THROUGH
2016-07-19 16:08:28,702 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,703 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,731 INFO  type (AbstractClient.java:connect) - Alluxio client (version 1.1.0) is trying to connect with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
2016-07-19 16:08:28,732 INFO  type (AbstractClient.java:connect) - Client registered with FileSystemWorkerClient FileSystemWorker @ master1/10.197.0.3:29998
Passed the test!
runTest Basic NO_CACHE ASYNC_THROUGH
2016-07-19 16:08:28,743 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:28,745 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:28,746 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:28,748 INFO  type (BasicOperations.java:writeFile) - writeFile to file /default_tests_files/Basic_NO_CACHE_ASYNC_THROUGH took 6 ms.
2016-07-19 16:08:28,777 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ master4/10.197.0.6:29998
2016-07-19 16:08:28,779 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine master4/10.197.0.6:29999
2016-07-19 16:08:28,780 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3802002358272 from remote machine master4/10.197.0.6:29999 received
2016-07-19 16:08:28,780 INFO  type (BasicOperations.java:readFile) - readFile file /default_tests_files/Basic_NO_CACHE_ASYNC_THROUGH took 32 ms.
Passed the test!
runTest BasicNonByteBuffer NO_CACHE ASYNC_THROUGH
2016-07-19 16:08:28,783 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ slave5.dev-etl.ants.vn/10.197.0.7:29998
2016-07-19 16:08:28,784 INFO  type (NettyRemoteBlockWriter.java:write) - Connected to remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999
2016-07-19 16:08:28,787 INFO  type (NettyRemoteBlockWriter.java:write) - status: SUCCESS from remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999 received
2016-07-19 16:08:28,826 INFO  type (BlockWorkerClient.java:connectOperation) - Connecting to remote worker @ slave5.dev-etl.ants.vn/10.197.0.7:29998
2016-07-19 16:08:28,828 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Connected to remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999
2016-07-19 16:08:28,829 INFO  type (NettyRemoteBlockReader.java:readRemoteBlock) - Data 3802019135488 from remote machine slave5.dev-etl.ants.vn/10.197.0.7:29999 received
Passed the test!



On Tuesday, July 19, 2016 at 2:27:03 PM UTC+7, Chanh Le wrote:

Chanh Le

unread,
Jul 19, 2016, 6:28:51 AM7/19/16
to Alluxio Users
Still got error when upgrade to 1.1.2 RC2

2016-07-19 10:24:29,589 ERROR logger.type (BlockDataServerHandler.java:handleBlockWriteRequest) - Error writing remote block : Temp blockId 127,339,069,440 is not available, because it is already committed
alluxio.exception.BlockAlreadyExistsException: Temp blockId 127,339,069,440 is not available, because it is already committed
at alluxio.worker.block.TieredBlockStore.checkTempBlockIdAvailable(TieredBlockStore.java:393)
at alluxio.worker.block.TieredBlockStore.createBlockMetaInternal(TieredBlockStore.java:521)
at alluxio.worker.block.TieredBlockStore.createBlockMeta(TieredBlockStore.java:184)
at alluxio.worker.block.BlockWorker.createBlockRemote(BlockWorker.java:336)
at alluxio.worker.netty.BlockDataServerHandler.handleBlockWriteRequest(BlockDataServerHandler.java:145)
at alluxio.worker.netty.DataServerHandler.channelRead0(DataServerHandler.java:71)
at alluxio.worker.netty.DataServerHandler.channelRead0(DataServerHandler.java:40)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:831)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:322)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
2016-07-19 10:25:08,846 INFO  logger.type (BlockDataServerHandler.java:handleBlockReadRequest) - Preparation for responding to remote block request for: 136918859776 done.
2016-07-19 10:25:08,922 INFO  logger.type (BlockDataServerHandler.java:handleBlockReadRequest) - Preparation for responding to remote block request for: 136767864832 done.


On Tuesday, July 19, 2016 at 2:27:03 PM UTC+7, Chanh Le wrote:

Chanh Le

unread,
Jul 25, 2016, 11:02:49 PM7/25/16
to Alluxio Users
Any update on that?


On Tuesday, July 19, 2016 at 2:27:03 PM UTC+7, Chanh Le wrote:

Pei Sun

unread,
Jul 25, 2016, 11:42:05 PM7/25/16
to Chanh Le, Alluxio Users
By looking at your log, it seems that you didn't upgrade to 1.1 correctly. Can you please send a copy of your configuration page in your Alluxio web UI? We can start from confirming the version. 

Thanks
Pei

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Pei Sun

Chanh Le

unread,
Jul 26, 2016, 12:02:10 AM7/26/16
to Pei Sun, Alluxio Users
Sure.


alluxio-env.sh
alluxio-site.properties

Chanh Le

unread,
Jul 26, 2016, 12:06:21 AM7/26/16
to Pei Sun, Alluxio Users
I attached the screenshot of configuration page.



On Jul 26, 2016, at 11:02 AM, Chanh Le <giao...@gmail.com> wrote:

Sure.


<alluxio-env.sh>
<alluxio-site.properties>

Pei Sun

unread,
Jul 26, 2016, 12:23:00 AM7/26/16
to Chanh Le, Alluxio Users
Thank you. You have attached several error msgs. Are they all from this version? Your latest error indicates that it took long to cache block locally. That could be the network is slow. You can try to increase alluxio.user.network.netty.timeout.ms  to see whether that solves the problem. After that, if you see performance issue, we can discuss more. 

Pei
--
Pei Sun

Chanh Le

unread,
Jul 26, 2016, 12:36:25 AM7/26/16
to Pei Sun, Alluxio Users
Thank Pei.
You mean it because the timeout in which spark was writing into alluxio and because of slow response from alluxio it throws error.
But it should show the error in user side not in worker site right?
Just confuse.

Regards.

On Jul 26, 2016, at 11:22 AM, Pei Sun <pe...@alluxio.com> wrote:

Thank you. You have attached several error msgs. Are they all from this version? Your latest error indicates that it took long to cache block locally. That could be the network is slow. You can try to increase alluxio.user.network.netty.timeout.ms  to see whether that solves the problem. After that, if you see performance issue, we can discuss more. 

Pei
On Mon, Jul 25, 2016 at 9:05 PM, Chanh Le <giao...@gmail.com> wrote:
I attached the screenshot of configuration page.


<Screen Shot 2016-07-26 at 11.03.18 AM.png><Screen Shot 2016-07-26 at 11.03.32 AM.png><Screen Shot 2016-07-26 at 11.03.44 AM.png><Screen Shot 2016-07-26 at 11.03.53 AM.png><Screen Shot 2016-07-26 at 11.04.05 AM.png><Screen Shot 2016-07-26 at 11.04.17 AM.png><Screen Shot 2016-07-26 at 11.04.26 AM.png><Screen Shot 2016-07-26 at 11.04.36 AM.png><Screen Shot 2016-07-26 at 11.04.43 AM.png>



--
Pei Sun

Pei Sun

unread,
Jul 26, 2016, 12:41:44 AM7/26/16
to Chanh Le, Alluxio Users
Your latest error is from client side. (FileInStream is the client code).

--
Pei Sun

Chanh Le

unread,
Jul 26, 2016, 12:44:52 AM7/26/16
to Pei Sun, Alluxio Users
Oh, no. You get it wrong because of slow response (>10 mins) from alluxio therefore Spark throws that error.

Pei Sun

unread,
Jul 26, 2016, 12:57:52 AM7/26/16
to Chanh Le, Alluxio Users
The current timeout is 30s.  Did you try to increase this timeout?  And could you share your full Alluxio log? 

Pei
--
Pei Sun

Chanh Le

unread,
Jul 26, 2016, 1:01:35 AM7/26/16
to Pei Sun, Alluxio Users
Thank you Pei.
Not yet. BTW timeout default 3000ms -> 3s. I will try to increase that.
Please help.

Pei Sun

unread,
Jul 26, 2016, 1:11:40 AM7/26/16
to Chanh Le, Alluxio Users
How did you get this 3s? If you look at configuration file sent to me, it was 30s. It was set to 3 seconds in 1.0.1 and changed to 30 seconds in 1.1.

--
Pei Sun

Chanh Le

unread,
Jul 26, 2016, 1:13:02 AM7/26/16
to Pei Sun, Alluxio Users
Sorry I read it wrong 
alluxio.user.network.netty.timeout.ms30000
30s. I confirm.

Pei Sun

unread,
Jul 26, 2016, 2:44:30 AM7/26/16
to Chanh Le, Alluxio Users
No worry. Let me know how it works. 
--
Pei Sun

Pei Sun

unread,
Oct 20, 2016, 7:33:32 PM10/20/16
to Chanh Le, Alluxio Users
Hi,
    Have you resolved the problem?

Pei

Pei

Sure.


<alluxio-env.sh>
<alluxio-site.properties>


To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



-- 
Pei Sun





--
Pei Sun




--
Pei Sun




--
Pei Sun




--
Pei Sun




--
Pei Sun



--
Pei Sun
Reply all
Reply to author
Forward
0 new messages