Failed to upload policy when writing to GS
vacuu...@gmail.com
KakaKuwa
> I get failed to upload policy caused by insufficient data written when secor is writing data to GS.
Any luck with this problem ?
I am getting something like this
2016-06-27 09:06:38,432 [Thread-5] (com.pinterest.secor.uploader.HadoopS3UploadManager:51) INFO uploading file /tmp/secor_dev/message_logs/backup/13605_15/dt=20160627/staging-error_log-20160627-00-00.gz to s3n://saavninterns/milindv/dt=20160627/staging-error_log-20160627-00-00.gz
2016-06-27 09:06:40,851 [pool-1-thread-2] (org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker:157) WARN Problem opening checksum file: file:/tmp/secor_dev/message_logs/backup/13605_15/dt=20160627/staging-error_log-20160627-01-00.gz. Ignoring exception:
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:149)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:364)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1933)
at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1920)
at com.pinterest.secor.util.FileUtil.moveToCloud(FileUtil.java:207)
at com.pinterest.secor.uploader.HadoopS3UploadManager$1.run(HadoopS3UploadManager.java:67)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-06-27 09:06:41,520 [pool-1-thread-2] (org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream:242) INFO OutputStream for key 'milindv/dt=20160627/staging-error_log-20160627-01-00_1.gz' writing to tempfile '/tmp/hadoop-root/s3/output-3405607316787673864.tmp'
2016-06-27 09:06:41,520 [pool-1-thread-2] (org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream:276) INFO OutputStream for key 'milindv/dt=20160627/staging-error_log-20160627-01-00_1.gz' closed. Now beginning upload
2016-06-27 09:06:41,746 [pool-1-thread-2] (org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream:288) INFO OutputStream for key 'milindv/dt=20160627/staging-error_log-20160627-01-00_1.gz' upload complete
Jun 27, 2016 9:06:41 AM com.twitter.common.zookeeper.DistributedLockImpl unlock
INFO: Cleaning up this locks ephemeral node.
Jun 27, 2016 9:06:41 AM com.twitter.common.zookeeper.DistributedLockImpl cleanup
INFO: Cleaning up!
2016-06-27 09:06:41,751 [Thread-5] (com.pinterest.secor.main.ConsumerMain$1:65) ERROR Thread Thread[Thread-5,5,main] failed
java.lang.RuntimeException: Failed to apply upload policy
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:115)
at com.pinterest.secor.consumer.Consumer.run(Consumer.java:105)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.FileNotFoundException: File /tmp/secor_dev/message_logs/backup/13605_15/dt=20160627/staging-error_log-20160627-00-00.gz does not exist
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at com.pinterest.secor.uploader.FutureHandle.get(FutureHandle.java:34)
at com.pinterest.secor.uploader.Uploader.uploadFiles(Uploader.java:96)
at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:187)
at com.pinterest.secor.uploader.Uploader.applyPolicy(Uploader.java:210)
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:113)
... 1 more
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File /tmp/secor_dev/message_logs/backup/13605_15/dt=20160627/staging-error_log-20160627-00-00.gz does not exist
at com.pinterest.secor.uploader.HadoopS3UploadManager$1.run(HadoopS3UploadManager.java:69)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: File /tmp/secor_dev/message_logs/backup/13605_15/dt=20160627/staging-error_log-20160627-00-00.gz does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1933)
at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1920)
at com.pinterest.secor.util.FileUtil.moveToCloud(FileUtil.java:207)