TempBlockMeta not found when writing parquet file in Spark

[Dragisa Krsmanovic]

I am trying a simple JDBC table dump to parquet in Spark and I am getting "TempBlockMeta not found" every time Spark tries to finish writing parquet file.

Alluxio 1.0.1 with standalone Spark 1.6.1 cluster with 6 workers. Each worker has 5g reserved for Spark and 5g for Alluxio.

val jdbcDF = sqlContext.read.format("jdbc").options(
  Map(
    "mergeSchema"     -> "false",
    "driver"          -> dbConf.getString("driver"),
    "url"             -> dbConf.getString("url"),
    "user"            -> dbConf.getString("user"),
    "password"        -> dbConf.getString("password"),
    "dbtable"         -> "event_product_refs"
  )
).load()


jdbcDF.write
  .parquet("alluxio://diva/eventProductRefs.parquet")

 
It's a very small data set (~2m). I am just trying to write a simple "hello world" example with Spark and Alluxio.



Full stack trace:

16/04/28 22:24:02 WARN hadoop.ParquetOutputCommitter: could not write summary file for alluxio://dev17-spark-master01:19998/diva/eventProductRefs.parquet
java.io.IOException: Could not read footer: java.io.IOException: Could not read footer for file FileStatus{path=alluxio://dev17-spark-master01:19998/diva/eventProductRefs.parquet/part-r-00000-a5df7b1b-e9e4-4663-bdbb-e3bf68a32f0c.gz.parquet; isDirectory=false; length=1407788; replication=3; blocksize=536870912; modification_time=1461882237757; access_time=1461882237757; owner=; group=; permission=rw-rw-rw-; isSymlink=false}
    at org.apache.parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:247)
    at org.apache.parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:262)
    at org.apache.parquet.hadoop.ParquetOutputCommitter.writeMetaDataFile(ParquetOutputCommitter.java:56)
    at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:48)
    at org.apache.spark.sql.execution.datasources.BaseWriterContainer.commitJob(WriterContainer.scala:230)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:151)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)
    at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
    at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
    at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
    at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
    at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)
    at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:256)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:139)
    at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:334)
    at com.ticketfly.etl.diva.DumpProductRefsToAlluxio$delayedInit$body.apply(DumpProductRefsToAlluxio.scala:28)
    at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App$$anonfun$main$1.apply(App.scala:71)
    at scala.App$$anonfun$main$1.apply(App.scala:71)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
    at scala.App$class.main(App.scala:71)
    at com.ticketfly.etl.diva.DumpProductRefsToAlluxio$.main(DumpProductRefsToAlluxio.scala:6)
    at com.ticketfly.etl.diva.DumpProductRefsToAlluxio.main(DumpProductRefsToAlluxio.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Could not read footer for file FileStatus{path=alluxio://dev17-spark-master01:19998/diva/eventProductRefs.parquet/part-r-00000-a5df7b1b-e9e4-4663-bdbb-e3bf68a32f0c.gz.parquet; isDirectory=false; length=1407788; replication=3; blocksize=536870912; modification_time=1461882237757; access_time=1461882237757; owner=; group=; permission=rw-rw-rw-; isSymlink=false}
    at org.apache.parquet.hadoop.ParquetFileReader$2.call(ParquetFileReader.java:239)
    at org.apache.parquet.hadoop.ParquetFileReader$2.call(ParquetFileReader.java:233)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: alluxio.exception.BlockDoesNotExistException: TempBlockMeta not found for blockId 167,772,160
    at alluxio.client.block.RemoteBlockOutStream.cancel(RemoteBlockOutStream.java:85)
    at alluxio.client.file.FileInStream.closeCacheStream(FileInStream.java:288)
    at alluxio.client.file.FileInStream.close(FileInStream.java:117)
    at alluxio.hadoop.HdfsFileInputStream.close(HdfsFileInputStream.java:114)
    at java.io.FilterInputStream.close(FilterInputStream.java:181)
    at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:432)
    at org.apache.parquet.hadoop.ParquetFileReader$2.call(ParquetFileReader.java:237)
    ... 5 more
Caused by: alluxio.exception.BlockDoesNotExistException: TempBlockMeta not found for blockId 167,772,160
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at alluxio.exception.AlluxioException.from(AlluxioException.java:73)
    at alluxio.AbstractClient.retryRPC(AbstractClient.java:324)
    at alluxio.client.block.BlockWorkerClient.cancelBlock(BlockWorkerClient.java:156)
    at alluxio.client.block.RemoteBlockOutStream.cancel(RemoteBlockOutStream.java:83)
    ... 11 more

[Dragisa Krsmanovic]

Interestingly.

Looks like, despite this error, parquet file is correctly created.

...

[Pei Sun]

Hi Dragisa,

    I was not able to reproduce this. Do you mind sharing your hello world program, the small dataset, and how you setup the alluxio, spark and ufs? It will be great if you can help me to reproduce the problem.

Thanks

Pei

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Pei Sun

[Dragisa Krsmanovic]

Hi Pei,

The example is dumping a small table from MySQL (~400k rows). I am not event doing any partitioning. I posted the code above.

Table schema is simple:
CREATE TABLE event_product_refs
(
    event_id BIGINT(20) NOT NULL,
    product_id BIGINT(20) NOT NULL,
    entombed BIT(1) DEFAULT b'0'
);

Oracle Java 1.8.74

alluxio runTests is passing

UnderFS is s3:
export ALLUXIO_UNDERFS_ADDRESS=s3n://<bucket>/alluxio
export ALLUXIO_RAM_FOLDER=/mnt/ramfs
export ALLUXIO_WORKER_MEMORY_SIZE=5GB

export ALLUXIO_JAVA_OPTS+="
  -Xmx1g
  -Dlog4j.configuration=file:${CONF_DIR}/log4j.properties
  -Dalluxio.worker.tieredstore.levels=1
  -Dalluxio.worker.tieredstore.level0.alias=MEM
  -Dalluxio.worker.tieredstore.level0.dirs.path=${ALLUXIO_RAM_FOLDER}
  -Dalluxio.worker.tieredstore.level0.dirs.quota=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.underfs.address=${ALLUXIO_UNDERFS_ADDRESS}
  -Dalluxio.worker.memory.size=${ALLUXIO_WORKER_MEMORY_SIZE}
  -Dalluxio.master.hostname=${ALLUXIO_MASTER_ADDRESS}
  -Dalluxio.master.journal.folder=/opt/alluxio/master_journal
  -Dfs.s3n.awsAccessKeyId=${AWS_ACCESS_KEY_ID}
  -Dfs.s3n.awsSecretAccessKey=${AWS_SECRET_ACCESS_KEY}
  -Dorg.apache.jasper.compiler.disablejsr199=true
  -Djava.net.preferIPv4Stack=true
"

I don't see this error with smaller tables. Under 200k rows.


In master.log I see warnings like this:

2016-05-06 04:42:37,310 WARN  httpclient.RestStorageService (RestStorageService.java:performRequest) - Error Response: HEAD '/alluxio%2Fdiva%2FeventProductRefs.parquet_%24folder%24' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Fri, 06 May 2016 04:42:37 GMT, Authorization: AWS xxxxxxxxxxxxxxx:xxxxxxxxxxxxxxxx, User-Agent: JetS3t/0.8.1 (Linux/3.2.0-72-virtual; amd64; en; JVM 1.8.0_74), Host: tfly-spark-dev17.s3.amazonaws.com], Response Headers: [x-amz-request-id: A2D6B4E27047095B, x-amz-id-2: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Fri, 06 May 2016 04:42:36 GMT, Server: AmazonS3]

Let me know if you need more details.

[Pei Sun]

Hi Dragisa,

    Thank you for the details. I can reproduce the problem now and will come up with fixes. I will keep you posted.

Pei

[Dragisa Krsmanovic]

Great ! I am glad it's not only me.

Thank you.

[Dragisa Krsmanovic]

Pei,

Do you have a Jira ticket for this that I can watch ?

--

Dragiša Krsmanović | Platform Engineer | Ticketfly

drag...@ticketfly.com

@ticketfly | ticketfly.com/blog | facebook.com/ticketfly

[Pei Sun]

Hi Dragisa,

   You can watch this one https://alluxio.atlassian.net/browse/ALLUXIO-1942 . I didn't put much information there. But I will link the PRs to it in the future. 

Pei

[Pei Sun]

Hi Dragisa, 

   I have sent one fix. https://github.com/Alluxio/alluxio/pull/3223 . You can patch that can give it a try. You need to set 

--conf spark.driver.extraJavaOptions="-Dalluxio.user.file.cache.partially.read.block=false" 

in spark to use it. Let me know if you find any problem. 

It is recommended to colocate Alluxio and Spark. You can see bigger performance gain that way. 

Thank you.

Pei

--

Pei Sun

[Dragisa Krsmanovic]

Thank you Pei,

Unfortunately, I don't think I'll have bandwidth to test this soon.

Dragisa

[Pei Sun]

No problem. Take you time. It will be merged to the master. Just let us know if you see any new problem in the future. 

Thanks

Pei

--

Pei Sun

[Jais Sebastian]

Hi Pei,
I am also facing similar issue while writing  parquet file from Spark. If I disbale parquet summary enabled to false in Spark - there is no error. But the error is consistent if I enable it. Also I tried setting partially read block to disabled as you suggested in last discussion.

I am deploying in Mesos Alluxio version 1.4 and Spark 2.1.1

[Pei Sun]

Hi Jais,

    There is an known issue to use spark 2.1 with alluxio 1.4. The workaround is documented here. But it does seem that you are hitting that issue if you are seeing TempBlockMeta not found. I tried to reproduce the issue with 1.4 and it worked for me. Can you tell me a little more about your workloads and attach your logs?

Pei

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Pei Sun

[Jais Sebastian]

Hi Pei,

I dont see temp block issue with new environment. I will recheck again on this.

Can you point me to the specific solution for parquet summary error ?

Regards,

Jais

--

Regards,
Jais Sebastian
+919980722994

"There are only 10 types of people in the world: Those who understand binary, and those who don't."

(¨`•.•´¨) Keep
`•.¸(¨`•.•´¨) Smiling
      (¨`•.•´¨)¸.•´ Always
       `•.¸.•´

[Pei Sun]

Hi Jais,

    The parquet summary error was caused by the temp block issue and was fixed in PR.  It should have already been fixed in 1.4.  If you are seeing parquet summary error, i need to see the actual alluxio log to further debug.

Thank you.

Pei

--

Pei Sun