java.io.IOException: Stream closed
at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:453)
at java.io.OutputStream.write(OutputStream.java:127)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:93)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:137)
at org.apache.hadoop.hive.ql.exec.TextRecordWriter.write(TextRecordWriter.java:53)
org.apache.spark.sql.hive.execution.ScriptTransformationWriterThread$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(ScriptTransformation.scala:255)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)Unsupported language features in query: INSERT INTO TABLE q07_spark_run_query_0_result
SELECT
ca_state,
COUNT(*) AS cnt
FROM
customer_address a,
customer c,
store_sales s,
...
(
...
)
GROUP BY ca_state
HAVING cnt >= 10 --at least 10 customers
ORDER BY cnt DESC, ca_state --top 10 states in descending order
LIMIT 1016/01/01 21:54:12 ERROR Utils: Uncaught exception in thread Thread-ScriptTransformation-Feed
java.lang.NullPointerException
at org.apache.spark.unsafe.memory.TaskMemoryManager.getPage(TaskMemoryManager.java:235)
at org.apache.spark.util.collection.unsafe.sort. UnsafeInMemorySorter$SortedIterator.loadNext(UnsafeInMemorySorter.java:165)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter$1.next(UnsafeExternalRowSorter.java:142)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter$1.next(UnsafeExternalRowSorter.java:129)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at org.apache.spark.sql.hive.execution.ScriptTransformationWriterThread$$anonfun$run$1.apply$mcV$sp(ScriptTransformation.scala:255)It is spark 1.5.2
I found the exact error from query 2,4,8.
It was the path issue, but after solving these issues I found the other error, which is Error in query: cannot recognize input near '$' '{' 'hiveconf' in table name; line 1 pos 21.
I guess sql expression in the queries does not create or delete temp_table. It is still in JIRA issue "https://issues.apache.org/jira/browse/SPARK-11972"
But you know the other people succeded the queries before. How could it be?
--
You received this message because you are subscribed to a topic in the Google Groups "Big Data Benchmark for BigBench" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/big-bench/yuy3VkP--ow/unsubscribe.
To unsubscribe from this group and all its topics, send an email to big-bench+...@googlegroups.com.
To post to this group, send email to big-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/big-bench/bc0b9831-023d-497f-8d12-a7b414b1c4cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
To view this discussion on the web visit https://groups.google.com/d/msgid/big-bench/66993ae4-e2be-43d2-a07e-3f7163ff7cfb%40googlegroups.com.
For example, reproduce steps:
/usr/lib/spark/bin/spark-sql -v --driver-memory 4g --executor-memory 7g
--executor-cores 5 --num-executors 31 --master yarn-client --conf
spark.yarn.executor.memoryOverhead=1024 --hiveconf
RESULT_TABLE=test_result01
>use test; >DROP TABLE IF EXISTS ${hiveconf:RESULT_TABLE};
To view this discussion on the web visit https://groups.google.com/d/msgid/big-bench/e5e74dfd-a4d4-49bb-b30b-9b296639ddd3%40googlegroups.com.
Thanks for your try.
So did you run queries with external scripts, which are 2, 4, 8? What about trying --master local(*) ? This option was the recommend option.
But first you were supposed to see the error with return type value unless you change the return type value in Reduce ... Using Code.
To view this discussion on the web visit https://groups.google.com/d/msgid/big-bench/eea5b098-b7b3-4538-8770-fdd236ab2200%40googlegroups.com.