size exceeds integer.max_value spark

[ashu.shri]

Hi,

We are facing an issue where - "size exceeds integer.max_value spark". This exception is raised when "localIterator" start to write data. Not sure what is the probable cause, but our use case is to write data on driver node. This data size is too huge.

We have used localIterator in other cases too but have not encountered this error.

Many Thanks in advance.

[Russell Spitzer]

Could you please give a code example? This should only come up in a integer literal that is to big or an integer overflow

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--

Russell Spitzer
Software Engineer

https://github.com/datastax/spark-cassandra-connector/blob/master/doc/FAQ.md
http://spark-packages.org/package/datastax/spark-cassandra-connector

[ashu.shri]

Here is the code:

JavaPairRDD<String, Iterable<CassandraRow>> theSpanRDD = RDDSparkHelper.spanBy(theCasssandraJavaRDD);
JavaRDD<String> theJavaRDD = RDDSparkHelper.mapSpanToDumpJavaRDD(theSpanRDD, true, theStringRecords, map, id);
JavaRDD<String> theRepartitionedRDD = theJavaRDD.repartition(Integer.parseInt(theNumPartitions) * 3);
JavaRDD<String> theCachedRDD = theRepartitionedRDD.persist(StorageLevel.MEMORY_AND_DISK());
Iterator<String> theLocalRDD = theCachedRDD.toLocalIterator();

[Russell Spitzer]

And the actual exception?  Also what is theNumPartitions?

[sachin sharma]

Hi Russell,

I am working with Ashish on this problem,

"theNumPartitions" = 16

and the actual exception is:

Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times,
most recent failure: Lost task 0.3 in stage 1.0 (TID 3334, ):
java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:860)
at org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:123)
at org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:132)
at org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:517)
at org.apache.spark.storage.BlockManager.getLocal(BlockManager.scala:432)
at org.apache.spark.storage.BlockManager.get(BlockManager.scala:618)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:44)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:228)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:

[Russell Spitzer]

I would just try more partitions seems like you are going past some internal api's limit. Also when you get an executor side exception like this it helps to get the log from that executot

[sachin sharma]

Hi Russell,

We have increase no of partitions to around 3000 and we got following exception, while there was space available on the disk:-

org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device

        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:221)

        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

        at java.io.FilterOutputStream.close(FilterOutputStream.java:157)

        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)

        at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)

        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:398)

        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)

        at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)

        at org.apache.hadoop.mapred.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:108)

        at org.apache.spark.SparkHadoopWriter.close(SparkHadoopWriter.scala:102)

        at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1068)

        at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1047)

        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)

        at org.apache.spark.scheduler.Task.run(Task.scala:56)

        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: No space left on device

        at java.io.FileOutputStream.writeBytes(Native Method)

        at java.io.FileOutputStream.write(FileOutputStream.java:345)

        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:219)

        ... 18 more

[Russell Spitzer]

Sorry, but this does means you have no space left on the disk. It most likely is talking about temp files while it's writing shuffle files to disk. These will be close out/be removed when the job is finished so you won't see large files when the job is done.