Java Spark spilling in-memory map OutOfMemoryError error

Question

I'm running the following code (a contains about 10000 rows):

IndexedRowMatrix qTmp = new IndexedRowMatrix(a.rows());
IndexedRowMatrix qTmpT = qTmp.toCoordinateMatrix().transpose().toIndexedRowMatrix();

I get the following error:

15/06/19 10:08:52 INFO ExternalAppendOnlyMap: Thread 66 spilling in-memory map of 24.8 MB to disk (1 time so far)
15/06/19 10:08:54 INFO ExternalAppendOnlyMap: Thread 66 spilling in-memory map of 24.8 MB to disk (2 times so far)
15/06/19 10:08:57 INFO ExternalAppendOnlyMap: Thread 66 spilling in-memory map of 24.8 MB to disk (3 times so far)
15/06/19 10:09:02 ERROR Executor: Exception in task 0.0 in stage 9.0 (TID 9)
java.lang.OutOfMemoryError: Java heap space

I have tried tuning memory with various parameters as well:

SparkConf conf = new SparkConf();
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
conf.setAppName("test");
conf.setMaster("local[2]");
conf.set("spark.executor.memory", "6g");
conf.set("spark.driver.maxResultSize", "1g");
conf.set("spark.shuffle.spill", "false");
conf.set("spark.storage.memoryFraction", "0.3");
conf.set("spark.eventLog.enabled", "false");
conf.set("driver-memory", "4g")`;
JavaSparkContext sc = new JavaSparkContext(conf);

Is there a reason I'm consistently getting this spilling error?

working with eclipse ?

Kshitij Kulshrestha
– Kshitij Kulshrestha

2015-06-19 10:23:52 +00:00
Commented Jun 19, 2015 at 10:23 — Kshitij Kulshrestha
– Kshitij Kulshrestha, Commented Jun 19, 2015 at 10:23

Aleksei · Accepted Answer · 2015-06-19 07:18:39Z

1

You should run your application with overridden java options.

Start the program with -Xms=[size] -Xmx -XX:MaxPermSize=[size] -XX:MaxNewSize=[size]

For example -

-Xms512m -Xmx1152m -XX:MaxPermSize=256m -XX:MaxNewSize=256m

answered Jun 19, 2015 at 7:18

Aleksei

1028 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Java Spark spilling in-memory map OutOfMemoryError error

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related