Spark Scala Programming for not serializable objects and functions

Question

I have a "Task not serializable" Exception when I run a Spark Scala program with

Spark RDDs is of not serializable type (java class)
called functions are from a not serializable class (java class, again)

my code is something like this

object Main{
    def main(args : Array(String){
        ...
        var rdd = sc.textFile(filename)
                  .map(line => new NotSerializableJClass(line)).cache() 
        //rdd is RDD[NotSerializableJClass]
        ...
        var test = new NotSerializableJPredicate()
        rdd = rdd.filter(elem => test.test(elem))
        //throws TaskNotSerializable on test Predicate class
    }
}

I notice that i can resolve the second part with

rdd = rdd.filter(elem => (new NotSerializableJPredicate()).test(elem))

but I still get that Exception for the class of the objects in RDDs. And I would in another way also the second part in another way just because I don't want to create a great number of PredicateClass's object.

Can you help me? How can I go forward with non-serializable class?

Is NotSerializableJClass a third party class or a class defined in your application ? — himanshuIIITian
– himanshuIIITian, Commented Apr 8, 2017 at 13:58

puhlen · Accepted Answer · 2017-04-10 14:10:31Z

1

RDDs must be serializable so you cannot create an RDD of a non serializable class.

For your predicate, you could write it using mapPartitions.

rdd.mapPartitions{
  part => 
    val test = new NotSerializableJPredicate()
    part.filter{elem => test.test(elem)}
   }

mapPartitons will run once per partition, so it allows you to instantiate non-serializable classes on the executor, but it only needs to do it once per partition rather than for every record.

answered Apr 10, 2017 at 14:10

puhlen

8,5591 gold badge19 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

sourabh · Accepted Answer · 2017-04-10 08:04:10Z

1

Some of the general rules that has helped me avoid Task Serialization Issue:

If you are calling a method of any class from your code;Spark will need to serialize the entire class containing the method.Ways to go around can be any of the following : a> Declare the method as a function variable in NotSerializableClass; so instead of writing : def foo(x:Int)={blah blah} try using val foo = (x:Int)=>{blah blah } So; spark no longer needs to serialize the entire class now. b> Refactoring the code to extract the relevant portions in a separate class may be the way to go in certain cases. c>Mark objects in the class which are actually not necessary for the job as @transient and mark the class Serializable

answered Apr 10, 2017 at 8:04

sourabh

4764 silver badges13 bronze badges

Collectives™ on Stack Overflow

Spark Scala Programming for not serializable objects and functions

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related